can you load these scripts : {
    "./main.py": "# File: main.py\n\nimport os\nimport logging\nimport torch\nimport torch.multiprocessing as mp\nimport yaml\nimport argparse\n\nfrom training.runner import train_worker\nfrom training.utils import (\n    validate_and_cast_config,\n    setup_logging,\n    setup_seed,\n)\n\ndef main(config):\n    \"\"\"\n    Entry point for training. Initializes multiprocessing for distributed training.\n    \"\"\"\n    logger = logging.getLogger(\"Main\")\n    logger.info(\"Entering main() function.\")\n\n    # Set random seed\n    setup_seed(config[\"random_seed\"])\n\n    # Number of GPUs available\n    num_gpus = torch.cuda.device_count()\n    configured_world_size = config[\"distributed\"][\"world_size\"]\n    world_size = min(num_gpus, configured_world_size) if num_gpus >= 1 else 1\n    logger.info(f\"Number of GPUs available: {num_gpus}\")\n    logger.info(f\"Configured world_size: {configured_world_size}, using world_size: {world_size}\")\n\n    if world_size > 1:\n        # Use DistributedDataParallel (DDP)\n        logger.info(\"Using DistributedDataParallel for multi-GPU training.\")\n\n        # Set up environment variables for distributed training\n        os.environ[\"MASTER_ADDR\"] = config[\"distributed\"][\"master_addr\"]\n        os.environ[\"MASTER_PORT\"] = str(config[\"distributed\"][\"master_port\"])  # Ensure it's a string\n        logger.info(f\"MASTER_ADDR set to {config['distributed']['master_addr']}\")\n        logger.info(f\"MASTER_PORT set to {config['distributed']['master_port']}\")\n\n        try:\n            mp.spawn(\n                train_worker,\n                args=(config, world_size),\n                nprocs=world_size,\n                join=True\n            )\n        except Exception as e:\n            logger.error(f\"Training failed during mp.spawn: {e}\", exc_info=True)\n            raise\n    else:\n        # Single GPU or CPU\n        logger.info(\"Single GPU or CPU detected.\")\n        try:\n            train_worker(0, config, world_size)\n        except Exception as e:\n            logger.error(f\"Training failed: {e}\", exc_info=True)\n            raise\n\n\nif __name__ == \"__main__\":\n    # Parse command-line arguments for flexibility\n    parser = argparse.ArgumentParser(description=\"Distributed Training Script\")\n    parser.add_argument('--config_path', type=str, default=\"config/config.yaml\",\n                        help='Path to the configuration YAML file.')\n    args = parser.parse_args()\n\n    logger = setup_logging(\"logs/main.log\")\n    logger.info(\"Starting main execution.\")\n\n    config_path = args.config_path\n    if not os.path.exists(config_path):\n        logger.error(f\"Configuration file not found at path: {config_path}\")\n        raise FileNotFoundError(f\"Configuration file not found at path: {config_path}\")\n\n    try:\n        with open(config_path, \"r\") as f:\n            raw_cfg = yaml.safe_load(f)\n            logger.debug(f\"Loaded raw config: {raw_cfg}\")\n    except yaml.YAMLError as e:\n        logger.error(f\"Failed to parse the configuration file: {e}\")\n        raise\n\n    try:\n        config = validate_and_cast_config(raw_cfg)\n        logger.info(\"Configuration validated successfully.\")\n        logger.info(f\"Loaded configuration:\\n{yaml.dump(config)}\")  # Log the full configuration\n    except Exception as e:\n        logger.error(f\"Configuration validation failed: {e}\", exc_info=True)\n        raise\n\n    logger.info(\"Configuration loaded and validated successfully.\")\n\n    # Start training\n    try:\n        # Explicitly set multiprocessing start method for compatibility\n        if os.name == 'nt':\n            mp.set_start_method('spawn', force=True)\n        main(config)\n    except Exception as e:\n        logger.error(f\"Training failed with exception: {e}\", exc_info=True)\n        raise",
    "./collection_json.py": "# File: collection_json.py\n\nimport os\nimport json\nimport argparse\nimport logging\n\ndef setup_logging():\n    \"\"\"\n    Configure logging to output to both file and console for this script.\n    \"\"\"\n    logger = logging.getLogger(\"CollectionJSON\")\n    logger.setLevel(logging.DEBUG)\n\n    # Create console handler\n    ch = logging.StreamHandler()\n    ch.setLevel(logging.INFO)\n\n    # Create file handler\n    fh = logging.FileHandler(\"logs/collection_json.log\")\n    fh.setLevel(logging.DEBUG)\n\n    # Create formatter and add to handlers\n    formatter = logging.Formatter(\"%(asctime)s - %(levelname)s - %(message)s\")\n    ch.setFormatter(formatter)\n    fh.setFormatter(formatter)\n\n    # Add handlers to the logger if none exist\n    if not logger.hasHandlers():\n        logger.addHandler(ch)\n        logger.addHandler(fh)\n\n    return logger\n\ndef collect_scripts_to_json(root_dir, output_file, logger):\n    \"\"\"\n    Traverse the given directory and its subdirectories to collect .py and .yaml files,\n    then save their contents into a JSON file.\n    \"\"\"\n    logger.info(\"Starting script collection.\")\n    scripts = {}\n\n    for dirpath, _, filenames in os.walk(root_dir):\n        for file in filenames:\n            if file.endswith(('.py', '.yaml')):\n                file_path = os.path.join(dirpath, file)\n                try:\n                    with open(file_path, 'r') as f:\n                        scripts[file_path] = f.read()\n                    logger.debug(f\"Collected script: {file_path}\")\n                except Exception as e:\n                    logger.error(f\"Failed to read {file_path}: {e}\")\n\n    try:\n        with open(output_file, 'w') as json_file:\n            json.dump(scripts, json_file, indent=4)\n        logger.info(f\"Scripts successfully collected and saved to {output_file}\")\n    except Exception as e:\n        logger.error(f\"Failed to write to {output_file}: {e}\")\n\ndef main():\n    parser = argparse.ArgumentParser(description=\"Collect .py and .yaml scripts into a JSON file.\")\n    parser.add_argument('--root_dir', type=str, default='./', help='Root directory to search for scripts.')\n    parser.add_argument('--output_file', type=str, default='scripts_collection.json', help='Output JSON file path.')\n\n    args = parser.parse_args()\n\n    logger = setup_logging()\n    logger.info(f\"Starting collection from root directory: {args.root_dir}\")\n\n    collect_scripts_to_json(args.root_dir, args.output_file, logger)\n\nif __name__ == \"__main__\":\n    main()",
    "./monitoring.py": "# File: monitoring.py\n\nimport os\nimport logging\nimport json\nfrom torch.utils.tensorboard import SummaryWriter\nfrom training.utils import log_memory_usage, log_gpu_memory\nimport gc\n\n# Initialize logger for Monitoring\nlogger = logging.getLogger(\"Monitoring\")\nlogger.setLevel(logging.INFO)\nif not logger.hasHandlers():\n    os.makedirs(\"logs\", exist_ok=True)\n    fh = logging.FileHandler(\"logs/monitoring.log\")\n    fh.setLevel(logging.INFO)\n    ch = logging.StreamHandler()\n    ch.setLevel(logging.INFO)\n    formatter = logging.Formatter(\"%(asctime)s - %(levelname)s - %(name)s - %(message)s\")\n    fh.setFormatter(formatter)\n    ch.setFormatter(formatter)\n    logger.addHandler(fh)\n    logger.addHandler(ch)\n\n\ndef interpret_metrics(avg_metrics, eval_metrics, logger: logging.Logger):\n    \"\"\"\n    Analyze and provide insights based on training and evaluation metrics.\n    \n    Args:\n        avg_metrics (dict): Dictionary containing average training metrics.\n        eval_metrics (dict): Dictionary containing evaluation metrics.\n        logger (logging.Logger): Logger instance for logging insights.\n    \"\"\"\n    try:\n        # Analyze training loss\n        actor_loss = avg_metrics.get(\"actor_loss\", None)\n        critic_loss = avg_metrics.get(\"critic_loss\", None)\n        entropy = avg_metrics.get(\"entropy\", None)\n    \n        if actor_loss is not None:\n            if actor_loss > 1.0:\n                logger.info(\"Actor loss is high. Consider lowering the learning rate or adjusting model complexity.\")\n            elif actor_loss < 0.01:\n                logger.info(\"Actor loss is very low. Model might be converging well.\")\n    \n        if critic_loss is not None:\n            if critic_loss > 1.0:\n                logger.info(\"Critic loss is high. Monitor for potential overfitting or adjust model.\")\n            elif critic_loss < 0.01:\n                logger.info(\"Critic loss is very low. Model might be underfitting.\")\n    \n        # Analyze entropy\n        if entropy is not None:\n            if entropy > 1.0:\n                logger.info(\"Entropy is high. Model is exploring more.\")\n            elif entropy < 0.1:\n                logger.info(\"Entropy is low. Model is exploiting more.\")\n    \n        # Analyze evaluation metrics\n        if eval_metrics:\n            average_reward = eval_metrics.get(\"average_reward\", 0)\n            sharpe_ratio = eval_metrics.get(\"sharpe_ratio\", 0)\n            max_drawdown = eval_metrics.get(\"max_drawdown\", 0)\n            average_trade_count = eval_metrics.get(\"average_trade_count\", 0)\n            average_win_rate = eval_metrics.get(\"average_win_rate\", 0)\n    \n            if average_reward > 0:\n                logger.info(\"Model is achieving positive returns on evaluation.\")\n            else:\n                logger.info(\"Model is not achieving positive returns on evaluation.\")\n    \n            if sharpe_ratio > 1.0:\n                logger.info(\"Sharpe Ratio is good, indicating a favorable risk-return trade-off.\")\n            elif sharpe_ratio < 1.0 and sharpe_ratio > 0:\n                logger.info(\"Sharpe Ratio is moderate, consider optimizing the strategy for better performance.\")\n            else:\n                logger.info(\"Sharpe Ratio is poor, consider re-evaluating the trading strategy.\")\n    \n            if max_drawdown < 0.1:\n                logger.info(\"Max Drawdown is low, indicating controlled risk.\")\n            else:\n                logger.info(\"Max Drawdown is high, consider implementing risk management strategies.\")\n    \n            logger.info(f\"Average Trade Count per Episode: {average_trade_count:.2f}\")\n            logger.info(f\"Average Win Rate: {average_win_rate:.2%}\")\n    except Exception as e:\n        logger.error(f\"Exception in interpret_metrics: {e}\", exc_info=True)\n\n\ndef monitor_training(\n    epoch: int,\n    rank: int,\n    avg_metrics: dict,\n    eval_metrics: dict,\n    writer: SummaryWriter = None\n) -> None:\n    \"\"\"\n    Monitor and log training progress, metrics, and insights.\n\n    Args:\n        epoch (int): Current epoch number.\n        rank (int): Process rank (for compatibility).\n        avg_metrics (dict): Dictionary of average training metrics.\n        eval_metrics (dict): Dictionary of evaluation metrics.\n        writer (SummaryWriter, optional): TensorBoard SummaryWriter.\n    \"\"\"\n    logs_dir = \"logs\"\n    os.makedirs(logs_dir, exist_ok=True)\n\n    # Log metrics in console logs\n    logger.info(f\"Epoch {epoch + 1} - Training Metrics: {avg_metrics}\")\n    logger.info(f\"Epoch {epoch + 1} - Evaluation Metrics: {eval_metrics}\")\n\n    # Write metrics to a JSON file\n    metrics_file = os.path.join(logs_dir, \"metrics.json\")\n    try:\n        with open(metrics_file, \"a\", encoding='utf-8') as f:\n            json.dump(\n                {\n                    \"epoch\": epoch + 1,\n                    \"train_metrics\": avg_metrics,\n                    \"eval_metrics\": eval_metrics,\n                },\n                f,\n                ensure_ascii=False\n            )\n            f.write(\"\\n\")\n        logger.info(f\"Metrics appended to {metrics_file}\")\n    except Exception as e:\n        logger.error(f\"Failed to write to {metrics_file}: {e}\")\n\n    if writer:\n        # Write training metrics\n        for k, v in avg_metrics.items():\n            writer.add_scalar(f\"Train/{k}\", v, epoch)\n        # Write evaluation metrics\n        for k, v in eval_metrics.items():\n            writer.add_scalar(f\"Eval/{k}\", v, epoch)\n        writer.flush()\n\n    # Optionally log CPU/GPU usage\n    log_memory_usage(logger, phase=\"After Logging Metrics\")\n    log_gpu_memory(logger, phase=\"After Logging Metrics\")\n\n    # Interpret metrics\n    try:\n        interpret_metrics(avg_metrics, eval_metrics, logger)\n    except Exception as e:\n        logger.error(f\"Failed to interpret metrics: {e}\", exc_info=True)\n\n\ndef setup_tensorboard(rank: int, log_dir: str = \"runs\") -> SummaryWriter:\n    \"\"\"\n    Setup TensorBoard writer.\n\n    Args:\n        rank (int): Process rank.\n        log_dir (str): Directory for TensorBoard logs.\n\n    Returns:\n        SummaryWriter: TensorBoard SummaryWriter instance.\n    \"\"\"\n    writer = None\n    os.makedirs(log_dir, exist_ok=True)\n    try:\n        writer = SummaryWriter(log_dir=log_dir)\n        logger.info(f\"TensorBoard writer set up at {log_dir}.\")\n    except Exception as e:\n        logger.error(f\"Failed to set up TensorBoard writer: {e}\")\n    return writer",
    "./environment/trading_env.py": "# environment/trading_env.py\n\nimport gym\nfrom gym import spaces\nimport torch\nimport numpy as np\nimport os\nimport glob\nimport logging\n\n# Initialize logger for TradingEnv\ndef setup_trading_env_logging(log_dir=\"logs\", log_file=\"trading_env.log\", file_logging_enabled=False):\n    \"\"\"\n    Configure logging for the TradingEnv with an on/off switch for file logging.\n\n    Args:\n        log_dir (str): Directory to store log files.\n        log_file (str): Log file name.\n        file_logging_enabled (bool): Flag to enable/disable file logging.\n    \"\"\"\n    os.makedirs(log_dir, exist_ok=True)\n    logger = logging.getLogger(\"TradingEnv\")\n    logger.setLevel(logging.DEBUG)\n\n    # Create console handler\n    ch = logging.StreamHandler()\n    ch.setLevel(logging.INFO)\n\n    # Create formatter and add to handlers\n    formatter = logging.Formatter(\"%(asctime)s - %(levelname)s - %(name)s - %(message)s\")\n    ch.setFormatter(formatter)\n\n    # Add handlers to the logger if none exist\n    if not logger.hasHandlers():\n        logger.addHandler(ch)\n\n        # Add file handler based on the flag\n        if file_logging_enabled:\n            fh = logging.FileHandler(os.path.join(log_dir, log_file))\n            fh.setLevel(logging.DEBUG)\n            fh.setFormatter(formatter)\n            logger.addHandler(fh)\n        else:\n            # Add a NullHandler to effectively disable file logging\n            fh = logging.NullHandler()\n            logger.addHandler(fh)\n\n    return logger\n\nlogger = logging.getLogger(\"TradingEnv\")\nif not logger.hasHandlers():\n    logger = setup_trading_env_logging()\n\nclass TradingEnv(gym.Env):\n    \"\"\"\n    Custom Trading Environment that follows OpenAI Gym interface.\n    \"\"\"\n    metadata = {'render.modes': ['human']}\n\n    def __init__(self, config):\n        super(TradingEnv, self).__init__()\n\n        self.config = config\n\n        # Initialize attributes\n        self.feature_columns = config['data']['feature_columns']\n        self.n_steps = config['data']['n_steps']\n        self.max_steps = config['environment']['max_steps']\n\n        # Ensure feature_dim matches number of feature_columns\n        feature_dim = config['model']['feature_dim']\n        assert feature_dim == len(self.feature_columns), \\\n            f\"feature_dim ({feature_dim}) does not match the number of feature columns ({len(self.feature_columns)}).\"\n        logger.debug(f\"feature_dim ({feature_dim}) matches number of feature columns ({len(self.feature_columns)}).\")\n\n        # Load data from PT files\n        pt_data_dir = config['data'].get('pt_data_dir', 'data/processed/')\n        pt_data_prefix = config['data'].get('pt_data_prefix', 'processed_data_part')\n        pt_files = sorted(glob.glob(os.path.join(pt_data_dir, f\"{pt_data_prefix}_*.pt\")))\n\n        if not pt_files:\n            logger.error(f\"No processed data files found in {pt_data_dir}\")\n            raise FileNotFoundError(f\"No processed data files found in {pt_data_dir}\")\n\n        self.pt_files = pt_files\n        self.current_pt_file_idx = -1\n        self.data = None\n        self.close_prices = None  # Unnormalized close prices for current data chunk\n\n        # Call load_next_data_chunk() after initializing attributes\n        self.load_next_data_chunk()\n\n        # Define action and observation space\n        self.action_dim = config['model']['action_dim']\n        self.action_space = spaces.Discrete(self.action_dim)\n        self.observation_space = spaces.Box(\n            low=-np.inf,\n            high=np.inf,\n            shape=(self.n_steps, len(self.feature_columns)),\n            dtype=np.float32\n        )\n        logger.debug(f\"Action space: {self.action_space}, Observation space: {self.observation_space}\")\n\n        # Initialize variables\n        self.current_step = 0\n        self.position = 0.0  # Position can now be fractional\n        self.total_profit = 0.0\n        self.entry_price = None\n\n        # Action mappings for scalping\n        if self.action_dim != 7:\n            logger.warning(f\"Expected action_dim=7, but got action_dim={self.action_dim}. Action mappings may be incorrect.\")\n        self.action_mappings = {\n            0: ('sell', 1.0),   # Sell 100%\n            1: ('sell', 0.5),   # Sell 50%\n            2: ('sell', 0.25),  # Sell 25%\n            3: ('hold', 0.0),   # Hold\n            4: ('buy', 0.25),   # Buy 25%\n            5: ('buy', 0.5),    # Buy 50%\n            6: ('buy', 1.0),    # Buy 100%\n        }\n\n        # Initialize previous price\n        self.previous_price = None\n\n    def load_next_data_chunk(self):\n        self.current_pt_file_idx = (self.current_pt_file_idx + 1) % len(self.pt_files)\n        try:\n            loaded = torch.load(self.pt_files[self.current_pt_file_idx])\n            if 'data' not in loaded or 'close_prices' not in loaded:\n                logger.error(f\"PT file {self.pt_files[self.current_pt_file_idx]} missing required keys.\")\n                raise KeyError(f\"PT file {self.pt_files[self.current_pt_file_idx]} missing required keys.\")\n            self.data = loaded['data']\n            self.close_prices = loaded['close_prices']\n            logger.info(f\"Loaded data chunk from {self.pt_files[self.current_pt_file_idx]}\")\n            logger.debug(f\"Data chunk shape: {self.data.shape}, Close prices length: {len(self.close_prices)}\")\n        except Exception as e:\n            logger.error(f\"Failed to load data chunk from {self.pt_files[self.current_pt_file_idx]}: {e}\", exc_info=True)\n            raise\n\n        # Recalculate end_step based on new data\n        self.end_step = len(self.data) - self.n_steps\n        self.current_step = 0  # Reset current_step when loading new data\n        logger.debug(f\"Data chunk loaded. Total steps: {self.end_step}\")\n\n    def reset(self):\n        if self.current_step >= self.end_step:\n            logger.debug(\"Current step exceeds end_step. Loading next data chunk.\")\n            self.load_next_data_chunk()\n\n        self.current_step = 0\n        self.position = 0.0\n        self.total_profit = 0.0\n        self.entry_price = None\n        self.previous_price = self.close_prices[self.current_step + self.n_steps - 1]\n        logger.debug(f\"Environment reset. Starting at step {self.current_step}\")\n        return self._next_observation()\n\n    def _next_observation(self):\n        obs = self.data[self.current_step:self.current_step + self.n_steps]\n        if torch.isnan(obs).any():\n            logger.error(f\"Observation contains NaN values at step {self.current_step}\")\n            raise ValueError(f\"Observation contains NaN values at step {self.current_step}\")\n        logger.debug(f\"Next observation shape: {obs.shape}\")\n        return obs.numpy().astype(np.float32)\n\n    def step(self, action):\n        try:\n            logger.debug(f\"Step {self.current_step}: Received action {action}\")\n            if action not in self.action_mappings:\n                logger.error(f\"Invalid action: {action}\")\n                raise ValueError(f\"Invalid action: {action}\")\n\n            action_type, amount = self.action_mappings[action]\n\n            idx = self.current_step + self.n_steps - 1\n            if idx >= len(self.data):\n                done = True\n                obs = np.zeros_like(self.observation_space.sample())\n                info = {'total_profit': self.total_profit}\n                logger.debug(f\"Reached end of data at step {self.current_step}. Ending episode.\")\n                return obs, 0.0, done, info\n\n            current_price = self.close_prices[idx]\n            logger.debug(f\"Current price at step {self.current_step}: {current_price}\")\n\n            # Transaction cost\n            fee_rate = self.config['environment'].get('transaction_fee', 0.001)  # 0.1%\n            transaction_cost = 0.0\n\n            # Execute action\n            if action_type == 'buy':\n                delta_position = amount\n                transaction_cost = fee_rate * delta_position * current_price\n                self.position += delta_position\n                self.total_profit -= transaction_cost\n                logger.debug(f\"Action: Buy {amount*100}%. New position: {self.position}\")\n            elif action_type == 'sell':\n                delta_position = -amount\n                transaction_cost = fee_rate * abs(delta_position) * current_price\n                self.position += delta_position\n                self.total_profit -= transaction_cost\n                logger.debug(f\"Action: Sell {abs(amount)*100}%. New position: {self.position}\")\n            # If 'hold', do nothing\n            else:\n                logger.debug(\"Action: Hold. No change in position.\")\n\n            # Calculate reward as change in portfolio value\n            price_change = current_price - self.previous_price\n            reward = self.position * price_change - transaction_cost\n            logger.debug(f\"Price change: {price_change}, Reward: {reward}, Transaction cost: {transaction_cost}\")\n\n            # Update total profit\n            self.total_profit += reward\n            logger.debug(f\"Total profit after step {self.current_step}: {self.total_profit}\")\n\n            self.previous_price = current_price\n            self.current_step += 1\n            done = self.current_step >= self.end_step or self.current_step >= self.max_steps\n\n            obs = self._next_observation() if not done else np.zeros_like(self.observation_space.sample())\n            info = {'total_profit': self.total_profit}\n\n            logger.debug(f\"Step {self.current_step}: Done={done}\")\n\n            return obs, reward, done, info\n        except Exception as e:\n            logger.error(f\"Exception in step method: {e}\", exc_info=True)\n            done = True\n            obs = np.zeros_like(self.observation_space.sample())\n            reward = 0.0\n            info = {'total_profit': self.total_profit, 'error': str(e)}\n            return obs, reward, done, info\n\n    def render(self, mode='human'):\n        # Implement rendering if needed\n        pass",
    "./training/runner.py": "# training/runner.py\n\nimport os\nimport torch\nimport torch.distributed as dist\nfrom torch.nn.parallel import DistributedDataParallel as DDP\nfrom training.train_loop import train_model\nfrom training.utils import setup_logging, setup_seed, load_checkpoint, validate_and_cast_config\nfrom environment.trading_env import TradingEnv\nfrom models.models import PPOAgent\nfrom data_processing.preprocess import export_and_save_preprocessed_data\nimport logging\nimport sys\n\n\ndef setup_runner_logging(rank, log_dir=\"logs\"):\n    \"\"\"\n    Configure logging for the runner.\n    Each rank will have its own log file.\n    \"\"\"\n    os.makedirs(log_dir, exist_ok=True)\n    logger = logging.getLogger(f\"RunnerRank{rank}\")\n    logger.setLevel(logging.DEBUG)\n\n    # Create handlers only if they haven't been created yet\n    if not logger.hasHandlers():\n        # Console handler for INFO level\n        ch = logging.StreamHandler()\n        ch.setLevel(logging.INFO)\n        # File handler for DEBUG level\n        fh = logging.FileHandler(os.path.join(log_dir, f\"runner_rank{rank}.log\"))\n        fh.setLevel(logging.DEBUG)\n\n        # Create formatter and add to handlers\n        formatter = logging.Formatter(\"%(asctime)s - %(levelname)s - %(name)s - %(message)s\")\n        ch.setFormatter(formatter)\n        fh.setFormatter(formatter)\n\n        # Add handlers to the logger\n        logger.addHandler(ch)\n        logger.addHandler(fh)\n\n    return logger\n\n\ndef train_worker(rank, config, world_size):\n    \"\"\"\n    Worker function for distributed training.\n    Initializes the environment, model, and starts the training loop.\n    \"\"\"\n    logger = setup_runner_logging(rank)\n    logger.info(f\"Rank {rank}: Starting training worker.\")\n\n    try:\n        # Set random seed for reproducibility\n        setup_seed(config.get(\"random_seed\", 42) + rank)\n        logger.debug(f\"Rank {rank}: Random seed set.\")\n\n        # Initialize the process group\n        backend = \"nccl\" if torch.cuda.is_available() else \"gloo\"\n        logger.debug(f\"Rank {rank}: Initializing process group with backend={backend}\")\n        dist.init_process_group(\n            backend=backend,\n            init_method=\"env://\",\n            world_size=world_size,\n            rank=rank\n        )\n        logger.info(f\"Rank {rank}: Process group initialized.\")\n\n        # Set device for each rank\n        if torch.cuda.is_available():\n            if rank >= torch.cuda.device_count():\n                logger.error(f\"Rank {rank}: No GPU available for this rank.\")\n                raise ValueError(f\"Rank {rank} exceeds available GPUs.\")\n            torch.cuda.set_device(rank)\n            device = torch.device(f\"cuda:{rank}\")\n            logger.debug(f\"Rank {rank}: Device set to {device}\")\n        else:\n            device = torch.device(\"cpu\")\n            logger.debug(f\"Rank {rank}: Device set to CPU\")\n\n        # Only rank 0 performs data preprocessing\n        if rank == 0 and config.get(\"preprocessing\", {}).get(\"enable\", True):\n            logger.info(\"Rank 0: Starting data preprocessing.\")\n            export_and_save_preprocessed_data(config, logger)\n        elif config.get(\"preprocessing\", {}).get(\"enable\", True):\n            logger.info(f\"Rank {rank}: Waiting for data preprocessing to complete.\")\n        else:\n            logger.info(\"Data preprocessing is disabled in the configuration.\")\n\n        # Ensure all processes wait for data preprocessing to complete\n        logger.debug(f\"Rank {rank}: Waiting at barrier after preprocessing.\")\n        dist.barrier()\n        logger.debug(f\"Rank {rank}: Passed barrier after preprocessing.\")\n\n        # Initialize environment\n        try:\n            env = TradingEnv(config)\n            logger.info(f\"Rank {rank}: Environment initialized.\")\n        except Exception as e:\n            logger.error(f\"Rank {rank}: Failed to initialize environment: {e}\", exc_info=True)\n            raise\n\n        # Initialize model\n        try:\n            model = PPOAgent(\n                feature_dim=config[\"model\"][\"feature_dim\"],\n                hidden_dim=config[\"model\"][\"ppo_hidden_dim\"],\n                action_dim=config[\"model\"][\"action_dim\"],\n                n_heads=config[\"model\"][\"ppo_n_heads\"],\n                n_layers=config[\"model\"][\"ppo_n_layers\"],\n                dropout=config[\"model\"][\"ppo_dropout\"],\n                temperature=config[\"model\"].get('temperature', 1.0),\n                layer_norm=config[\"model\"].get('layer_norm', False)\n            ).to(device)\n            logger.info(f\"Rank {rank}: PPOAgent initialized.\")\n        except Exception as e:\n            logger.error(f\"Rank {rank}: Failed to initialize PPOAgent: {e}\", exc_info=True)\n            raise\n\n        # Load checkpoint if required\n        start_epoch = 0\n        best_reward = float(\"-inf\")\n        resume_ckpt = config[\"checkpoint\"].get(\"resume_checkpoint\", \"\")\n        if resume_ckpt:\n            logger.info(f\"Rank {rank}: Resuming from checkpoint {resume_ckpt}\")\n            if os.path.exists(resume_ckpt):\n                try:\n                    epoch_loaded, best_reward_loaded = load_checkpoint(\n                        resume_ckpt,\n                        model,\n                        None,  # If you want to also load optimizer states, pass them here\n                        None,\n                        device,\n                    )\n                    start_epoch = epoch_loaded + 1\n                    best_reward = best_reward_loaded\n                    logger.info(f\"Rank {rank}: Resumed from epoch {start_epoch}, best_reward={best_reward}\")\n                except Exception as e:\n                    logger.error(f\"Rank {rank}: Failed to load checkpoint: {e}\", exc_info=True)\n                    raise\n            else:\n                logger.warning(f\"Rank {rank}: Checkpoint {resume_ckpt} not found. Starting fresh.\")\n        else:\n            logger.info(f\"Rank {rank}: No checkpoint specified. Starting from epoch 0.\")\n\n        # Wrap model in DDP\n        try:\n            if world_size > 1:\n                model = DDP(model, device_ids=[rank] if torch.cuda.is_available() else None)\n                logger.debug(f\"Rank {rank}: Model wrapped in DDP.\")\n            else:\n                logger.debug(f\"Rank {rank}: Single GPU mode. DDP not applied.\")\n        except Exception as e:\n            logger.error(f\"Rank {rank}: Failed to wrap model in DDP: {e}\", exc_info=True)\n            raise\n\n        # Start training loop\n        try:\n            logger.info(f\"Rank {rank}: Starting training loop from epoch {start_epoch + 1} to {config['training']['epochs']}\")\n            train_model(\n                env=env,\n                config=config,\n                device=device,\n                model=model,\n                rank=rank,\n                world_size=world_size,\n                start_epoch=start_epoch,\n                num_epochs=config[\"training\"][\"epochs\"],\n                best_reward=best_reward\n            )\n        except Exception as e:\n            logger.error(f\"Rank {rank}: Training failed: {e}\", exc_info=True)\n            raise\n    finally:\n        if dist.is_initialized():\n            logger.info(f\"Rank {rank}: Destroying process group.\")\n            dist.destroy_process_group()\n\ndef main():\n    import argparse\n    import yaml\n\n    # Parse command-line arguments\n    parser = argparse.ArgumentParser(description=\"Distributed PPO Training\")\n    parser.add_argument(\"--config_path\", type=str, required=True, help=\"Path to the config.yaml file\")\n    args = parser.parse_args()\n\n    # Load configuration\n    try:\n        with open(args.config_path, \"r\") as f:\n            config = yaml.safe_load(f)\n        # Validate and cast configuration\n        config = validate_and_cast_config(config)\n    except Exception as e:\n        print(f\"Failed to load or validate configuration: {e}\")\n        sys.exit(1)\n\n    # Setup logging (main logger)\n    main_logger = setup_logging(log_dir=\"logs\", log_file=\"main.log\")\n    main_logger.info(\"Main: Configuration loaded and validated.\")\n\n    # Set random seed\n    try:\n        setup_seed(config.get(\"random_seed\", 42))\n        main_logger.info(\"Main: Random seed set for reproducibility.\")\n    except Exception as e:\n        main_logger.error(f\"Main: Failed to set random seed: {e}\", exc_info=True)\n        sys.exit(1)\n\n    # Distributed setup\n    world_size = config[\"distributed\"][\"world_size\"]\n    master_addr = config[\"distributed\"][\"master_addr\"]\n    master_port = config[\"distributed\"][\"master_port\"]\n\n    os.environ['MASTER_ADDR'] = master_addr\n    os.environ['MASTER_PORT'] = master_port\n\n    if world_size > 1:\n        main_logger.info(f\"Main: Initializing Distributed Data Parallel with world_size={world_size}\")\n        try:\n            torch.multiprocessing.spawn(\n                train_worker,\n                args=(config, world_size),\n                nprocs=world_size,\n                join=True\n            )\n        except Exception as e:\n            main_logger.error(f\"Main: DDP training failed to spawn processes: {e}\", exc_info=True)\n            sys.exit(1)\n    else:\n        main_logger.info(\"Main: Running training on a single GPU without DDP.\")\n        try:\n            train_worker(0, config, world_size)\n        except Exception as e:\n            main_logger.error(f\"Main: Single GPU training failed: {e}\", exc_info=True)\n            sys.exit(1)\n\n\nif __name__ == \"__main__\":\n    main()",
    "./training/train_loop.py": "# training/train_loop.py\n\nimport os\nimport logging\nimport torch\nimport numpy as np\nimport torch.distributed as dist  # Distributed training\nfrom training.utils import (\n    compute_advantages_and_returns,\n    save_checkpoint,\n    log_memory_usage,\n    log_gpu_memory,\n)\nfrom monitoring import monitor_training, setup_tensorboard\nfrom training.evaluation import evaluate_model\nfrom torch.distributions import Categorical\nimport gc\nimport sys\nimport time\nimport datetime\nfrom torch.utils.data import DataLoader, TensorDataset, DistributedSampler\nfrom environment.trading_env import TradingEnv\n\n# Import autocast and GradScaler from torch.cuda.amp\nfrom torch.cuda.amp import autocast, GradScaler\n\n\ndef get_logger(rank):\n    logger = logging.getLogger(f\"TrainLoopRank{rank}\")\n    logger.setLevel(logging.DEBUG)  # Set to DEBUG for detailed tracing\n    if not logger.hasHandlers():\n        os.makedirs(\"logs\", exist_ok=True)\n        fh = logging.FileHandler(f\"logs/train_loop_rank{rank}.log\")\n        fh.setLevel(logging.DEBUG)\n        ch = logging.StreamHandler()\n        ch.setLevel(logging.INFO)  # Console logs set to INFO\n        formatter = logging.Formatter(\n            \"%(asctime)s - %(levelname)s - %(name)s - %(message)s\"\n        )\n        fh.setFormatter(formatter)\n        ch.setFormatter(formatter)\n        logger.addHandler(fh)\n        logger.addHandler(ch)\n    return logger\n\n\ndef collect_trajectories(env, model, device, config, num_episodes=10, logger=None):\n    \"\"\"\n    Collect trajectories by interacting with the environment.\n    \"\"\"\n    if logger is None:\n        logger = logging.getLogger(\"TrainLoop\")\n\n    model.eval()\n    trajectories = {\n        'states': [],\n        'actions': [],\n        'rewards': [],\n        'dones': [],\n        'values': [],\n        'log_probs': []\n    }\n    max_steps_per_episode = config[\"environment\"][\"max_steps\"]\n\n    try:\n        for episode in range(num_episodes):\n            logger.debug(f\"Collecting trajectory for Episode {episode + 1}\")\n            state = env.reset()\n            # No need for dist.barrier() here anymore\n            done = False\n            steps = 0\n\n            while not done and steps < max_steps_per_episode:\n                logger.debug(f\"Episode {episode + 1}, Step {steps + 1}: Collecting action.\")\n                state_tensor = torch.tensor(state, dtype=torch.float32).unsqueeze(0).to(device)\n                with torch.no_grad():\n                    if device.type == 'cuda':\n                        with autocast():\n                            logits, value = model(state_tensor)\n                    else:\n                        logits, value = model(state_tensor)\n\n                    if torch.isnan(logits).any() or torch.isinf(logits).any():\n                        logger.error(\"Logits contain NaNs or Infs during trajectory collection.\")\n                        raise ValueError(\"Invalid logits detected.\")\n\n                    action_dist = Categorical(logits=logits)\n                    action = action_dist.sample()\n                    log_prob = action_dist.log_prob(action)\n                    value = value.item()\n\n                next_state, reward, done, _ = env.step(action.item())\n\n                # Logging actions\n                logger.debug(f\"Action taken: {action.item()}, Log Prob: {log_prob.item()}, Value: {value}\")\n\n                # Store experience\n                trajectories['states'].append(state)\n                trajectories['actions'].append(action.item())\n                trajectories['rewards'].append(reward)\n                trajectories['dones'].append(done)\n                trajectories['values'].append(value)\n                trajectories['log_probs'].append(log_prob.item())\n\n                state = next_state\n                steps += 1\n                logger.debug(f\"Episode {episode + 1}, Step {steps}: State updated.\")\n\n    except Exception as e:\n        logger.error(f\"Exception during trajectory collection: {e}\", exc_info=True)\n        raise\n    finally:\n        model.train()\n\n    logger.info(f\"Collected {num_episodes} episodes with total steps: {len(trajectories['rewards'])}\")\n    return trajectories\n\n\ndef analyze_metrics(avg_metrics, eval_metrics, logger):\n    \"\"\"\n    Analyze metrics and provide insights.\n    \"\"\"\n    try:\n        # Analyze training loss\n        if 'actor_loss' in avg_metrics:\n            if avg_metrics['actor_loss'] > 1.0:\n                logger.info(\"Actor loss is high. Consider lowering the learning rate.\")\n            elif avg_metrics['actor_loss'] < 0.01:\n                logger.info(\"Actor loss is very low. Model might be converging.\")\n\n        if 'critic_loss' in avg_metrics:\n            if avg_metrics['critic_loss'] > 1.0:\n                logger.info(\"Critic loss is high. Monitor for potential overfitting.\")\n            elif avg_metrics['critic_loss'] < 0.01:\n                logger.info(\"Critic loss is very low. Model might be underfitting.\")\n\n        # Analyze entropy\n        if 'entropy' in avg_metrics:\n            if avg_metrics['entropy'] > 1.0:\n                logger.info(\"Entropy is high. Model is exploring.\")\n            elif avg_metrics['entropy'] < 0.1:\n                logger.info(\"Entropy is low. Model is exploiting.\")\n\n        # Analyze evaluation metrics\n        if eval_metrics.get('average_reward', 0) > 0:\n            logger.info(\"Model is making positive returns on evaluation.\")\n        else:\n            logger.info(\"Model is making negative returns on evaluation.\")\n\n        if 'sharpe_ratio' in eval_metrics:\n            logger.info(f\"Sharpe Ratio: {eval_metrics['sharpe_ratio']:.6f}\")\n        if 'max_drawdown' in eval_metrics:\n            logger.info(f\"Max Drawdown: {eval_metrics['max_drawdown']:.6f}\")\n        if 'average_win_rate' in eval_metrics:\n            logger.info(f\"Average Win Rate: {eval_metrics['average_win_rate']:.2%}\")\n    except Exception as e:\n        logger.error(f\"Exception during metric analysis: {e}\", exc_info=True)\n\n\ndef train_model(\n    env,\n    config,\n    device,\n    model,\n    rank,\n    world_size,\n    start_epoch,\n    num_epochs,\n    best_reward=-float(\"inf\"),\n):\n    \"\"\"\n    Main training loop for PPO with DDP.\n    \"\"\"\n    logger = get_logger(rank)\n\n    try:\n        logger.info(f\"Starting training from epoch {start_epoch + 1} to {num_epochs}\")\n        logger.info(f\"Rank {rank}: Using batch size {config['training']['batch_size']}\")\n\n        # Optimizer setup\n        optimizer_actor = torch.optim.AdamW(\n            model.module.actor.parameters(),\n            lr=config[\"training\"][\"actor_lr\"],\n            weight_decay=config[\"training\"].get(\"weight_decay\", 0.0),\n        )\n        optimizer_critic = torch.optim.AdamW(\n            model.module.critic.parameters(),\n            lr=config[\"training\"][\"critic_lr\"],\n            weight_decay=config[\"training\"].get(\"weight_decay\", 0.0),\n        )\n\n        max_grad_norm = config[\"training\"].get(\"max_grad_norm\", 0.5)\n        batch_size = config[\"training\"][\"batch_size\"]\n        ppo_epochs = config[\"training\"][\"ppo_epochs\"]\n\n        # Setup TensorBoard writer (only on rank 0)\n        writer = None\n        if config[\"logging\"][\"use_tensorboard\"] and rank == 0:\n            writer = setup_tensorboard(rank, log_dir=config['logging']['log_dir'])\n            logger.debug(\"TensorBoard writer initialized.\")\n\n        # Initialize AMP scaler\n        scaler = GradScaler()\n        logger.debug(\"AMP GradScaler initialized.\")\n\n        # Check PyTorch version for timeout support in dist.barrier()\n        torch_version = torch.__version__\n        supports_timeout = tuple(map(int, torch_version.split('+')[0].split('.'))) >= (1, 8, 0)\n\n        for epoch in range(start_epoch, num_epochs):\n            logger.info(f\"Rank {rank}, Epoch {epoch + 1}/{num_epochs}\")\n            log_memory_usage(logger, phase=f\"Epoch {epoch + 1} Start\")\n            log_gpu_memory(logger, phase=f\"Epoch {epoch + 1} Start\")\n\n            # Reset environment for new epoch\n            logger.debug(f\"Rank {rank}: Resetting environment for new epoch.\")\n            state = env.reset()\n\n            # Collect trajectories\n            trajectories = collect_trajectories(env, model, device, config, num_episodes=10, logger=logger)\n            logger.info(f\"Rank {rank}: Collected {len(trajectories['rewards'])} steps.\")\n            logger.debug(\"After collect_trajectories\")\n\n            # Convert collected data to tensors\n            try:\n                states = torch.stack([torch.from_numpy(np.array(s)) for s in trajectories['states']]).float().to(device)\n                actions = torch.tensor(trajectories['actions'], dtype=torch.long, device=device)\n                rewards = torch.tensor(trajectories['rewards'], dtype=torch.float32, device=device)\n                dones = torch.tensor(trajectories['dones'], dtype=torch.float32, device=device)\n                values = torch.tensor(trajectories['values'], dtype=torch.float32, device=device)\n                log_probs_old = torch.tensor(trajectories['log_probs'], dtype=torch.float32, device=device)\n                logger.debug(\"Converted collected trajectories to tensors.\")\n            except Exception as e:\n                logger.error(f\"Exception during tensor conversion: {e}\", exc_info=True)\n                raise\n\n            # Compute advantages and returns\n            try:\n                advantages, returns = compute_advantages_and_returns(\n                    rewards=rewards,\n                    values=values,\n                    dones=dones,\n                    gamma=config[\"training\"][\"gamma\"],\n                    gae_lambda=config[\"training\"][\"gae_lambda\"]\n                )\n                # Normalize advantages\n                advantages = (advantages - advantages.mean()) / (advantages.std() + 1e-8)\n                logger.debug(\"Computed advantages and returns.\")\n            except Exception as e:\n                logger.error(f\"Exception during advantage and return computation: {e}\", exc_info=True)\n                raise\n\n            # Create a TensorDataset from the collected trajectories\n            dataset = TensorDataset(states, actions, returns, advantages, log_probs_old)\n\n            # Create a DistributedSampler to distribute the data across processes\n            sampler = DistributedSampler(dataset, num_replicas=world_size, rank=rank, shuffle=True)\n\n            # Create a DataLoader to handle batching and shuffling\n            dataloader = DataLoader(dataset, batch_size=batch_size, sampler=sampler)\n\n            # PPO Update\n            for ppo_epoch in range(ppo_epochs):\n                try:\n                    logger.debug(f\"Rank {rank}: Starting PPO Epoch {ppo_epoch + 1}\")\n\n                    for batch_idx, (batch_states, batch_actions, batch_returns, batch_advantages, batch_log_probs_old) in enumerate(dataloader):\n                        # Move batch data to the correct device\n                        batch_states = batch_states.to(device)\n                        batch_actions = batch_actions.to(device)\n                        batch_returns = batch_returns.to(device)\n                        batch_advantages = batch_advantages.to(device)\n                        batch_log_probs_old = batch_log_probs_old.to(device)\n                        \n                        optimizer_actor.zero_grad()\n                        optimizer_critic.zero_grad()\n\n                        if device.type == 'cuda':\n                            with autocast():\n                                logits, values_pred = model(batch_states)\n                                if torch.isnan(logits).any() or torch.isinf(logits).any():\n                                    logger.error(\"Logits contain NaNs or Infs during PPO update.\")\n                                    raise ValueError(\"Invalid logits detected during PPO update.\")\n                                action_dist = Categorical(logits=logits)\n                                log_probs = action_dist.log_prob(batch_actions)\n                                entropy = action_dist.entropy().mean()\n\n                                # Calculate ratios\n                                ratios = torch.exp(log_probs - batch_log_probs_old)\n\n                                # Surrogate loss\n                                surr1 = ratios * batch_advantages\n                                surr2 = torch.clamp(\n                                    ratios,\n                                    1.0 - config[\"training\"][\"clip_epsilon\"],\n                                    1.0 + config[\"training\"][\"clip_epsilon\"]\n                                ) * batch_advantages\n                                actor_loss = -torch.min(surr1, surr2).mean() - config[\"training\"][\"entropy_coeff\"] * entropy\n\n                                # Critic loss\n                                critic_loss = torch.nn.functional.mse_loss(\n                                    values_pred.squeeze(-1),\n                                    batch_returns\n                                )\n                            logger.debug(\"Forward pass with autocast completed.\")\n\n                            # Backward pass with AMP\n                            scaler.scale(actor_loss).backward()\n                            scaler.unscale_(optimizer_actor)\n                            torch.nn.utils.clip_grad_norm_(model.module.actor.parameters(), max_grad_norm)\n                            scaler.step(optimizer_actor)\n\n                            scaler.scale(critic_loss).backward()\n                            scaler.unscale_(optimizer_critic)\n                            torch.nn.utils.clip_grad_norm_(model.module.critic.parameters(), max_grad_norm)\n                            scaler.step(optimizer_critic)\n\n                            scaler.update()\n                            logger.debug(\"Backward pass and optimizer steps completed with AMP.\")\n                        else:\n                            # Forward pass without autocast\n                            logits, values_pred = model(batch_states)\n                            if torch.isnan(logits).any() or torch.isinf(logits).any():\n                                logger.error(\"Logits contain NaNs or Infs during PPO update.\")\n                                raise ValueError(\"Invalid logits detected during PPO update.\")\n                            action_dist = Categorical(logits=logits)\n                            log_probs = action_dist.log_prob(batch_actions)\n                            entropy = action_dist.entropy().mean()\n\n                            # Calculate ratios\n                            ratios = torch.exp(log_probs - batch_log_probs_old)\n\n                            # Surrogate loss\n                            surr1 = ratios * batch_advantages\n                            surr2 = torch.clamp(\n                                ratios,\n                                1.0 - config[\"training\"][\"clip_epsilon\"],\n                                1.0 + config[\"training\"][\"clip_epsilon\"]\n                            ) * batch_advantages\n                            actor_loss = -torch.min(surr1, surr2).mean() - config[\"training\"][\"entropy_coeff\"] * entropy\n\n                            # Critic loss\n                            critic_loss = torch.nn.functional.mse_loss(\n                                values_pred.squeeze(-1),\n                                batch_returns\n                            )\n\n                            # Backward pass\n                            actor_loss.backward()\n                            torch.nn.utils.clip_grad_norm_(model.module.actor.parameters(), max_grad_norm)\n                            optimizer_actor.step()\n\n                            critic_loss.backward()\n                            torch.nn.utils.clip_grad_norm_(model.module.critic.parameters(), max_grad_norm)\n                            optimizer_critic.step()\n                            logger.debug(\"Backward pass and optimizer steps completed without AMP.\")\n\n                        # Logging\n                        logger.info(\n                            f\"Rank {rank}, Epoch {epoch + 1}, PPO Epoch {ppo_epoch + 1}, \"\n                            f\"Actor Loss: {actor_loss.item():.6f}, Critic Loss: {critic_loss.item():.6f}, \"\n                            f\"Entropy: {entropy.item():.6f}\"\n                        )\n\n                except Exception as e:\n                    logger.error(f\"Exception during PPO epoch {ppo_epoch + 1}: {e}\", exc_info=True)\n                    raise\n\n            # Reduce metrics across all processes\n            try:\n                avg_reward = rewards.mean()\n                dist.all_reduce(avg_reward, op=dist.ReduceOp.SUM)\n                avg_reward = avg_reward.item() / world_size\n                logger.info(f\"Rank {rank}, Epoch {epoch + 1} - Average Reward: {avg_reward:.6f}\")\n            except Exception as e:\n                logger.error(f\"Exception during metric reduction: {e}\", exc_info=True)\n                raise\n\n            # Only rank 0 performs evaluation and checkpointing\n            if rank == 0:\n                try:\n                    # Ensure all processes reach this point but only rank 0 runs this block\n                    logger.debug(\"Rank 0: Waiting at barrier before evaluation and checkpointing.\")\n                    dist.barrier()\n                    logger.debug(\"Rank 0: Passed barrier before evaluation and checkpointing.\")\n\n                    # Evaluation\n                    if (epoch + 1) % config['evaluation']['interval_eval'] == 0:\n                        eval_env = TradingEnv(config)  # Create separate environment for evaluation\n                        logger.info(\"Rank 0: Starting model evaluation.\")\n                        eval_metrics = evaluate_model(model.module.actor, eval_env, config, device)\n                        logger.info(\"Rank 0: Model evaluation completed.\")\n                    else:\n                        eval_metrics = {}\n\n                    # Monitoring\n                    avg_metrics = {\n                        \"actor_loss\": actor_loss.item(),\n                        \"critic_loss\": critic_loss.item(),\n                        \"entropy\": entropy.item()\n                    }\n                    monitor_training(epoch, rank, avg_metrics, eval_metrics, writer)\n                    logger.debug(\"Monitoring metrics completed.\")\n\n                    # Analyze metrics and provide insights\n                    analyze_metrics(avg_metrics, eval_metrics, logger)\n\n                    # Checkpointing\n                    if avg_reward > best_reward:\n                        best_reward = avg_reward\n                        checkpoint_path = config[\"checkpoint\"][\"best_model_path\"]\n                        save_checkpoint(\n                            checkpoint_path,\n                            model.module,\n                            optimizer_actor,\n                            optimizer_critic,\n                            epoch,\n                            best_reward\n                        )\n                        logger.info(f\"New best model saved at {checkpoint_path} with reward {best_reward:.6f}\")\n                except Exception as e:\n                    logger.error(f\"Exception during evaluation or checkpointing: {e}\", exc_info=True)\n                    raise\n            else:\n                logger.debug(f\"Rank {rank}: Waiting for evaluation and checkpointing.\")\n                dist.barrier()\n                logger.debug(f\"Rank {rank}: Finished waiting for evaluation and checkpointing.\")\n\n            # Log resource usage at the end of the epoch\n            try:\n                log_memory_usage(logger, phase=f\"Epoch {epoch + 1} End\")\n                log_gpu_memory(logger, phase=f\"Epoch {epoch + 1} End\")\n                logger.debug(f\"Rank {rank}: Resource usage logged at epoch end.\")\n            except Exception as e:\n                logger.error(f\"Exception during resource logging: {e}\", exc_info=True)\n\n            # Synchronize processes at the end of each epoch\n            try:\n                # Adding logs around the barrier\n                logger.debug(f\"Rank {rank}: About to enter barrier (end of epoch)\")\n                dist.barrier()\n                logger.debug(f\"Rank {rank}: Passed barrier (end of epoch).\")\n            except Exception as e:\n                logger.error(f\"Exception during end-of-epoch barrier: {e}\", exc_info=True)\n                raise\n\n            # Resource cleanup\n            try:\n                if torch.cuda.is_available():\n                    torch.cuda.empty_cache()\n                    logger.debug(\"GPU cache emptied.\")\n                gc.collect()\n                logger.debug(\"Garbage collection completed.\")\n            except Exception as e:\n                logger.error(f\"Exception during resource cleanup: {e}\", exc_info=True)\n\n        if writer:\n            writer.close()\n            logger.debug(\"TensorBoard writer closed.\")\n        logger.info(f\"Rank {rank}: Training completed.\")\n\n    except Exception as e:\n        logger.error(f\"Training failed on Rank {rank} with exception: {e}\", exc_info=True)\n        # Ensure all processes exit if one fails\n        if dist.is_initialized():\n            dist.destroy_process_group()\n            logger.debug(\"Process group destroyed due to training failure.\")\n        sys.exit(1)",
    "./training/utils.py": "# File: training/utils.py\n\nimport os\nimport torch\nimport logging\nimport numpy as np\nimport random\nfrom typing import List, Tuple, Dict\nimport psutil\nimport gc\nfrom pydantic import BaseModel, ValidationError\n\n# Initialize logger\nlogger = logging.getLogger(\"Utils\")\nlogger.setLevel(logging.INFO)\nif not logger.hasHandlers():\n    os.makedirs(\"logs\", exist_ok=True)\n    fh = logging.FileHandler(\"logs/utils.log\")\n    fh.setLevel(logging.INFO)\n    ch = logging.StreamHandler()\n    ch.setLevel(logging.INFO)\n    formatter = logging.Formatter(\"%(asctime)s - %(levelname)s - %(name)s - %(message)s\")\n    fh.setFormatter(formatter)\n    ch.setFormatter(formatter)\n    logger.addHandler(fh)\n    logger.addHandler(ch)\n\ndef setup_logging(name, log_file=\"utils.log\"):\n    \"\"\"\n    Configure logging for utilities.\n    \"\"\"\n    return logger  # Logger is already set up\n\ndef setup_seed(seed):\n    \"\"\"\n    Set the random seed for reproducibility.\n    \"\"\"\n    torch.manual_seed(seed)\n    np.random.seed(seed)\n    random.seed(seed)\n    os.environ[\"PYTHONHASHSEED\"] = str(seed)\n    if torch.cuda.is_available():\n        torch.cuda.manual_seed_all(seed)\n        torch.backends.cudnn.deterministic = True\n        torch.backends.cudnn.benchmark = False\n    logger.info(f\"Random seed set to {seed}\")\n\ndef load_checkpoint(checkpoint_path, model, optimizer_actor, optimizer_critic, device):\n    logger.info(f\"Attempting to load checkpoint from {checkpoint_path}\")\n\n    if os.path.isfile(checkpoint_path):\n        try:\n            checkpoint = torch.load(checkpoint_path, map_location=device)\n            model.load_state_dict(checkpoint[\"model_state_dict\"])\n            if optimizer_actor and optimizer_critic:\n                optimizer_actor.load_state_dict(checkpoint[\"optimizer_actor_state_dict\"])\n                optimizer_critic.load_state_dict(checkpoint[\"optimizer_critic_state_dict\"])\n            epoch = checkpoint[\"epoch\"]\n            best_reward = checkpoint.get(\"best_reward\", float(\"-inf\"))\n            logger.info(f\"Checkpoint loaded successfully from {checkpoint_path}, resuming at epoch {epoch}.\")\n            return epoch, best_reward\n        except Exception as e:\n            logger.error(f\"Error loading checkpoint '{checkpoint_path}': {e}\", exc_info=True)\n            raise\n    else:\n        logger.warning(f\"No checkpoint found at {checkpoint_path}.\")\n        return 0, float(\"-inf\")\n\ndef save_checkpoint(checkpoint_path, model, optimizer_actor, optimizer_critic, epoch, best_reward):\n    \"\"\"\n    Saves a checkpoint of the model and optimizer states.\n    \"\"\"\n    try:\n        os.makedirs(os.path.dirname(checkpoint_path), exist_ok=True)\n        torch.save(\n            {\n                \"model_state_dict\": model.state_dict(),\n                \"optimizer_actor_state_dict\": optimizer_actor.state_dict() if optimizer_actor else None,\n                \"optimizer_critic_state_dict\": optimizer_critic.state_dict() if optimizer_critic else None,\n                \"epoch\": epoch,\n                \"best_reward\": best_reward,\n            },\n            checkpoint_path,\n        )\n        logger.info(f\"Checkpoint saved successfully at {checkpoint_path}.\")\n    except Exception as e:\n        logger.error(f\"Failed to save checkpoint: {e}\", exc_info=True)\n\n# Define Nested Pydantic Models\nclass DataConfig(BaseModel):\n    raw_data_path: str\n    feature_columns: List[str]\n    n_steps: int\n    reward_scale: float\n    processed_data_path: str = \"data/processed/processed_data.csv\"\n    normalize_features: bool = True\n    pt_data_dir: str\n    num_splits: int\n\nclass ModelConfig(BaseModel):\n    action_dim: int\n    feature_dim: int\n    ppo_hidden_dim: int\n    ppo_n_heads: int\n    ppo_n_layers: int\n    ppo_dropout: float\n    temperature: float = 1.0\n    layer_norm: bool = False\n\nclass TrainingConfig(BaseModel):\n    batch_size: int\n    epochs: int\n    actor_lr: float\n    critic_lr: float\n    clip_epsilon: float\n    gamma: float\n    gae_lambda: float\n    entropy_coeff: float\n    critic_coeff: float\n    ppo_epochs: int\n    weight_decay: float = 0.0\n    max_grad_norm: float = 0.5\n\nclass CheckpointConfig(BaseModel):\n    best_model_path: str\n    resume_checkpoint: str = \"\"\n\nclass EvaluationConfig(BaseModel):\n    interval_eval: int\n    episodes: int\n    baseline_reward: float\n    gamma: float = 0.99\n\nclass LoggingConfig(BaseModel):\n    log_dir: str\n    use_tensorboard: bool\n\nclass DistributedConfig(BaseModel):\n    master_addr: str\n    master_port: int           # Corrected: Defined as an integer\n    world_size: int = 1\n\nclass PreprocessingConfig(BaseModel):\n    enable: bool\n\nclass EnvironmentConfig(BaseModel):\n    max_steps: int\n    transaction_fee: float = 0.001  # Added default transaction fee\n\nclass ConfigModel(BaseModel):\n    random_seed: int\n    data: DataConfig\n    model: ModelConfig\n    training: TrainingConfig\n    checkpoint: CheckpointConfig\n    evaluation: EvaluationConfig\n    logging: LoggingConfig\n    distributed: DistributedConfig\n    preprocessing: PreprocessingConfig\n    environment: EnvironmentConfig\n\ndef validate_and_cast_config(config):\n    \"\"\"\n    Validate and cast the configuration dictionary to ensure all required keys\n    are present and of correct types.\n    \"\"\"\n    logger.info(\"Validating configuration parameters.\")\n\n    try:\n        validated_config = ConfigModel(**config)\n        logger.info(\"Configuration validated successfully.\")\n        return validated_config.dict()\n    except ValidationError as e:\n        logger.error(f\"Configuration validation failed: {e}\", exc_info=True)\n        raise\n\ndef compute_advantages_and_returns(rewards, values, dones, gamma, gae_lambda, reward_scale=1.0):\n    \"\"\"\n    Compute Generalized Advantage Estimation (GAE) and returns.\n\n    Args:\n        rewards (Tensor): Tensor of rewards.\n        values (Tensor): Tensor of value predictions.\n        dones (Tensor): Tensor of done flags.\n        gamma (float): Discount factor for future rewards.\n        gae_lambda (float): GAE lambda for smoothing.\n        reward_scale (float, optional): Scaling factor for rewards.\n\n    Returns:\n        Tuple[Tensor, Tensor]: Advantages and returns tensors.\n    \"\"\"\n    rewards = rewards * reward_scale\n    advantages = torch.zeros_like(rewards)\n    returns = torch.zeros_like(rewards)\n    gae = 0\n    next_value = 0\n\n    for t in reversed(range(len(rewards))):\n        mask = 1.0 - dones[t]\n        delta = rewards[t] + gamma * next_value * mask - values[t]\n        gae = delta + gamma * gae_lambda * mask * gae\n        advantages[t] = gae\n        next_value = values[t]\n\n    returns = advantages + values\n    return advantages, returns\n\ndef log_memory_usage(logger, phase=\"\"):\n    \"\"\"\n    Logs the current CPU and memory usage.\n    \"\"\"\n    try:\n        process = psutil.Process(os.getpid())\n        mem_info = process.memory_info()\n        mem_usage_mb = mem_info.rss / (1024 ** 2)  # Convert bytes to MB\n        cpu_percent = psutil.cpu_percent(interval=1)\n        logger.info(f\"Resource Usage [{phase}]: CPU: {cpu_percent}% | Memory: {mem_usage_mb:.2f} MB\")\n    except Exception as e:\n        logger.error(f\"Failed to log memory usage: {e}\", exc_info=True)\n\ndef log_gpu_memory(logger, phase=\"\"):\n    \"\"\"\n    Logs the current GPU memory usage.\n    \"\"\"\n    try:\n        if torch.cuda.is_available():\n            allocated = torch.cuda.memory_allocated() / (1024 ** 2)  # in MB\n            reserved = torch.cuda.memory_reserved() / (1024 ** 2)     # in MB\n            logger.info(f\"GPU Memory Usage [{phase}]: Allocated: {allocated:.2f} MB | Reserved: {reserved:.2f} MB\")\n        else:\n            logger.info(f\"GPU Memory Usage [{phase}]: GPU not available.\")\n    except Exception as e:\n        logger.error(f\"Failed to log GPU memory usage: {e}\", exc_info=True)",
    "./training/__init__.py": "",
    "./training/evaluation.py": "# training/evaluation.py\n\nimport torch\nimport numpy as np\nimport logging\nfrom collections import Counter\nfrom training.utils import log_memory_usage, log_gpu_memory\nimport gc\nimport os\nimport glob\n\ndef setup_logging(log_dir=\"logs\", log_file=\"evaluation.log\"):\n    \"\"\"\n    Configure logging for the evaluation script.\n    \"\"\"\n    os.makedirs(log_dir, exist_ok=True)\n    logger = logging.getLogger(\"Evaluation\")\n    logger.setLevel(logging.DEBUG)\n\n    # Create console handler\n    ch = logging.StreamHandler()\n    ch.setLevel(logging.INFO)\n\n    # Create file handler\n    fh = logging.FileHandler(os.path.join(log_dir, log_file))\n    fh.setLevel(logging.DEBUG)\n\n    # Create formatter and add to handlers\n    formatter = logging.Formatter(\"%(asctime)s - %(levelname)s - %(name)s - %(message)s\")\n    ch.setFormatter(formatter)\n    fh.setFormatter(formatter)\n\n    # Add handlers to the logger if none exist\n    if not logger.hasHandlers():\n        logger.addHandler(ch)\n        logger.addHandler(fh)\n\n    return logger\n\ndef evaluate_model(actor, environment, config, device):\n    \"\"\"\n    Evaluate the model with comprehensive financial metrics.\n\n    Args:\n        actor: Actor network for policy evaluation.\n        environment: Trading environment.\n        config: Full configuration dictionary.\n        device: Computation device (e.g., 'cpu' or 'cuda').\n\n    Returns:\n        Dictionary containing evaluation metrics.\n    \"\"\"\n    logger = logging.getLogger(\"Evaluation\")\n    if not logger.hasHandlers():\n        logger = setup_logging()\n\n    logger.info(\"Starting model evaluation.\")\n    log_memory_usage(logger, phase=\"Before Evaluation\")\n    log_gpu_memory(logger, phase=\"Before Evaluation\")\n\n    actor.eval()  # Set actor to evaluation mode\n    total_rewards = []\n    sharpe_ratios = []\n    max_drawdowns = []\n    trade_counts = []\n    win_rates = []\n\n    num_episodes = config[\"evaluation\"][\"episodes\"]\n    gamma = config[\"training\"].get(\"gamma\", 0.99)  # Discount factor used in returns\n    reward_scale = config['data'].get('reward_scale', 1.0)\n\n    # Ensure 'close' is in feature columns\n    if 'close' not in config['data']['feature_columns']:\n        logger.error(\"'close' column is missing from feature_columns.\")\n        raise ValueError(\"'close' column is required in feature_columns for evaluation.\")\n\n    # Define action labels matching the environment's action mappings\n    action_labels = {\n        0: 'Sell 100%',\n        1: 'Sell 50%',\n        2: 'Sell 25%',\n        3: 'Hold',\n        4: 'Buy 25%',\n        5: 'Buy 50%',\n        6: 'Buy 100%',\n    }\n\n    for episode in range(num_episodes):\n        try:\n            logger.debug(f\"Starting Evaluation Episode {episode + 1}\")\n            state = environment.reset()\n            episode_reward = 0.0\n            cumulative_returns = 0.0\n            peak = 0.0\n            done = False\n            steps = 0\n            max_steps_per_episode = config[\"environment\"][\"max_steps\"]\n            action_counts = Counter()\n            episode_returns = []\n\n            while not done and steps < max_steps_per_episode:\n                logger.debug(f\"Evaluation Episode {episode + 1}, Step {steps + 1}: Selecting action.\")\n                state_tensor = torch.tensor(state, dtype=torch.float32).unsqueeze(0).to(device)\n                with torch.no_grad():\n                    logits = actor(state_tensor)\n                    action = torch.argmax(logits, dim=-1).item()\n\n                logger.debug(f\"Evaluation Episode {episode + 1}, Step {steps + 1}: Action selected: {action}\")\n\n                step_result = environment.step(action)\n                next_state, reward, done, info = step_result\n\n                scaled_reward = reward * reward_scale\n                cumulative_returns += scaled_reward\n                episode_reward += scaled_reward\n                peak = max(peak, cumulative_returns)\n                episode_returns.append(scaled_reward)\n\n                # Map action index to label\n                action_label = action_labels.get(action, \"Unknown\")\n                action_counts[action_label] += 1\n\n                logger.debug(f\"Evaluation Episode {episode + 1}, Step {steps + 1}: Reward: {reward}, Scaled Reward: {scaled_reward}\")\n                steps += 1\n                state = next_state\n\n            total_rewards.append(episode_reward)\n\n            # Compute Sharpe ratio\n            avg_return = np.mean(episode_returns) if len(episode_returns) > 0 else 0\n            std_dev_return = np.std(episode_returns) + 1e-8\n            # Annualized Sharpe ratio (assuming 252 trading days)\n            sharpe_ratio = (avg_return / std_dev_return) * np.sqrt(252)\n            sharpe_ratios.append(sharpe_ratio)\n\n            # Compute max drawdown\n            max_drawdown = peak - cumulative_returns\n            max_drawdowns.append(max_drawdown)\n\n            # Compute win rate\n            wins = sum(1 for r in episode_returns if r > 0)\n            total_trades = len(episode_returns)\n            win_rate = wins / total_trades if total_trades > 0 else 0\n            win_rates.append(win_rate)\n\n            trade_counts.append(total_trades)\n            logger.info(\n                f\"Evaluation Episode {episode + 1}: Reward={episode_reward:.6f}, Trades={total_trades}, \"\n                f\"Sharpe Ratio={sharpe_ratio:.6f}, Max Drawdown={max_drawdown:.6f}, Win Rate={win_rate:.2%}\"\n            )\n            logger.info(f\"Evaluation Episode {episode + 1}: Action counts: {action_counts}\")\n\n            # Resource cleanup\n            del state_tensor\n            if torch.cuda.is_available():\n                torch.cuda.empty_cache()\n            gc.collect()\n\n        except Exception as e:\n            logger.error(f\"Exception during episode {episode + 1} evaluation: {e}\", exc_info=True)\n            continue  # Proceed to next episode\n\n    # Aggregate metrics\n    try:\n        avg_reward = np.mean(total_rewards) if total_rewards else 0\n        avg_sharpe_ratio = np.mean(sharpe_ratios) if sharpe_ratios else 0\n        avg_max_drawdown = np.mean(max_drawdowns) if max_drawdowns else 0\n        avg_trade_count = np.mean(trade_counts) if trade_counts else 0\n        avg_win_rate = np.mean(win_rates) if win_rates else 0\n\n        metrics = {\n            \"average_reward\": avg_reward,\n            \"sharpe_ratio\": avg_sharpe_ratio,\n            \"max_drawdown\": avg_max_drawdown,\n            \"average_trade_count\": avg_trade_count,\n            \"average_win_rate\": avg_win_rate,\n        }\n\n        logger.info(f\"Evaluation Metrics: {metrics}\")\n        log_memory_usage(logger, phase=\"After Evaluation\")\n        log_gpu_memory(logger, phase=\"After Evaluation\")\n        return metrics\n\n    except Exception as e:\n        logger.error(f\"Exception during metric aggregation: {e}\", exc_info=True)\n        raise",
    "./config/config.yaml": "# config/config.yaml\n\n# =====================\n# Random Seed for Reproducibility\n# =====================\nrandom_seed: 42\n\n# =====================\n# Logging Configuration\n# =====================\nlogging:\n  log_dir: \"./logs\"            # Directory to store log files\n  use_tensorboard: true        # Whether to use TensorBoard\n\n# =====================\n# Distributed Training Configuration\n# =====================\ndistributed:\n  master_addr: \"localhost\"     # Master node address\n  master_port: \"29500\"         # Master node port\n  world_size: 2                # Number of GPUs to use for training\n\n# =====================\n# Data Configuration\n# =====================\ndata:\n  raw_data_path: \"./data/dataset.csv\"          # Path to the raw data CSV file\n  feature_columns:\n    - open\n    - high\n    - low\n    - close\n    - volume\n    - elapsed_time\n    - rsi\n    - adx\n    - roc\n    - stochastic_k\n    - stochastic_d\n    - macd\n    - macd_signal\n    - macd_hist\n    - mfi\n    - bb_upper\n    - bb_middle\n    - bb_lower\n    - kc_upper\n    - kc_lower\n    - dc_upper\n    - dc_lower\n    - atr\n    - pivot\n    - fibonacci_23\n    - fibonacci_38\n    - fibonacci_50\n    - fibonacci_61\n    - vpt\n    - cmf\n    - obv\n    - supertrend\n  n_steps: 20                                    # Number of steps to consider in each sample\n  reward_scale: 1.0                              # Scaling factor for rewards\n  pt_data_dir: \"./data/processed/\"              # Directory to store processed PT files\n  num_splits: 2                                  # Number of splits for processed data\n  normalize_features: true                       # Whether to normalize features\n\n# =====================\n# Model Configuration\n# =====================\nmodel:\n  feature_dim: 32                # Number of input features\n  ppo_hidden_dim: 256            # Hidden size of the model layers\n  action_dim: 7                  # Number of possible actions (Sell 100%, Sell 50%, Sell 25%, Hold, Buy 25%, Buy 50%, Buy 100%)\n  ppo_n_heads: 4                 # Number of attention heads in the Transformer\n  ppo_n_layers: 3                # Number of Transformer encoder layers\n  ppo_dropout: 0.1               # Dropout rate for regularization\n  temperature: 1.0               # Temperature parameter for action selection\n  layer_norm: true               # Whether to apply layer normalization\n\n# =====================\n# Training Configuration\n# =====================\ntraining:\n  epochs: 100\n  batch_size: 64                 # Reduced batch size for scalping\n  actor_lr: 1e-5                 # Lower learning rates\n  critic_lr: 1e-6\n  ppo_epochs: 4\n  clip_epsilon: 0.1              # Smaller clipping range\n  entropy_coeff: 0.005           # Adjusted entropy coefficient\n  critic_coeff: 0.5\n  gamma: 0.99\n  gae_lambda: 0.95\n  weight_decay: 0.0\n  max_grad_norm: 0.5\n\n# =====================\n# Checkpointing Configuration\n# =====================\ncheckpoint:\n  best_model_path: \"./checkpoints/best_model.pth\"  # Path to save the best model\n  resume_checkpoint: \"\"                            # Path to resume training from (leave empty if not resuming)\n\n# =====================\n# Evaluation Configuration\n# =====================\nevaluation:\n  interval_eval: 5                # Frequency of evaluation (in epochs)\n  episodes: 20                    # Number of episodes during evaluation\n  baseline_reward: 0.0            # Baseline reward for comparison\n  gamma: 0.99                     # Discount factor used during evaluation\n\n# =====================\n# Environment Configuration\n# =====================\nenvironment:\n  max_steps: 200                   # Shorter episodes for scalping\n  transaction_fee: 0.001           # Transaction fee rate (0.1%)\n\n# =====================\n# Preprocessing Configuration\n# =====================\npreprocessing:\n  enable: true                     # Whether to enable data preprocessing",
    "./models/models.py": "# models/models.py\n\nimport os\nimport torch\nimport torch.nn as nn\nfrom models.transformer import TransformerEncoder\nimport logging\nimport torch.nn.init as init\n\ndef setup_models_logging(log_dir=\"logs\", log_file=\"models.log\", file_logging_enabled=False):\n    \"\"\"\n    Configure logging for the models module with an on/off switch for file logging.\n\n    Args:\n        log_dir (str): Directory to store log files.\n        log_file (str): Log file name.\n        file_logging_enabled (bool): Flag to enable/disable file logging.\n    \"\"\"\n    os.makedirs(log_dir, exist_ok=True)\n    logger = logging.getLogger(\"Models\")\n    logger.setLevel(logging.DEBUG)\n\n    # Create console handler\n    ch = logging.StreamHandler()\n    ch.setLevel(logging.INFO)\n\n    # Create formatter and add to handlers\n    formatter = logging.Formatter(\"%(asctime)s - %(levelname)s - %(name)s - %(message)s\")\n    ch.setFormatter(formatter)\n\n    # Add handlers to the logger if none exist\n    if not logger.hasHandlers():\n        logger.addHandler(ch)\n\n        # Add file handler based on the flag\n        if file_logging_enabled:\n            fh = logging.FileHandler(os.path.join(log_dir, log_file))\n            fh.setLevel(logging.DEBUG)\n            fh.setFormatter(formatter)\n            logger.addHandler(fh)\n        else:\n            # Add a NullHandler to effectively disable file logging\n            fh = logging.NullHandler()\n            logger.addHandler(fh)\n\n    return logger\n\n# Initialize logger for Models\ndef initialize_models_logging():\n    logger = logging.getLogger(\"Models\")\n    if not logger.hasHandlers():\n        logger = setup_models_logging()\n    return logger\n\nlogger = initialize_models_logging()\n\ndef initialize_weights(m):\n    \"\"\"\n    Initialize weights of the model.\n    Uses He (Kaiming) initialization for Linear layers assuming they are followed by LeakyReLU activations.\n    Uses default initialization for LayerNorm layers.\n    \"\"\"\n    if isinstance(m, nn.Linear):\n        # Initialize weights using Kaiming Uniform for layers followed by LeakyReLU\n        init.kaiming_uniform_(m.weight, nonlinearity='leaky_relu')\n        if m.bias is not None:\n            init.constant_(m.bias, 0)\n        logger.debug(f\"Initialized Linear layer with Kaiming Uniform.\")\n    elif isinstance(m, nn.LayerNorm):\n        # Initialize LayerNorm weights to 1 and biases to 0\n        init.constant_(m.weight, 1)\n        init.constant_(m.bias, 0)\n        logger.debug(f\"Initialized LayerNorm layer with constant weights and biases.\")\n\n    # Check for NaN/Inf in weights\n    if hasattr(m, 'weight') and (torch.isnan(m.weight).any() or torch.isinf(m.weight).any()):\n        logger.error(f\"NaN/Inf detected in weights of {type(m).__name__}\")\n        raise ValueError(f\"NaN/Inf detected in weights of {type(m).__name__}\")\n    # Check for NaN/Inf in biases\n    if hasattr(m, 'bias') and m.bias is not None and (torch.isnan(m.bias).any() or torch.isinf(m.bias).any()):\n        logger.error(f\"NaN/Inf detected in bias of {type(m).__name__}\")\n        raise ValueError(f\"NaN/Inf detected in bias of {type(m).__name__}\")\n\nclass Actor(nn.Module):\n    def __init__(self, feature_dim, hidden_dim, action_dim, n_heads, n_layers, temperature=1.0, layer_norm=False):\n        super().__init__()\n        self.encoder = TransformerEncoder(feature_dim, hidden_dim, n_heads, n_layers, dropout=0.1)\n        self.layer_norm_flag = layer_norm\n        if self.layer_norm_flag:\n            self.layer_norm = nn.LayerNorm(hidden_dim)\n            logger.debug(\"Actor: LayerNorm initialized.\")\n\n        self.fc = nn.Linear(hidden_dim, action_dim)  # Output logits for actions\n        self.temperature = temperature\n        logger.debug(f\"Actor initialized with feature_dim={feature_dim}, hidden_dim={hidden_dim}, action_dim={action_dim}\")\n\n    def forward(self, x):\n        logger.debug(f\"Actor forward pass: input shape {x.shape}\")\n        x = self.encoder(x)\n        logger.debug(f\"Actor forward pass: after encoder, shape {x.shape}\")\n        x = x[:, -1]  # Get the output from the last time step\n        logger.debug(f\"Actor forward pass: after slicing, shape {x.shape}\")\n        if self.layer_norm_flag:\n            x = self.layer_norm(x)\n            logger.debug(\"Actor forward pass: LayerNorm applied.\")\n        logits = self.fc(x)\n        logger.debug(f\"Actor forward pass: logits shape {logits.shape}\")\n        if torch.isnan(logits).any():\n            logger.error(\"Logits contain NaNs in Actor forward pass\")\n            raise ValueError(\"Logits contain NaNs in Actor\")\n        if self.temperature != 1.0:\n            logits = logits / self.temperature  # Apply temperature scaling\n            logger.debug(f\"Actor forward pass: Temperature scaling applied with temperature={self.temperature}\")\n\n        return logits\n\nclass Critic(nn.Module):\n    def __init__(self, feature_dim, hidden_dim, n_heads, n_layers, layer_norm=False):\n        super().__init__()\n        self.encoder = TransformerEncoder(feature_dim, hidden_dim, n_heads, n_layers, dropout=0.1)\n        self.layer_norm_flag = layer_norm\n        if self.layer_norm_flag:\n            self.layer_norm = nn.LayerNorm(hidden_dim)\n            logger.debug(\"Critic: LayerNorm initialized.\")\n\n        self.fc = nn.Sequential(\n            nn.Linear(hidden_dim, hidden_dim),\n            nn.LeakyReLU(negative_slope=0.01),\n            nn.Linear(hidden_dim, 1)\n        )\n        logger.debug(f\"Critic initialized with feature_dim={feature_dim}, hidden_dim={hidden_dim}\")\n\n    def forward(self, x):\n        logger.debug(f\"Critic forward pass: input shape {x.shape}\")\n        x = self.encoder(x)\n        logger.debug(f\"Critic forward pass: after encoder, shape {x.shape}\")\n        x = x[:, -1]  # Get the output from the last time step\n        logger.debug(f\"Critic forward pass: after slicing, shape {x.shape}\")\n        if self.layer_norm_flag:\n            x = self.layer_norm(x)\n            logger.debug(\"Critic forward pass: LayerNorm applied.\")\n        values = self.fc(x)\n        logger.debug(f\"Critic forward pass: values shape {values.shape}\")\n        if torch.isnan(values).any():\n            logger.error(\"Values contain NaNs in Critic forward pass\")\n            raise ValueError(\"Values contain NaNs in Critic\")\n        return values\n\nclass PPOAgent(nn.Module):\n    def __init__(self, feature_dim, hidden_dim, action_dim, n_heads, n_layers,\n                 dropout=0.1, temperature=1.0, layer_norm=False):\n        super().__init__()\n        self.actor = Actor(\n            feature_dim=feature_dim,\n            hidden_dim=hidden_dim,\n            action_dim=action_dim,\n            n_heads=n_heads,\n            n_layers=n_layers,\n            temperature=temperature,\n            layer_norm=layer_norm\n        )\n        self.critic = Critic(\n            feature_dim=feature_dim,\n            hidden_dim=hidden_dim,\n            n_heads=n_heads,\n            n_layers=n_layers,\n            layer_norm=layer_norm\n        )\n        self.apply(initialize_weights)\n        logger.info(\"PPOAgent weights initialized.\")\n\n    def forward(self, x):\n        logger.debug(f\"PPOAgent forward pass: input shape {x.shape}\")\n        logits = self.actor(x)\n        value = self.critic(x)\n        logger.debug(f\"PPOAgent forward pass: logits shape {logits.shape}, value shape {value.shape}\")\n        return logits, value",
    "./models/transformer.py": "# models/transformer.py\n\nimport os\nimport torch\nimport torch.nn as nn\nimport math\nimport logging\n\ndef setup_transformer_logging(log_dir=\"logs\", log_file=\"transformer.log\", file_logging_enabled=False):\n    \"\"\"\n    Configure logging for the transformer module with an on/off switch for file logging.\n\n    Args:\n        log_dir (str): Directory to store log files.\n        log_file (str): Log file name.\n        file_logging_enabled (bool): Flag to enable/disable file logging.\n    \"\"\"\n    os.makedirs(log_dir, exist_ok=True)\n    logger = logging.getLogger(\"Transformer\")\n    logger.setLevel(logging.DEBUG)\n\n    # Create console handler\n    ch = logging.StreamHandler()\n    ch.setLevel(logging.INFO)\n\n    # Create formatter and add to handlers\n    formatter = logging.Formatter(\"%(asctime)s - %(levelname)s - %(name)s - %(message)s\")\n    ch.setFormatter(formatter)\n\n    # Add handlers to the logger if none exist\n    if not logger.hasHandlers():\n        logger.addHandler(ch)\n\n        # Add file handler based on the flag\n        if file_logging_enabled:\n            fh = logging.FileHandler(os.path.join(log_dir, log_file))\n            fh.setLevel(logging.DEBUG)\n            fh.setFormatter(formatter)\n            logger.addHandler(fh)\n        else:\n            # Add a NullHandler to effectively disable file logging\n            fh = logging.NullHandler()\n            logger.addHandler(fh)\n\n    return logger\n\n# Initialize logger for Transformer\ndef initialize_transformer_logging():\n    logger = logging.getLogger(\"Transformer\")\n    if not logger.hasHandlers():\n        logger = setup_transformer_logging()\n    return logger\n\nlogger = initialize_transformer_logging()\n\nclass PositionalEncoding(nn.Module):\n    def __init__(self, d_model, max_len=5000):\n        super().__init__()\n        self.encoding = torch.zeros(max_len, d_model)\n        position = torch.arange(0, max_len).unsqueeze(1).float()\n        div_term = torch.exp(torch.arange(0, d_model, 2).float() * -(math.log(10000.0) / d_model))\n        self.encoding[:, 0::2] = torch.sin(position * div_term)\n        self.encoding[:, 1::2] = torch.cos(position * div_term)\n        self.encoding = self.encoding.unsqueeze(0)  # Shape: (1, max_len, d_model)\n        logger.debug(f\"PositionalEncoding initialized with d_model={d_model}, max_len={max_len}\")\n\n    def forward(self, x):\n        \"\"\"\n        Add positional encoding to input tensor.\n\n        Args:\n            x (Tensor): Input tensor of shape (batch_size, seq_len, d_model).\n\n        Returns:\n            Tensor: Tensor with positional encoding added.\n        \"\"\"\n        try:\n            device = x.device\n            seq_len = x.size(1)\n            logger.debug(f\"PositionalEncoding forward: input shape {x.shape}, seq_len={seq_len}\")\n            if seq_len > self.encoding.size(1):\n                logger.error(f\"Sequence length {seq_len} exceeds maximum length {self.encoding.size(1)}.\")\n                raise ValueError(f\"Sequence length {seq_len} exceeds maximum length {self.encoding.size(1)}.\")\n            pos_enc = self.encoding[:, :seq_len, :].to(device)\n            logger.debug(f\"PositionalEncoding forward: position encoding shape {pos_enc.shape}\")\n            return x + pos_enc\n        except Exception as e:\n            logger.error(f\"Exception in PositionalEncoding forward: {e}\", exc_info=True)\n            raise\n\nclass TransformerEncoder(nn.Module):\n    def __init__(self, input_dim, hidden_dim, n_heads, n_layers, dropout=0.1):\n        super().__init__()\n        self.embedding = nn.Linear(input_dim, hidden_dim)\n        logger.debug(f\"TransformerEncoder: Embedding layer initialized with input_dim={input_dim}, hidden_dim={hidden_dim}\")\n        self.pos_enc = PositionalEncoding(hidden_dim)\n        self.transformer_encoder = nn.TransformerEncoder(\n            nn.TransformerEncoderLayer(\n                d_model=hidden_dim,\n                nhead=n_heads,\n                dropout=dropout,\n                batch_first=True\n            ),\n            num_layers=n_layers\n        )\n        logger.debug(f\"TransformerEncoder: TransformerEncoder initialized with n_layers={n_layers}, n_heads={n_heads}, dropout={dropout}\")\n        self.dropout = nn.Dropout(dropout)\n\n    def forward(self, x):\n        \"\"\"\n        Forward pass for the Transformer encoder.\n\n        Args:\n            x (Tensor): Input tensor of shape (batch_size, seq_len, input_dim).\n\n        Returns:\n            Tensor: Output tensor of shape (batch_size, seq_len, hidden_dim).\n        \"\"\"\n        try:\n            logger.debug(f\"TransformerEncoder forward: input shape {x.shape}\")\n            x = self.embedding(x)\n            logger.debug(f\"TransformerEncoder forward: after embedding, shape {x.shape}\")\n            x = self.pos_enc(x)\n            logger.debug(f\"TransformerEncoder forward: after positional encoding, shape {x.shape}\")\n            x = self.dropout(x)\n            logger.debug(f\"TransformerEncoder forward: after dropout, shape {x.shape}\")\n            x = self.transformer_encoder(x)\n            logger.debug(f\"TransformerEncoder forward: after transformer encoder, shape {x.shape}\")\n            return x\n        except Exception as e:\n            logger.error(f\"Exception in TransformerEncoder forward: {e}\", exc_info=True)\n            raise",
    "./data_processing/__init__.py": "",
    "./data_processing/preprocess.py": "# File: data_processing/preprocess.py\n\nimport os\nimport logging\nimport pandas as pd\nimport numpy as np\nimport joblib\nimport torch\nfrom sklearn.preprocessing import MinMaxScaler\nfrom typing import Dict\n\ndef setup_logging(log_dir=\"logs\", log_file=\"preprocess.log\"):\n    \"\"\"\n    Configure logging for the preprocessing script.\n\n    Args:\n        log_dir (str): Directory to store log files.\n        log_file (str): Log file name.\n    \n    Returns:\n        logging.Logger: Configured logger instance.\n    \"\"\"\n    os.makedirs(log_dir, exist_ok=True)\n    logger = logging.getLogger(\"Preprocessing\")\n    logger.setLevel(logging.INFO)\n\n    # Create console handler\n    ch = logging.StreamHandler()\n    ch.setLevel(logging.INFO)\n\n    # Create file handler\n    fh = logging.FileHandler(os.path.join(log_dir, log_file))\n    fh.setLevel(logging.DEBUG)\n\n    # Create formatter and add to handlers\n    formatter = logging.Formatter(\"%(asctime)s - %(levelname)s - %(name)s - %(message)s\")\n    ch.setFormatter(formatter)\n    fh.setFormatter(formatter)\n\n    # Add handlers to the logger if none exist\n    if not logger.hasHandlers():\n        logger.addHandler(ch)\n        logger.addHandler(fh)\n\n    return logger\n\n# Initialize logger\nlogger = logging.getLogger(\"Preprocessing\")\nif not logger.hasHandlers():\n    logger = setup_logging()\n\n\ndef export_and_save_preprocessed_data(config: Dict, logger: logging.Logger) -> None:\n    \"\"\"\n    Preprocess the raw data and split it into multiple PT files.\n\n    Args:\n        config (Dict): Configuration dictionary containing data paths and parameters.\n        logger (logging.Logger): Logger instance for logging messages.\n\n    Raises:\n        FileNotFoundError: If the raw data file does not exist.\n        KeyError: If required keys are missing in PT files.\n        ValueError: If the preprocessed dataframe is empty after cleaning.\n    \"\"\"\n    raw_data_path = config['data']['raw_data_path']\n    pt_data_dir = config['data'].get('pt_data_dir', 'data/processed/')\n    pt_data_prefix = config.get('data', {}).get('pt_data_prefix', 'processed_data_part')\n    os.makedirs(pt_data_dir, exist_ok=True)\n\n    # Check if already preprocessed\n    existing_pt_files = [fname for fname in os.listdir(pt_data_dir) if fname.startswith(pt_data_prefix)]\n    if existing_pt_files:\n        logger.info(f\"Processed data files already exist in '{pt_data_dir}'. Skipping preprocessing.\")\n        return\n\n    try:\n        logger.info(f\"Reading raw data from: {raw_data_path}\")\n        if not os.path.exists(raw_data_path):\n            logger.error(f\"Raw data file not found at: {raw_data_path}\")\n            raise FileNotFoundError(f\"Raw data file not found at: {raw_data_path}\")\n        \n        df = pd.read_csv(raw_data_path)\n        logger.info(f\"Initial data shape: {df.shape}\")\n\n        # Ensure 'close' is in feature columns\n        feat_cols = config['data']['feature_columns']\n        if 'close' not in feat_cols:\n            logger.error(\"'close' column is missing from feature_columns.\")\n            raise KeyError(\"'close' column is required in feature_columns for preprocessing.\")\n\n        # Replace infinite values with NaN\n        df.replace([np.inf, -np.inf], np.nan, inplace=True)\n\n        # Drop rows with NaN or infinite values\n        initial_length = len(df)\n        df.dropna(inplace=True)\n        cleaned_length = len(df)\n        logger.info(f\"Removed {initial_length - cleaned_length} rows containing NaN or infinite values.\")\n\n        # Check if dataframe is empty after cleaning\n        if df.empty:\n            logger.error(\"Preprocessed dataframe is empty after removing NaNs and infinite values.\")\n            raise ValueError(\"No data left after preprocessing.\")\n\n        # Normalize features if required\n        normalize_features = config['data'].get('normalize_features', True)\n        if normalize_features:\n            logger.info(\"Normalizing feature columns, including 'close'.\")\n            df = normalize_feature_columns(df, feat_cols, logger)\n\n        # Save unnormalized 'close' prices separately\n        unnormalized_close_prices = df['close'].values  # Assuming 'close' is in feat_cols\n        unnormalized_close_path = os.path.join(pt_data_dir, 'unnormalized_close_prices.npy')\n        np.save(unnormalized_close_path, unnormalized_close_prices)\n        logger.info(f\"Unnormalized 'close' prices saved to {unnormalized_close_path}\")\n\n        # Split data into parts and save as PT files\n        num_splits = config['data'].get('num_splits', 10)\n        \n        # Determine minimum required steps per split to avoid issues\n        n_steps = config['data'].get('n_steps', 20)\n        min_required_steps = n_steps + 1  # To ensure at least one sample per split\n        total_data_length = len(df)\n        max_possible_splits = total_data_length // min_required_steps\n\n        if max_possible_splits < num_splits:\n            logger.warning(f\"Requested num_splits={num_splits} is too high for the data size. Reducing to {max_possible_splits}.\")\n            num_splits = max_possible_splits\n\n        if num_splits == 0:\n            logger.error(\"Not enough data to create even one split with the required number of steps.\")\n            raise ValueError(\"Insufficient data for preprocessing.\")\n\n        data_splits = np.array_split(df[feat_cols].values, num_splits)\n        close_price_splits = np.array_split(unnormalized_close_prices, num_splits)\n\n        logger.info(f\"Splitting data into {num_splits} parts.\")\n\n        for idx, (split_data, split_close_prices) in enumerate(zip(data_splits, close_price_splits)):\n            # Verify that each split has enough data\n            if len(split_data) < min_required_steps:\n                logger.warning(f\"Split {idx} has only {len(split_data)} steps, which is less than the minimum required {min_required_steps}. Skipping this split.\")\n                continue\n\n            data_tensor = torch.tensor(split_data, dtype=torch.float32)\n            pt_file_path = os.path.join(pt_data_dir, f\"{pt_data_prefix}_{idx}.pt\")\n            torch.save({'data': data_tensor, 'close_prices': split_close_prices}, pt_file_path)\n            logger.info(f\"Saved split {idx} to {pt_file_path} with {len(split_data)} steps.\")\n\n        # Verification step: Check all PT files\n        verify_pt_files(pt_data_dir, pt_data_prefix, num_splits, n_steps, logger)\n\n    except FileNotFoundError as fnf_error:\n        logger.error(fnf_error)\n        raise\n    except KeyError as key_error:\n        logger.error(key_error)\n        raise\n    except Exception as e:\n        logger.error(f\"Failed to preprocess data: {e}\", exc_info=True)\n        raise\n\ndef normalize_feature_columns(df: pd.DataFrame, feat_cols: list, logger: logging.Logger) -> pd.DataFrame:\n    \"\"\"\n    Normalize the feature columns using MinMaxScaler, including 'close'.\n\n    Args:\n        df (pd.DataFrame): DataFrame containing the data to normalize.\n        feat_cols (list): List of feature column names to normalize.\n        logger (logging.Logger): Logger instance for logging messages.\n\n    Returns:\n        pd.DataFrame: DataFrame with normalized feature columns.\n\n    Raises:\n        Exception: If scaler loading or fitting fails.\n    \"\"\"\n    scaler_path = os.path.join('scalers', 'minmax_scaler.joblib')\n    os.makedirs('scalers', exist_ok=True)\n\n    features_to_normalize = feat_cols\n\n    if os.path.exists(scaler_path):\n        logger.info(f\"Loading scaler from {scaler_path}\")\n        try:\n            scaler = joblib.load(scaler_path)\n            # Check if scaler was fitted with the same features\n            if hasattr(scaler, 'feature_names_in_'):\n                scaler_features = scaler.feature_names_in_\n                if not np.array_equal(scaler_features, features_to_normalize):\n                    logger.warning(\"Scaler feature names do not match features to normalize. Re-fitting the scaler.\")\n                    scaler = MinMaxScaler(feature_range=(0, 1))\n                    scaler.fit(df[features_to_normalize])\n                    joblib.dump(scaler, scaler_path)\n                    logger.info(f\"Scaler re-fitted and saved to {scaler_path}\")\n            else:\n                logger.warning(\"Scaler does not have 'feature_names_in_'. Re-fitting the scaler.\")\n                scaler = MinMaxScaler(feature_range=(0, 1))\n                scaler.fit(df[features_to_normalize])\n                joblib.dump(scaler, scaler_path)\n                logger.info(f\"Scaler re-fitted and saved to {scaler_path}\")\n\n            df[features_to_normalize] = scaler.transform(df[features_to_normalize])\n        except Exception as e:\n            logger.error(f\"Error loading or fitting scaler: {e}\", exc_info=True)\n            raise\n    else:\n        logger.info(f\"No scaler found at {scaler_path}. Fitting a new scaler.\")\n        try:\n            scaler = MinMaxScaler(feature_range=(0, 1))\n            scaler.fit(df[features_to_normalize])\n            df[features_to_normalize] = scaler.transform(df[features_to_normalize])\n            joblib.dump(scaler, scaler_path)\n            logger.info(f\"Scaler fitted and saved to {scaler_path}\")\n        except Exception as e:\n            logger.error(f\"Error fitting scaler: {e}\", exc_info=True)\n            raise\n\n    return df\n\ndef verify_pt_files(pt_data_dir: str, pt_data_prefix: str, expected_num_splits: int, n_steps: int, logger: logging.Logger) -> None:\n    \"\"\"\n    Verify that all PT files have the correct structure and sufficient data.\n\n    Args:\n        pt_data_dir (str): Directory containing PT files.\n        pt_data_prefix (str): Prefix of PT file names.\n        expected_num_splits (int): Expected number of PT splits.\n        n_steps (int): Number of steps required per environment sample.\n        logger (logging.Logger): Logger instance for logging messages.\n\n    Raises:\n        ValueError: If any PT file is invalid or does not meet requirements.\n    \"\"\"\n    logger.info(\"Verifying PT files for data integrity and sufficiency.\")\n    for idx in range(expected_num_splits):\n        pt_file_path = os.path.join(pt_data_dir, f\"{pt_data_prefix}_{idx}.pt\")\n        if not os.path.exists(pt_file_path):\n            logger.error(f\"PT file missing: {pt_file_path}\")\n            raise ValueError(f\"PT file missing: {pt_file_path}\")\n\n        try:\n            content = torch.load(pt_file_path)\n            if 'data' not in content or 'close_prices' not in content:\n                logger.error(f\"PT file {pt_file_path} missing required keys.\")\n                raise KeyError(f\"PT file {pt_file_path} missing required keys.\")\n\n            data_tensor = content['data']\n            close_prices = content['close_prices']\n\n            # Verify data tensor\n            if not isinstance(data_tensor, torch.Tensor):\n                logger.error(f\"'data' in {pt_file_path} is not a torch.Tensor.\")\n                raise TypeError(f\"'data' in {pt_file_path} is not a torch.Tensor.\")\n\n            if len(data_tensor) < (n_steps + 1):\n                logger.error(f\"PT file {pt_file_path} has insufficient data: {len(data_tensor)} steps.\")\n                raise ValueError(f\"PT file {pt_file_path} has insufficient data: {len(data_tensor)} steps.\")\n\n            # Verify close_prices\n            if not isinstance(close_prices, np.ndarray):\n                logger.error(f\"'close_prices' in {pt_file_path} is not a numpy.ndarray.\")\n                raise TypeError(f\"'close_prices' in {pt_file_path} is not a numpy.ndarray.\")\n\n            if len(close_prices) != len(data_tensor):\n                logger.error(f\"Length mismatch between 'data' and 'close_prices' in {pt_file_path}.\")\n                raise ValueError(f\"Length mismatch between 'data' and 'close_prices' in {pt_file_path}.\")\n\n            logger.info(f\"PT file {pt_file_path} verified successfully with {len(data_tensor)} steps.\")\n\n        except Exception as e:\n            logger.error(f\"Verification failed for PT file {pt_file_path}: {e}\", exc_info=True)\n            raise"
}  and analyse those suggestions and tell me how you can implement those and at the end give me a final corrected version for each scripts, copy the whole corrected script here one by one.   make sure to apply implementation with minimal change from the original scripts.  here are the suggestions : Based on the analysis of your training metrics and the provided config.yaml, several adjustments can be made to address the identified issues and enhance your model's performance. Below are detailed recommendations along with explanations for each suggested change:

1. Adjust Entropy Coefficient to Enhance Exploration
Current Setting:

yaml
Copy code
training:
  entropy_coeff: 0.005
Issue Identified:

Premature Entropy Reduction: The entropy coefficient is quite low, leading to rapid entropy decay. This causes the policy to become overly deterministic too quickly, reducing exploration and potentially getting stuck in local minima.
Recommendation:

Increase Entropy Coefficient: Raising the entropy_coeff encourages the policy to maintain higher entropy, promoting continued exploration.
Suggested Change:

yaml
Copy code
training:
  entropy_coeff: 0.01  # Increased from 0.005
Explanation: A higher entropy coefficient (e.g., 0.01 or even higher) allows the policy to explore more diverse actions for a longer period during training, which can help in discovering more optimal strategies and prevent premature convergence to suboptimal policies.

2. Implement Entropy Scheduling
Issue Identified:

Static Entropy Coefficient: A fixed entropy coefficient may not be optimal throughout the training process. Early stages benefit from higher exploration, while later stages can reduce exploration as the policy becomes more refined.
Recommendation:

Use Entropy Scheduling: Implement a schedule that gradually decreases the entropy coefficient over time.
Suggested Addition:

yaml
Copy code
training:
  entropy_coeff: 0.01  # Initial value
  entropy_schedule:
    type: linear
    start: 0.01
    end: 0.001
    decay_epochs: 100
Explanation: By scheduling the entropy coefficient to decrease linearly from 0.01 to 0.001 over 100 epochs, the model maintains sufficient exploration in the early stages and gradually shifts towards exploitation as training progresses. This balances exploration and exploitation effectively.

3. Stabilize Policy Updates by Adjusting Learning Rates
Current Settings:

yaml
Copy code
training:
  actor_lr: 1e-5
  critic_lr: 1e-6
Issue Identified:

Actor and Critic Learning Rates Discrepancy: The actor's learning rate is significantly higher than the critic's, which can lead to unstable policy updates.
Recommendation:

Balance Learning Rates: Consider using closer learning rates for the actor and critic or implementing separate schedulers for each.
Suggested Change:

yaml
Copy code
training:
  actor_lr: 3e-5  # Increased from 1e-5
  critic_lr: 3e-5  # Increased from 1e-6 to match actor_lr
Explanation: Matching the learning rates of the actor and critic can promote more stable and synchronized updates between the policy and value networks. Additionally, slightly increasing the learning rates can help the model learn more effectively without being too slow.

4. Incorporate Learning Rate Schedulers
Issue Identified:

Static Learning Rates: Fixed learning rates may not adapt well to different training phases, potentially leading to suboptimal convergence.
Recommendation:

Add Learning Rate Schedulers: Implement schedulers that adjust the learning rate based on training progress, such as step decay or cosine annealing.
Suggested Addition:

yaml
Copy code
training:
  lr_scheduler:
    type: cosine
    eta_min: 1e-6
Explanation: A cosine annealing scheduler gradually decreases the learning rate following a cosine curve, allowing the model to take larger steps initially and finer steps as it approaches convergence. This can enhance both the speed and stability of training.

5. Enhance Batch Size for More Stable Gradient Estimates
Current Setting:

yaml
Copy code
training:
  batch_size: 64
Issue Identified:

Potential Gradient Instability: A smaller batch size can lead to noisy gradient estimates, causing instability in training.
Recommendation:

Increase Batch Size: If computational resources allow, increasing the batch size can provide more stable gradient estimates.
Suggested Change:

yaml
Copy code
training:
  batch_size: 128  # Increased from 64
Explanation: A larger batch size averages out the noise in gradient estimates, leading to more stable and reliable updates. This can help in reducing the fluctuations observed in the actor loss and improve overall training stability.

6. Refine PPO Clipping and Update Parameters
Current Settings:

yaml
Copy code
training:
  clip_epsilon: 0.1
  ppo_epochs: 4
Issue Identified:

Potential Over-Optimization: A clip_epsilon of 0.1 is standard, but coupled with a relatively high number of PPO epochs, it might lead to overfitting to the sampled data.
Recommendation:

Adjust PPO Parameters: Consider reducing the number of PPO epochs and/or the clipping epsilon to prevent over-optimization.
Suggested Change:

yaml
Copy code
training:
  clip_epsilon: 0.2  # Slightly increased to allow more exploration within updates
  ppo_epochs: 3  # Reduced from 4
Explanation: Reducing the number of PPO epochs per update cycle can help prevent the policy from overfitting to specific batches of data. Adjusting the clipping epsilon to 0.2 allows for more flexibility in policy updates without excessively large changes.

7. Incorporate Gradient Clipping and Normalization
Issue Identified:

Potential for Exploding Gradients: Although max_grad_norm is set to 0.5, ensuring effective gradient clipping and normalization can further stabilize training.
Recommendation:

Enhance Gradient Handling: Verify that gradient clipping is functioning correctly and consider adding gradient normalization if necessary.
Suggested Addition/Verification:

yaml
Copy code
training:
  max_grad_norm: 0.5  # Ensure this is effectively applied
  optimizer:
    type: Adam
    betas: [0.9, 0.999]
    eps: 1e-8
    weight_decay: 1e-4  # Introduce weight decay for regularization
Explanation: Proper gradient clipping prevents the gradients from becoming too large, which can destabilize training. Adding weight decay introduces regularization, helping to prevent overfitting and promoting generalization.

8. Enhance Data Preprocessing and Feature Engineering
Current Settings:

yaml
Copy code
data:
  normalize_features: true
preprocessing:
  enable: true
Issue Identified:

Potential Inadequate Feature Representation: While normalization is enabled, ensuring comprehensive feature engineering can significantly impact model performance.
Recommendation:

Expand Feature Set: Incorporate additional technical indicators or domain-specific features that might better capture market dynamics.
Ensure Proper Normalization: Verify that normalization is appropriately applied, possibly experimenting with different normalization techniques (e.g., standard scaling vs. min-max scaling).
Suggested Change:

yaml
Copy code
data:
  feature_columns:
    - ...  # Existing features
    - additional_feature_1
    - additional_feature_2
  normalize_features: true
preprocessing:
  normalization_type: "standard"  # or "min-max"
Explanation: Expanding the feature set can provide the model with more information to make informed decisions. Proper normalization ensures that all features contribute equally to the learning process, preventing any single feature from dominating due to scale differences.

9. Increase Evaluation Frequency for Better Monitoring
Current Setting:

yaml
Copy code
evaluation:
  interval_eval: 5
Issue Identified:

Delayed Performance Feedback: Evaluating every 5 epochs may delay the detection of performance degradation or issues.
Recommendation:

Increase Evaluation Frequency: Evaluate more frequently, such as every 2 or 3 epochs, to monitor performance trends more closely.
Suggested Change:

yaml
Copy code
evaluation:
  interval_eval: 2  # Increased evaluation frequency
Explanation: Frequent evaluations allow for quicker detection of issues like overfitting or performance drops, enabling timely interventions and adjustments to the training process.

10. Implement Early Stopping Based on Evaluation Metrics
Issue Identified:

Continued Training Despite Deteriorating Performance: Training continues for all 100 epochs even when evaluation metrics worsen.
Recommendation:

Add Early Stopping: Incorporate an early stopping mechanism that halts training when evaluation metrics (e.g., average reward) do not improve for a certain number of consecutive evaluations.
Suggested Addition:

yaml
Copy code
training:
  early_stopping:
    enabled: true
    patience: 10  # Number of evaluations with no improvement before stopping
    metric: "average_reward"
    mode: "max"  # or "min" based on the metric
Explanation: Early stopping prevents unnecessary training once the model stops improving, saving computational resources and preventing potential overfitting.

11. Fine-Tune Action Selection Temperature
Current Setting:

yaml
Copy code
model:
  temperature: 1.0
Issue Identified:

Action Selection Uncertainty: The temperature parameter controls the randomness of action selection. A temperature of 1.0 may not provide the optimal balance between exploration and exploitation.
Recommendation:

Experiment with Temperature Values: Adjust the temperature to see how it affects action selection and overall performance.
Suggested Change:

yaml
Copy code
model:
  temperature: 0.8  # Reduced from 1.0 for more deterministic actions
Explanation: Lowering the temperature makes the policy more deterministic, which can be beneficial if the current level of randomness is too high. Conversely, increasing it can enhance exploration. Fine-tuning this parameter based on validation performance can help achieve the desired balance.

12. Review and Optimize Feature Dimensions and Model Architecture
Current Settings:

yaml
Copy code
model:
  feature_dim: 32
  ppo_hidden_dim: 256
  ppo_n_heads: 4
  ppo_n_layers: 3
  ppo_dropout: 0.1
  layer_norm: true
Issue Identified:

Potential Overparameterization or Underparameterization: The current model configuration may not be optimal for the complexity of the task.
Recommendation:

Adjust Model Complexity: Depending on the data and task, experiment with increasing or decreasing the number of layers, hidden dimensions, or attention heads.
Ensure Sufficient Capacity: Ensure that the model has enough capacity to capture the underlying patterns without being excessively complex.
Suggested Change:

yaml
Copy code
model:
  ppo_n_layers: 2  # Reduced from 3 to prevent overfitting
  ppo_hidden_dim: 128  # Reduced from 256 to simplify the model
Explanation: Simplifying the model can help in preventing overfitting, especially if the dataset is not large enough to support a highly complex model. Alternatively, if underfitting is suspected, increasing model capacity can be beneficial.

13. Optimize Data Splits and Training-Evaluation Balance
Current Setting:

yaml
Copy code
data:
  num_splits: 2
Issue Identified:

Potential Insufficient Data for Training and Evaluation: With only two splits, there might be limited data available for either training or evaluation, affecting the model's ability to generalize.
Recommendation:

Increase Number of Splits: Consider using more splits to provide better training and validation data distribution.
Suggested Change:

yaml
Copy code
data:
  num_splits: 5  # Increased from 2 for better data distribution
Explanation: More data splits can facilitate better cross-validation, ensuring that the model is exposed to diverse data during training and evaluation phases, enhancing its generalization capabilities.

14. Incorporate Regularization Techniques
Issue Identified:

Potential Overfitting: Although weight decay is set to 0.0, introducing regularization can help prevent overfitting.
Recommendation:

Add Weight Decay: Introduce a small weight decay to regularize the model.
Suggested Change:

yaml
Copy code
training:
  weight_decay: 1e-4  # Introduced from 0.0 to add regularization
Explanation: Weight decay penalizes large weights, encouraging the model to maintain simpler representations and improving generalization.

15. Enhance Data Quality and Quantity
Issue Identified:

Potential Data Limitations: The model's performance is highly dependent on the quality and quantity of the training data.
Recommendation:

Augment Data: If possible, increase the size of the dataset or incorporate data augmentation techniques relevant to financial time series.
Ensure Data Integrity: Verify that the data preprocessing steps correctly handle missing values, outliers, and ensure accurate feature representation.
Suggested Addition/Action:

yaml
Copy code
data:
  raw_data_path: "./data/dataset_extended.csv"  # Use a more extensive dataset if available
preprocessing:
  handle_missing: "interpolate"  # Example strategy for handling missing data
Explanation: A larger and cleaner dataset provides the model with more information to learn robust patterns, enhancing its ability to generalize to unseen data.

16. Monitor and Analyze Additional Metrics
Issue Identified:

Limited Evaluation Metrics: Focusing solely on average reward and Sharpe ratio may not provide a comprehensive view of performance.
Recommendation:

Track More Metrics: Incorporate additional evaluation metrics such as Precision, Recall, F1-Score, or custom financial metrics relevant to your trading strategy.
Suggested Addition:

yaml
Copy code
evaluation:
  additional_metrics:
    - precision
    - recall
    - f1_score
    - custom_financial_metric_1
Explanation: Monitoring a broader set of metrics can provide deeper insights into the model's strengths and weaknesses, guiding more informed adjustments to the training process.

17. Utilize Checkpointing and Model Saving Strategically
Current Setting:

yaml
Copy code
checkpoint:
  best_model_path: "./checkpoints/best_model.pth"
  resume_checkpoint: ""
Issue Identified:

Potential Loss of Best Models: Without frequent checkpointing, you might miss saving the best-performing models during training.
Recommendation:

Save Checkpoints Based on Evaluation Metrics: Configure the training process to save the model whenever a new best evaluation metric is achieved.
Suggested Change:

yaml
Copy code
checkpoint:
  best_model_path: "./checkpoints/best_model.pth"
  save_best_only: true  # Ensure only the best model based on evaluation metrics is saved
Explanation: Saving the best model ensures that you retain the most effective policy, even if later training epochs lead to performance degradation.

18. Review Distributed Training Configuration
Current Settings:

yaml
Copy code
distributed:
  master_addr: "localhost"
  master_port: "29500"
  world_size: 2
Issue Identified:

Potential Overhead or Misconfiguration: Distributed training can introduce complexity. If not properly configured, it might lead to inefficiencies or training instability.
Recommendation:

Verify Distributed Setup: Ensure that the distributed training setup is correctly configured and that the number of GPUs (world_size) matches the available hardware.
Monitor for Synchronization Issues: Check for potential synchronization problems that could affect training stability.
Suggested Change:

yaml
Copy code
distributed:
  master_addr: "localhost"
  master_port: "29500"
  world_size: <number_of_available_GPUs>  # Ensure this matches your hardware
Explanation: Proper configuration of distributed training is crucial for leveraging multiple GPUs effectively. Mismatches or misconfigurations can lead to suboptimal performance or training issues.

19. Fine-Tune Environment Parameters
Current Settings:

yaml
Copy code
environment:
  max_steps: 200
  transaction_fee: 0.001
Issue Identified:

Impact of Transaction Fees: High transaction fees can significantly affect trading performance, especially in strategies like scalping that involve frequent trades.
Recommendation:

Reassess Transaction Fees: Ensure that the transaction_fee accurately reflects real-world costs. If set too high, it can negate the benefits of profitable trades.
Suggested Change:

yaml
Copy code
environment:
  transaction_fee: 0.0005  # Reduced from 0.001 to simulate lower transaction costs
Explanation: Lowering the transaction fee can make the trading strategy more viable, especially when dealing with frequent trades. However, ensure that this value aligns with realistic trading scenarios to maintain the model's applicability.

20. Enhance Preprocessing Techniques
Current Settings:

yaml
Copy code
preprocessing:
  enable: true
Issue Identified:

Potential Underutilization of Preprocessing: Simply enabling preprocessing may not encompass all necessary steps for optimal data preparation.
Recommendation:

Expand Preprocessing Steps: Include feature engineering, handling missing values, outlier removal, and other relevant preprocessing techniques.
Suggested Addition:

yaml
Copy code
preprocessing:
  enable: true
  handle_missing: "impute_mean"  # Strategy to handle missing data
  remove_outliers: true
  feature_engineering:
    - create_moving_averages
    - create_volatility_indicators
Explanation: Comprehensive preprocessing ensures that the data fed into the model is clean, consistent, and rich with meaningful features, which can significantly enhance model performance.

21. Validate Data Normalization Approach
Current Setting:

yaml
Copy code
data:
  normalize_features: true
Issue Identified:

Potential Misalignment in Normalization: Different features may benefit from different normalization techniques based on their distributions.
Recommendation:

Customize Normalization per Feature: Apply appropriate normalization techniques tailored to each feature's characteristics.
Suggested Change:

yaml
Copy code
data:
  normalize_features: true
  normalization:
    open: "standard"
    high: "standard"
    low: "standard"
    close: "standard"
    volume: "min-max"
    # Continue for other features
Explanation: Applying suitable normalization techniques for each feature ensures that the model receives appropriately scaled inputs, improving learning efficiency and performance.

22. Explore Alternative Model Architectures
Issue Identified:

Potential Limitations of Current Model: The current Transformer-based architecture may not be the most suitable for the task.
Recommendation:

Experiment with Different Architectures: Consider testing other architectures like LSTM, GRU, or even hybrid models that combine CNNs with Transformers to capture both temporal and spatial patterns.
Suggested Action:

Prototype Alternative Models: Develop and evaluate alternative models to compare performance and identify the most effective architecture for your specific task.
Explanation: Different model architectures have varying strengths. For time series and sequential data like financial markets, architectures that effectively capture temporal dependencies (e.g., LSTMs) might offer performance benefits over purely Transformer-based models.

23. Incorporate Robust Evaluation Strategies
Issue Identified:

Limited Evaluation Scope: Relying on a single baseline reward may not provide a comprehensive performance assessment.
Recommendation:

Use Diverse Baselines and Benchmarks: Compare the model against multiple baselines, such as random policies, simple heuristic strategies, and other benchmark models.
Suggested Addition:

yaml
Copy code
evaluation:
  baselines:
    - random_policy
    - moving_average_strategy
    - momentum_strategy
Explanation: Comparing against diverse baselines helps in understanding the relative performance of your model, ensuring that improvements are meaningful and not just relative to a single baseline.

24. Ensure Reproducibility and Consistency
Current Setting:

yaml
Copy code
random_seed: 42
Issue Identified:

Potential Inconsistencies in Reproducibility: Reproducibility is crucial for consistent results, especially in distributed settings.
Recommendation:

Ensure Comprehensive Reproducibility Measures: Set seeds for all relevant libraries (e.g., NumPy, PyTorch), ensure deterministic operations, and document all sources of randomness.
Suggested Addition:

yaml
Copy code
random_seed: 42
reproducibility:
  deterministic: true
  benchmark: false
Explanation: Setting deterministic flags and disabling benchmarking can help in achieving more reproducible results across different runs, aiding in debugging and consistent performance assessment.

25. Optimize Distributed Training Parameters
Issue Identified:

Potential Overhead in Distributed Training: If not necessary, distributed training can introduce communication overheads that may not be justified by the performance gains.
Recommendation:

Assess the Need for Distributed Training: If training on a single machine with multiple GPUs suffices, ensure that configurations are optimized to minimize overhead.
Suggested Change:

yaml
Copy code
distributed:
  master_addr: "localhost"
  master_port: "29500"
  world_size: 2  # Ensure this matches the number of available GPUs without overcommitting
Explanation: Properly configuring distributed training ensures efficient utilization of resources. Overcommitting GPUs or misconfiguring distributed settings can lead to inefficiencies and training instability.

26. Incorporate Advanced Optimization Techniques
Issue Identified:

Potential for Enhanced Optimization: Utilizing advanced optimization techniques can further stabilize and accelerate training.
Recommendation:

Use Optimizers with Adaptive Learning Rates: Although Adam is standard, experimenting with optimizers like AdamW or RMSprop might yield better performance.
Suggested Change:

yaml
Copy code
training:
  optimizer:
    type: AdamW  # Switched from Adam to AdamW
    lr: 3e-5
    betas: [0.9, 0.999]
    eps: 1e-8
    weight_decay: 1e-4
Explanation: AdamW decouples weight decay from the optimization steps, providing better regularization and potentially more stable convergence compared to Adam.

27. Monitor and Address Overfitting
Issue Identified:

Discrepancy Between Training and Evaluation Metrics: Indicates possible overfitting, where the model performs well on training data but poorly on unseen data.
Recommendation:

Implement Regularization and Validation Strategies: Use techniques like dropout, early stopping, and cross-validation to mitigate overfitting.
Suggested Change:

yaml
Copy code
model:
  ppo_dropout: 0.2  # Increased from 0.1 to add more regularization
training:
  early_stopping:
    enabled: true
    patience: 10
    metric: "average_reward"
    mode: "max"
Explanation: Increasing dropout adds more regularization, helping the model generalize better. Early stopping based on evaluation metrics prevents training from continuing once overfitting starts.

28. Enhance Action Space Design
Current Setting:

yaml
Copy code
model:
  action_dim: 7  # Sell 100%, Sell 50%, Sell 25%, Hold, Buy 25%, Buy 50%, Buy 100%
Issue Identified:

Potential Complexity in Action Space: A large number of discrete actions can complicate the learning process, especially if some actions are rarely optimal.
Recommendation:

Simplify or Reconfigure Action Space: Consider reducing the number of discrete actions or introducing continuous action spaces if applicable.
Suggested Change:

yaml
Copy code
model:
  action_dim: 5  # Example: Sell 100%, Sell 50%, Hold, Buy 50%, Buy 100%
Explanation: Reducing the number of actions can simplify the decision-making process for the model, making learning more efficient. Alternatively, using a continuous action space allows the model to choose precise action magnitudes, which can be more flexible and effective in trading scenarios.

29. Validate and Enhance Reward Function
Issue Identified:

Potential Misalignment in Reward Function: The reward scaling and design may not effectively guide the learning process towards desired outcomes.
Recommendation:

Review and Refine Reward Structure: Ensure that the reward function accurately reflects profitable and safe trading behaviors. Consider incorporating additional components like risk penalties or multi-objective rewards.
Suggested Change:

yaml
Copy code
data:
  reward_scale: 1.0  # Consider experimenting with different scales
environment:
  reward_components:
    profit: 1.0
    risk_penalty: -0.5
Explanation: A well-designed reward function aligns the model's incentives with desired behaviors. Incorporating risk penalties can encourage the model to prioritize safer strategies, improving overall performance and stability.

30. Ensure Comprehensive Data Coverage and Temporal Consistency
Issue Identified:

Potential Temporal Bias: If the data splits do not account for temporal consistency, the model may encounter data leakage or unrealistic training scenarios.
Recommendation:

Implement Time-Series Aware Splitting: Use techniques like rolling window splits or walk-forward validation to maintain temporal integrity.
Suggested Change:

yaml
Copy code
data:
  split_type: "time_series"  # Ensures that training data precedes evaluation data temporally
Explanation: Time-series aware splitting prevents data leakage by ensuring that future information does not influence the training process, leading to more realistic and reliable evaluation metrics.

31. Optimize n_steps Parameter for Better Temporal Dependencies
Current Setting:

yaml
Copy code
data:
  n_steps: 20
Issue Identified:

Potential Misalignment with Market Dynamics: The number of steps considered may not align with the temporal dependencies relevant to the trading strategy.
Recommendation:

Experiment with Different n_steps: Adjust the n_steps parameter to better capture the temporal patterns and dependencies in the data.
Suggested Change:

yaml
Copy code
data:
  n_steps: 50  # Increased from 20 to capture longer temporal dependencies
Explanation: Increasing n_steps allows the model to consider a longer history of data points, potentially capturing more meaningful temporal patterns that can inform better trading decisions.

32. Validate Feature Importance and Redundancy
Issue Identified:

Potential Redundant or Irrelevant Features: Having too many features can introduce noise and reduce model performance.
Recommendation:

Perform Feature Selection: Use techniques like correlation analysis, feature importance from models, or dimensionality reduction to identify and retain the most relevant features.
Suggested Action:

Conduct Feature Importance Analysis: Utilize tools like SHAP values or permutation importance to assess and select the most impactful features.
Explanation: Removing redundant or irrelevant features can enhance the model's ability to learn meaningful patterns, improving both training efficiency and performance.

33. Ensure Proper Handling of Imbalanced Data
Issue Identified:

Potential Imbalance in Action Classes: If certain actions (e.g., Hold) are more frequent than others, the model might become biased towards those actions.
Recommendation:

Balance Action Classes: Implement techniques to balance the frequency of different actions during training.
Suggested Addition:

yaml
Copy code
training:
  action_sampling:
    balanced: true  # Ensures that each action is sampled equally during training
Explanation: Balancing action classes prevents the model from being biased towards more frequent actions, promoting a more diverse and effective policy.

34. Implement Robust Validation and Testing Procedures
Issue Identified:

Limited Validation Strategies: Relying solely on evaluation metrics without comprehensive validation can obscure underlying issues.
Recommendation:

Expand Validation Procedures: Incorporate cross-validation, out-of-sample testing, and stress testing to ensure the model's robustness.
Suggested Action:

yaml
Copy code
evaluation:
  validation_split: 0.2  # Allocate 20% of data for validation
  stress_test:
    scenarios:
      - market_crash
      - high_volatility
Explanation: Robust validation and testing ensure that the model performs well under various market conditions and is not just tailored to specific data patterns, enhancing its reliability in real-world scenarios.

35. Utilize Hyperparameter Tuning Techniques
Issue Identified:

Manual Hyperparameter Selection: Relying on manually selected hyperparameters may not yield the optimal configuration.
Recommendation:

Implement Automated Hyperparameter Tuning: Use techniques like grid search, random search, or Bayesian optimization to systematically explore hyperparameter spaces.
Suggested Action:

yaml
Copy code
hyperparameter_tuning:
  enabled: true
  method: "bayesian"  # Options: "grid", "random", "bayesian"
  parameters:
    actor_lr: [1e-5, 3e-5, 1e-4]
    critic_lr: [1e-6, 3e-6, 1e-5]
    entropy_coeff: [0.005, 0.01, 0.02]
    ppo_n_layers: [2, 3, 4]
Explanation: Automated hyperparameter tuning can identify optimal configurations that might not be evident through manual adjustments, potentially leading to significant performance improvements.

Summary of Key Recommendations
Enhance Exploration:
Increase and schedule the entropy coefficient to maintain exploration.
Stabilize Training:
Balance and possibly increase learning rates for the actor and critic.
Implement learning rate schedulers.
Increase batch size for more stable gradient estimates.
Prevent Overfitting:
Introduce weight decay and increase dropout rates.
Implement early stopping based on evaluation metrics.
Optimize Model and Data:
Adjust model architecture parameters.
Expand and refine the feature set.
Ensure robust data preprocessing and normalization.
Improve Evaluation and Monitoring:
Increase evaluation frequency and incorporate additional metrics.
Save checkpoints based on performance.
Leverage Advanced Techniques:
Experiment with different optimizers and action space configurations.
Utilize automated hyperparameter tuning for optimal configurations.
By implementing these recommendations, you can address the primary issues of insufficient exploration, policy instability, and potential overfitting, thereby improving your model's performance and reliability in trading scenarios.