Following this instruction :
1. Strictly adhere to PEP-8 standards, ensuring code is clean, well-structured, and follows Python's style guide for readability.
2. Use appropriate modules like `typing` for type hints, ensuring code clarity, maintainability, and leveraging Python's standard libraries effectively.
3. Focus on time, space, robustness, scalability; write optimized, efficient code that can handle large datasets and scale seamlessly.
4. Implement comprehensive error handling, catching exceptions gracefully, providing meaningful error messages, and ensuring code robustness under unexpected conditions.
5. Ensure the developer agent writes advanced, maintainable Python code, balancing performance and readability while adhering to best practices.

TASK : objective we have develop below TrPO  from scratch using only torch module :(assume actor , critic, any model or generalised model (may be text, video ,image , embedding wav_vec etc)

user_input :
actor :
modelconfig:
1 Model (nn.module) 
2) loss fumnction
3) optimiser

critic :
modelconfig:
1 Model (nn.module) 
2) loss fumnction
3) optimiser




---

### 1. TRPO Objective Function
Given a policy \( \pi_\theta \) with parameters \( \theta \), TRPO maximizes the expected advantage while keeping the policy close to the previous policy. This can be formalized as:

\[
\max_{\theta} \mathbb{E}_{s \sim \rho_{\pi_{\theta_{\text{old}}}}, a \sim \pi_{\theta_{\text{old}}}} \left[ \frac{\pi_\theta(a|s)}{\pi_{\theta_{\text{old}}}(a|s)} A^{\pi_{\theta_{\text{old}}}}(s, a) \right]
\]

where:
- \( \pi_{\theta_{\text{old}}} \): the old policy (before the update).
- \( \pi_\theta \): the new policy being optimized.
- \( A^{\pi_{\theta_{\text{old}}}}(s, a) \): the advantage function with respect to the old policy.

The objective is to maximize the advantage while ensuring that the updated policy doesn’t deviate too much from the old one.

---

### 2. KL Divergence Constraint
TRPO uses the Kullback-Leibler (KL) divergence to limit the size of policy updates. Specifically, it imposes a constraint on the average KL divergence between the old policy and the new policy:

\[
\mathbb{E}_{s \sim \rho_{\pi_{\theta_{\text{old}}}}} \left[ D_{\text{KL}} \left( \pi_{\theta_{\text{old}}}(\cdot|s) \| \pi_\theta(\cdot|s) \right) \right] \leq \delta
\]

where:
- \( D_{\text{KL}} \left( \pi_{\theta_{\text{old}}}(\cdot|s) \| \pi_\theta(\cdot|s) \right) \): the KL divergence between the old and new policies at state \( s \).
- \( \delta \): a hyperparameter controlling the trust region size, typically small (e.g., 0.01 or 0.02).

The KL divergence constraint ensures that the policy update does not move the new policy too far from the old one, helping maintain stability.

---

### 3. Linear Approximation of Objective
To make the optimization tractable, TRPO uses a linear approximation of the objective function around \( \theta_{\text{old}} \). Defining the probability ratio \( r_\theta(s, a) = \frac{\pi_\theta(a|s)}{\pi_{\theta_{\text{old}}}(a|s)} \), we can approximate the objective as:

\[
L(\theta) \approx g^T (\theta - \theta_{\text{old}})
\]

where \( g \) is the policy gradient given by:

\[
g = \mathbb{E}_{s \sim \rho_{\pi_{\theta_{\text{old}}}}, a \sim \pi_{\theta_{\text{old}}}} \left[ \nabla_\theta \log \pi_\theta(a|s) A^{\pi_{\theta_{\text{old}}}}(s, a) \right]_{\theta = \theta_{\text{old}}}
\]

---

### 4. Quadratic Approximation of the KL Divergence
The KL divergence constraint is also approximated quadratically to make it easier to solve. This is expressed as:

\[
\mathbb{E}_{s \sim \rho_{\pi_{\theta_{\text{old}}}}} \left[ D_{\text{KL}} \left( \pi_{\theta_{\text{old}}}(\cdot|s) \| \pi_\theta(\cdot|s) \right) \right] \approx \frac{1}{2} (\theta - \theta_{\text{old}})^T F (\theta - \theta_{\text{old}})
\]

where \( F \) is the Fisher information matrix, defined as:

\[
F = \mathbb{E}_{s \sim \rho_{\pi_{\theta_{\text{old}}}}, a \sim \pi_{\theta_{\text{old}}}} \left[ \nabla_\theta \log \pi_{\theta}(a|s) \nabla_\theta \log \pi_{\theta}(a|s)^T \right]
\]

The Fisher information matrix captures the sensitivity of the policy with respect to parameter changes, allowing TRPO to determine an appropriate trust region size.

---

### 5. Constrained Optimization Problem
With the linear approximation of the objective and quadratic approximation of the KL constraint, TRPO solves the following constrained optimization problem:

\[
\max_{\theta} \; g^T (\theta - \theta_{\text{old}})
\]
subject to:
\[
\frac{1}{2} (\theta - \theta_{\text{old}})^T F (\theta - \theta_{\text{old}}) \leq \delta
\]

This is a constrained quadratic programming problem, which can be solved using methods like the conjugate gradient algorithm for efficient updates.

---

### 6. Policy Update with Step Size Scaling
Once the direction \( \Delta \theta \) is determined, TRPO performs a line search along this direction to ensure that the new policy satisfies the KL divergence constraint. The update is given by:

\[
\theta_{\text{new}} = \theta_{\text{old}} + \alpha \Delta \theta
\]

where \( \alpha \) is a step size parameter, found by backtracking line search to satisfy the KL constraint within the limit \( \delta \).

---

### TRPO Algorithm Steps

1. **Collect Trajectories**: Run the current policy in the environment to collect trajectories.
2. **Estimate Advantage**: Calculate the advantage function \( A^{\pi_{\theta_{\text{old}}}}(s, a) \) using methods such as Generalized Advantage Estimation (GAE).
3. **Compute Policy Gradient \( g \)**: Compute the policy gradient \( g \) based on the trajectories.
4. **Compute Fisher Information Matrix \( F \)**: Estimate the Fisher information matrix for the KL constraint.
5. **Solve Constrained Optimization**: Use conjugate gradient or another method to solve the constrained optimization problem for \( \Delta \theta \).
6. **Perform Line Search**: Scale the step size \( \alpha \) to ensure the KL divergence constraint is satisfied.
7. **Update Policy**: Set \( \theta_{\text{new}} = \theta_{\text{old}} + \alpha \Delta \theta \).
8. **Repeat**: Repeat until convergence or for a set number of iterations.

---
\