Following this instruction :
1. Strictly adhere to PEP-8 standards, ensuring code is clean, well-structured, and follows Python's style guide for readability.
2. Use appropriate modules like `typing` for type hints, ensuring code clarity, maintainability, and leveraging Python's standard libraries effectively.
3. Focus on time, space, robustness, scalability; write optimized, efficient code that can handle large datasets and scale seamlessly.
4. Implement comprehensive error handling, catching exceptions gracefully, providing meaningful error messages, and ensuring code robustness under unexpected conditions.
5. Ensure the developer agent writes advanced, maintainable Python code, balancing performance and readability while adhering to best practices.

TASK : we develop using torch module end to end , DAPO , from scratch user input is  env (import gymnasium as gym  env = gym.make("LunarLander-v3", render_mode="human"))


----
Mirror descent (MD) is a powerful optimization framework that extends and generalizes classical methods such as gradient descent by taking advantage of a dual representation of the problem. The key component of MD is the use of **mirror maps**, which map the optimization iterates back and forth between the **primal space** (where the problem is defined) and the **dual space** (where computations are performed). These maps are typically associated with convex functions of Legendre-type, leading to updates that incorporate curvature information about the domain and objective function. Let's break down the concepts introduced in the provided text:

### Bregman Divergence and Bregman Projection

The **Bregman divergence** is a fundamental concept in mirror descent, and it generalizes the concept of squared Euclidean distance in a way that is more adapted to the geometry of the problem at hand, especially in non-Euclidean spaces.

Given a convex function \(\Phi\) of Legendre type, the Bregman divergence between two points \(x\) and \(y\) (where \(x \in \text{dom}(\Phi)\), and \(y \in \text{int(dom}(\Phi))\)) is defined as:
\[
D_\Phi(x, y) = \Phi(x) - \Phi(y) - \langle \nabla \Phi(y), x - y \rangle.
\]
This measures the difference between \(\Phi(x)\) and the first-order approximation of \(\Phi(x)\) around \(y\), capturing how different the structure of \(x\) and \(y\) is with respect to \(\Phi\)'s curvature.

The **Bregman projection** projects a point \(y\) onto a convex set \(C\) under the Bregman divergence induced by \(\Phi\):
\[
\text{proj}_{\Phi, C}(y) = \arg \min_{x \in C} D_\Phi(x, y).
\]
This operation is crucial in the mirror descent update step when we need to ensure iterates remain within a feasible region \(C\).

### Mirror Descent Algorithm

Given a convex function \(f\) and a closed convex set \(C\), MD proceeds in two main steps at each iteration \(k\):

1. **Dual variable update**:
   \[
   \nabla \Phi(y^{(k+1)}) = \nabla \Phi(x^{(k)}) - \eta_k g^{(k)},
   \]
   where \(\eta_k\) is the step size and \( g^{(k)} \) is the gradient (or sub-gradient) of \(f\) at \(x^{(k)}\).

2. **Bregman projection**:
   \[
   x^{(k+1)} = \text{proj}_{\Phi, C}(y^{(k+1)}).
   \]

In this process, \(\Phi\) is the **mirror map**, and \(\Phi^*\) is its **conjugate function**, defined as:
\[
\Phi^*(x^*) = \sup_{x \in \text{dom}(\Phi)} \{ \langle x, x^* \rangle - \Phi(x) \},
\]
which enables the primal-dual mapping structure. Additionally, if \(\Phi\) is differentiable, the relations \(\nabla \Phi^*(\nabla \Phi(x)) = x\) hold, facilitating practical implementations of the MD algorithm.

Using Bregman divergence, the MD update can be rewritten as:
\[
x^{(k+1)} = \arg \min_{x \in C} \left\{ \eta_k \langle g^{(k)}, x \rangle + D_\Phi(x, x^{(k)}) \right\}.
\]
This highlights the two parts of the update: minimizing the linearized form of the objective function \(\langle g^{(k)}, x \rangle\) and regularizing the move by penalizing large changes in terms of the Bregman divergence.

### Examples of Mirror Descent

The text gives three specific examples of MD algorithms applied to the problem of minimizing a convex function \(f(x)\) over the simplex \(\Delta\).

#### Example 2.1: Squared L2-norm

Here, the mirror map \(\Phi(x) = \frac{1}{2} \|x\|_2^2\) corresponds to standard Euclidean projection, leading to the **projected gradient descent algorithm**:
\[
x^{(k+1)} = \arg \min_{x \in \Delta} \|x - (x^{(k)} - \eta_k \nabla f(x^{(k)}))\|_2^2,
\]
which projects the new point \(x^{(k)} - \eta_k \nabla f(x^{(k)})\) back onto the simplex \(\Delta\).

#### Example 2.2: Negative Entropy on \( \mathbb{R}_+^n \)

Using the negative entropy function \(\Phi(x) = \sum_i (x_i \log x_i - x_i)\) as the mirror map leads to an **exponentiated gradient update**:
\[
x^{(k+1)} = \frac{x^{(k)} \exp(-\eta_k g^{(k)})}{\|x^{(k)} \exp(-\eta_k g^{(k)})\|_1},
\]
where the update is performed in the dual space using a multiplicative adjustment, followed by normalization to ensure the result remains on the simplex.

#### Example 2.3: Negative Entropy on the Simplex

This example refines Example 2.2 by restricting \(\Phi(x)\) to the simplex \(\Delta\). The resulting update is similar to Example 2.2:
\[
x^{(k+1)} = \frac{x^{(k)} \exp(-\eta_k g^{(k)})}{\|x^{(k)} \exp(-\eta_k g^{(k)})\|_1},
\]
but no projection step is necessary because the mirror map naturally keeps the iterates within the simplex.

### Remarks and Differences Between Examples

- **Example 2.1** corresponds to standard Euclidean geometry and results in classical projected gradient descent.
- **Examples 2.2 and 2.3** both lead to the same exponentiated gradient update rule, though they differ in terms of the underlying structure. Example 2.2 uses the negative entropy on \( \mathbb{R}_+^n \), while Example 2.3 focuses on the simplex itself. In Example 2.3, the mirror map is not differentiable, and the standard convex optimization machinery cannot be directly applied since \(\Phi\) in this case lacks a fully invertible gradient.

### Conclusion

Mirror descent generalizes traditional optimization approaches by leveraging problem-specific geometric information encoded in the mirror map \(\Phi\). This allows MD to adapt to different structures, such as probability distributions (simplex constraints) or non-Euclidean geometries, making it a flexible and powerful tool for various optimization problems. The specific examples discussed here—squared L2-norm and negative entropy—illustrate different ways MD can be tailored to problems that involve constraints like non-negativity or membership in the simplex.
====

## End-to-End Pseudo Algorithm Development for DAPO

The sources present DAPO (Dual Approximation Policy Optimization), a reinforcement learning framework that extends policy mirror descent methods to incorporate general function approximation. The algorithm leverages the concept of a Bregman projected policy class and utilizes the dual Bregman divergence for measuring the accuracy of function approximation. This response outlines the end-to-end pseudo-algorithm for DAPO, focusing on the DAPO-KL variant, along with its mathematical equations.

### Algorithm 1: Dual Approximation Policy Optimization (DAPO)

**Input:**

*   Initial policy π<sup>(0)</sup> with parameters θ<sup>(0)</sup>
*   Mirror map Φ
*   Learning rate schedule {η<sub>k</sub>}
*   Initial state distribution ρ ∈ Δ(S) (optional, for online training)
*   Reply buffer distribution ν<sup>(k)</sup> ∈ Δ(S) (optional, for off-policy training)

**Procedure:**

1.  **Initialization:** Set k = 0.
2.  **Loop:** Repeat until convergence or a predefined number of iterations:

    *   **Critic Update:** Find $\hat{Q}^{(k)}$ that approximates Q<sup>(k)</sup>.
    *   **Policy Update:**
        *   Find θ<sup>(k+1)</sup> by (approximately) solving:
            $$
            \min_{\theta \in \Theta} E_{s \sim \nu^{(k)}} \left[D_{\Phi^*} \left( \nabla \Phi(\pi^{(k)}_s) - \eta_k \hat{Q}^{(k)}_s, f_{\theta s} \right) \right],
            $$

            where:
            *   ν<sup>(k)</sup> can be replaced by d<sup>(k)</sup><sub>ρ</sub> for online training.
            *   $f_{\theta s}$ is a parameterized function used for approximating the dual update.

        *   Assign π<sup>(k+1)</sup><sub>s</sub> using the Bregman projection:
            $$
            \pi^{(k+1)}_s = \text{proj}^{\Delta(A)}_{\Phi} (\nabla \Phi^*(f_{\theta^{(k+1)}_s})), s \in S.
            $$
    *   Increment k: k ← k + 1.

**Output:**

*   Learned policy π<sup>(K)</sup> after K iterations