Hi, I got a wonderful task for you to implement. In the past you helped me implement many such tasks. I hope you have great time implementing this as well. TASK: The goal of this task which comprises of two parts is to implement the actor-critic algorithm and apply it to solve OpenAI gym environments. I'll provide you the structure of the implementation.                                                                                                                                                                                                                                                                      
Part I - Implementing Q Actor-Critic                                                                                                            
 Here is the Algorithm for  Q Actor Critic
Initialize parameters s, 0, w and learning rates ae, aw; sample a~ To (als). for t 1...T: do
s'~ P(s's, a) Sample reward rt ~ R(s, a) and next state s'
Then sample the next action a'~ Te (a's')
Update the policy parameters: ←0 + αo Qw (s, a) Ve log Te (als); Compute the correction (TD error) for action-value at time t:
dt = rt+YQw (s', a') - Qw (s, a) and use it to update the parameters of Q function:
ww+αw St VwQw (s, a)
Move to a a' and s← s' end for
In this part we will implement an actor-critic algorithm and test it on any simple environment which I'll provide you down below.
1. Implement the Q actor-critic algorithm. You may use any framework (Keras/Tensorflow/Pytorch).Choose the best , efficient,  framework for our task.
In this Part we need to implement only Q Actor-Critic version. Any other variations will not be allowed.
2. Train your implemented algorithm on the GridWorld Env given below.
3. Plots should include the total reward per episode and other metrics to check the performance of our model.
4. Provide the evaluation results. Run your environment for at least 10 episodes, where the agent chooses
only greedy actions from the learnt policy. Plot should include the total reward per episode and.
I want yout to implement one Q actor class, one train function and evaluate function. It's your choice to implement both train and evaluate as seperate funcs or you can implement a single function where I can toggle between both like saving weights and loading weights making the agent choose greedy actions.
Part II - Solving Complex Environments
In this part, test the Q actor-critic algorithm implemented in Part I on any other two complex environments.
1. Choose an environment. At least one of the environments has to be among "Suggested environments -
Part II".  Pick the easy environments among those suggested envs for part-II. The environment with multiple versions is considered as one environment.
2. Apply the Q actor-critic algorithm to solve it. You can adjust the neural network structure or
hyperparameters from your base implementation.
3. Show and discuss your results after applying the Q actor-critic implementation on the environment.
Plots should include the total reward per episode.
4. Provide the evaluation results. Run your environment for at least 10 episodes, where the agent chooses
only greedy actions from the learnt policy. Plot should include the total reward per episode.
5. Go to Step 1. Provide the results for TWO environments.
Suggested environments
Part I
• GridWorld env,
class GridWorldDeterministic(gym.Env):

    def __init__(self):
        self.grid = np.zeros((4, 4))
        self._initialize_grid()

        self.observation_space = gym.spaces.Discrete(16)
        self.action_space = gym.spaces.Discrete(4)
        self.reward_range = (-5, 10)
        self.agent_pos = (0, 0)

    def _initialize_grid(self):
      
        self.grid[1, 1] = -1
        self.grid[1, 2] = -1
        self.grid[1, 3] = 10
        self.grid[0, 1] = -5
        self.grid[2, 1] = -5

    def reset(self):
        self.agent_pos = (0, 0)
        return self._flatten_pos(self.agent_pos)

    def step(self, action):
        x, y = self.agent_pos

        x, y = self._apply_action(action, x, y)

        # Ensure agent does not move outside the grid boundaries
        x, y = self._clip_pos(x, y)

        reward = self.grid[x, y] if self.grid[x, y] in (-5, 10) else -1
        next_state = self._flatten_pos((x, y))
        done = self.grid[x, y] == 10

        self.agent_pos = (x, y)
        return next_state, reward, done, {}

    def _apply_action(self, action, x, y):
        "Returns new position after applying the action."
        if action == 0:  # up
            x -= 1
        elif action == 1:  # down
            x += 1
        elif action == 2:  # left
            y -= 1
        elif action == 3:  # right
            y += 1
        return x, y

    def _clip_pos(self, x, y):
        "Ensure position does not move outside the grid boundaries."
        return np.clip(x, 0, 3), np.clip(y, 0, 3)

    def _flatten_pos(self, pos):
        "Convert 2D position to 1D index."
        row, column = pos
        return row * 4 + column

    def render(self):
        print(self.grid)
        print("Agent position:", self.agent_pos)
Part II
• Any multi-agent environment
• Lunar Lander
• Car Racing
• Bipedal Walker
• MuJoCo Ant
• Any Atari env (more details)