This is my question for part 1 which you have already given the code and worked ---------------------Part I: Define an RL Environment [30 points]
In this part, we will define a grid-world reinforcement learning environment as an MDP.
While building an RL environment, you need to define possible states, actions, rewards
and other parameters.
STEPS:
1. Choose a scenario for your grid world. You are welcome to use the
visualizationdemo as a reference to visualize it.
An example of idea for RL environment:
• Theme: Lawnmower Grid World with
batteries as positive rewards and rocks as
negative rewards.
• States: {S1 = (0,0), S2 = (0,1), S3 = (0,2),
S4 = (0,3), S5 = (1,0), S6 = (1,1), S7 =
(1,2), S8 = (1,3), S9 = (2,0), S10 = (2,1),
S11 = (2,2), S12 = (2,3), S13 = (3,0), S14 =
(3,1), S15 = (3,2), S16 = (3,3)}
• Actions: {Up, Down, Right, Left}
• Rewards: {-5, -6, +5, +6}
• Objective: Reach the goal state with
maximum reward
2. Define an RL environment following the scenario that you chose.
Environment requirements:
• Min number of states: 12
• Min number of actions: 4
• Min number of rewards: 4
Environment definition should follow the OpenAI Gym structure, which includes
thebasic methods. You can use the “Defining RL env” demo as a base code.
def __init__:
# Initializes the class
# Define action and observation space
def step:
# Executes one timestep within the environment
# Input to the function is an action
def reset:
# Resets the state of the environment to an initial state
def render:
# Visualizes the environment
# Any form like vector representation or visualizing
usingmatplotlib is sufficient
3. Run a random agent for at least 10 timesteps to show that the environment logic
is defined correctly. Print the current state, chosen action, reward and return your
grid world visualization for each step         -------------------------------------------------------------------------------------------------------------------------------------------------------------------import numpy as np
import gym
from gym import spaces
import matplotlib.pyplot as plt

class LawnmowerGridWorld(gym.Env):
    def __init__(self):
        # Define action and observation space
        self.observation_space = spaces.Discrete(16)
        self.action_space = spaces.Discrete(4)
        self.state_matrix = np.zeros((4, 4))

        # Define rewards
        self.rewards = {-5: 'Rock', -6: 'Rock', 5: 'Battery', 6: 'Battery'}

        # Randomly initialize rewardable slots
        for _ in range(4): # 4 rewards
            x, y = np.random.randint(0, 4, 2)
            reward = np.random.choice(list(self.rewards.keys()))
            self.state_matrix[x, y] = reward

        self.agent_pos = (0, 0)
        self.goal = (3, 3)

    def step(self, action):
        x, y = self.agent_pos

        if action == 0:  # Up
            x = max(0, x - 1)
        elif action == 1:  # Down
            x = min(3, x + 1)
        elif action == 2:  # Right
            y = min(3, y + 1)
        else:  # Left
            y = max(0, y - 1)

        self.agent_pos = (x, y)
        reward = self.state_matrix[x, y]
        self.state_matrix[x, y] = 0  # Remove the reward after collecting

        done = (x, y) == self.goal

        return self.agent_pos, reward, done, {}

    def reset(self):
        self.agent_pos = (0, 0)
        return self.agent_pos

    def render(self):
        grid = np.zeros_like(self.state_matrix)
        for x in range(4):
            for y in range(4):
                if self.state_matrix[x, y] != 0:
                    grid[x, y] = list(self.rewards.keys()).index(self.state_matrix[x, y]) + 1

        grid[self.goal] = 9
        grid[self.agent_pos] = 8

        plt.imshow(grid, cmap='viridis', extent=(-0.5, 3.5, 3.5, -0.5))
        plt.xticks(range(4))
        plt.yticks(range(4))
        plt.grid(True, which='both', linestyle='-', linewidth=0.5)

        plt.show()
        return grid

# Test Environment
env = LawnmowerGridWorld()
state = env.reset()
print('Initial State:', state)
env.render()

# Run Random Agent
for i in range(10):
    action = np.random.choice([0, 1, 2, 3])
    state, reward, done, _ = env.step(action)
    print('After step {}: State={}, Action={}, Reward={}, Done={}'.format(i+1, state, action, reward, done))
    env.render()

    if done:
        break

env.close()                                                                                                                                                                                                                                                    
                   -----------------------------------------------------I need you to give the code for Part II: Solve your environment using –----------------------------------------------------------------------------------------
SARSA [40 points]
In this part, we implement SARSA (State-Action-Reward-State-Action) algorithm and
apply it to solve the env defined in Part 1.
SARSA is an on-policy reinforcement learning algorithm. The agent updates its Q-values
based on the current state, action, reward, and next state, action pair. It uses an
exploration-exploitation strategy to balance between exploring new actions and exploiting
the knowledge gained so far.
STEPS:
1. Apply SARSA algorithm to solve the environment that was defined in Part I.
2. Try hyperparameter tuning on at least two parameters to get better results for
SARSA. You can explore hyperparameter tuning libraries, e.g. Optuna or make it
manually. Parameters to tune:
a. Discount factor (γ)
b. Epsilon decay rate
c. Epsilon min/max values
d. Number of episodes
e. Max timesteps
Try at least 3 different values for each of the parameters that you choose.
3. Provide the reward graphs and your explanation for each result. In total, you
should have at least 3 graphs and your explanations. Make your suggestion
on the most efficient hyperparameters values for your problem setup.
-----------