Read the following Part I of the project I started implementing. I’ll provide you all the necessary code down below. Part I - Building the Environment

I implemented independent ROS nodes for part - I.
ROS Node - → Implemented a node that enables the car to scan its environment using a laser range finder and avoid obstacles in front of it.

Prerequisites
Dependencies: ROS (Robot Operating System) ; Gazebo (Simulation Environment)
The car uses Ackermann driving. You can drive the car by publishing AckermannDrive messages to the topic car_1/command .
The car is also equipped with a laser scanner which publishes LaserScan messages on the topic car_1/scan.
The laser scanner on the car has a field of view from -3π/4 to 3π/4 with a resolution of 0.0043633 radians.
The car publishes its odometry on the topic car_1/base/odom using an Odometry message (nav_msgs/odometry) . You can get the global position and orientation of the car from this message.
Setup

Launch the Gazebo simulator using
roslaunch audubon-gazebo world.launch. Command to use:
This will launch the testing environment in gazebo.
Now launch file pa2_spawner.launch to populate the testing space with randomly placed obstacles. Here is the command :
roslaunch audubon-gazebo spawner.launch
Once all obstacles are spawned, you can control the car using your keyboard.
Note : Gazebo terminal needs to be in focus to take keyboard inputs.


Part II - Implementing the Deep Reinforcement Learning Algorithm

Modify the below deep Q-network (DQN) to learn the control policy for robot navigation. Feel free to go out of the box for this task. You can also add or remove the layers to make it best suitable for our task.
class DQNAgentPytorch:
    def __init__(self, env, memory_size=10000, epsilon=1.0, gamma=0.99, learning_rate=0.001):
        self.env = env
        self.observation_space = env.observation_space
        self.action_space = env.action_space
        self.memory = deque(maxlen=memory_size)
        self.epsilon = epsilon
        self.gamma = gamma
        self.lr = learning_rate
        self.model = self.build_model()
        self.target_model = self.build_model()
        self.optimizer = optim.AdamW(self.model.parameters(), lr=self.lr, amsgrad= True)
        self.criterion = nn.SmoothL1Loss()
        self.update_target_net()

    def build_model(self):
        model = nn.Sequential(
            nn.Linear(self.observation_space.shape[0], 128),
            nn.ReLU(),
            nn.Linear(128, 128),
            nn.ReLU(),
            nn.Linear(128, self.action_space.n)
        )
        return model

    def remember(self, state, action, reward, next_state, done):
        self.memory.append((state, action, reward, next_state, done))

    def choose_action(self, state, test=False):
        if test or random.random() > self.epsilon:
            with torch.no_grad():
                state_t = torch.tensor(state, dtype=torch.float32).unsqueeze(0)
                q_values = self.model(state_t)
                return torch.argmax(q_values).item()
        else:
            return np.random.randint(self.action_space.n)

    def train(self, batch_size):
        if len(self.memory) < batch_size:
            return

        minibatch = random.sample(self.memory, batch_size)
        for state, action, reward, next_state, done in minibatch:
            target = self.target_model(torch.tensor(state, dtype=torch.float32).unsqueeze(0)).squeeze(0)
            if done:
                target[action] = reward
            else:
                next_q_values = self.target_model(torch.tensor(next_state, dtype=torch.float32).unsqueeze(0)).squeeze(0)
                target[action] = reward + self.gamma * torch.max(next_q_values)

            inputs = torch.tensor(state, dtype=torch.float32).unsqueeze(0)
            targets = target.unsqueeze(0)

            self.optimizer.zero_grad()

            outputs = self.model(inputs)
            loss = self.criterion(outputs, targets)
            loss.backward()
            
            # In-place gradient clipping
            torch.nn.utils.clip_grad_value_(self.model.parameters(), 100)
            self.optimizer.step()

    def update_target_net(self):
        self.target_model.load_state_dict(self.model.state_dict())

    def update_epsilon(self, new_epsilon):
        self.epsilon = new_epsilon

    def save_weights(self, file_name):
        torch.save(self.model.state_dict(), file_name)

    def load_weights(self, file_name):
        self.epsilon = 0
        self.model.load_state_dict(torch.load(file_name))
First, Make sure that the car be able to move using the control commands i.e linear and angular velocities based on the actions chosen by the DQNAgent.

Implement the training loop that consists of the following steps:
Read the sensor data from the robot’s sensors and preprocess it as the state input for the DQN.
Choose an action using the DQN and send it to the robot using the action server.
Calculate the reward based on the robot’s current position in the environment.
Update the DQN with the new state, action, and reward information.
Repeat until a certain number of episodes or a termination criterion is reached. Like Let's say if the car reaches the wall or boundary of our maze or if it reaches any random point given in the maze . You are free to go with the choice of your idea as well. So, feel free to put any criteria here.
Test the trained navigation policy in the Gazebo simulation to see how well the robot can navigate the maze.

Keep in mind that this is just a high-level outline, and the actual implementation will require you to put more effort, including experimenting with different DQN architectures, tuning hyperparameters, and potentially implementing methods to incorporate sensory feedback from the robot’s sensors to improve navigation performance.    Coming to the ROS Node Code:#! /usr/bin/env python3 

import rospy
from sensor_msgs.msg import LaserScan
from ackermann_msgs.msg import AckermannDrive
import random
import math


class EvaderNode:
    def __init__(self):
        # Initialize ROS node
        rospy.init_node('evader_node', anonymous=True)

        # Define subscriber to LaserScan topic
        rospy.Subscriber('car_1/scan', LaserScan, self.scan_callback)

        # Define publisher to AckermannDrive topic
        self.drive_pub = rospy.Publisher('car_1/command', AckermannDrive, queue_size=10)

        # Define distance and angle range for obstacle detection
        self.distance_threshold = 2  # meters
        self.angle_range = [(-150, -30), (30, 150)]  # degrees

    def find_min(self,lst):

        # Setting minimum distance to the maximum value of our laser's range
        self.min_dist = 3

        for i in lst:
            if (i < self.min_dist):
                self.min_dist = i
        return self.min_dist

    def scan_callback(self, scan):
        # Get minimum distance within angle range
        distances = scan.ranges
        min_distance = self.find_min(distances)
        print(min_distance)
        for angle_range in self.angle_range:
            min_angle = max(angle_range[0], scan.angle_min * 180 / 3.14159)
            max_angle = min(angle_range[1], scan.angle_max * 180 / 3.14159)
            idx_min = int((min_angle - scan.angle_min * 180 / 3.14159) / scan.angle_increment)
            idx_max = int((max_angle - scan.angle_min * 180 / 3.14159) / scan.angle_increment)
            print(idx_min,idx_max)
            if idx_min <= idx_max and idx_min >= 0 and idx_max < len(distances):
                min_distance = min(min_distance, min(distances[idx_min:idx_max+1]))
                print(min_distance)
        # Check if obstacle is present within distance threshold
        if min_distance > self.distance_threshold:
            # No obstacle, go forward at 2 m/s
            drive_msg = AckermannDrive()
            drive_msg.speed = 2.0
            drive_msg.steering_angle = 0.0
        else:
            # Obstacle detected, turn randomly in any direction and continue moving forward
            drive_msg = AckermannDrive()
            drive_msg.speed = 2.0
            drive_msg.steering_angle = float(random.uniform(0, math.pi/2))
            print(drive_msg)
        # Publish AckermannDrive message
        self.drive_pub.publish(drive_msg)


if __name__ == '__main__':
    try:
        node = EvaderNode()
        rospy.spin()
    except rospy.ROSInterruptException:
        pass

and also the spawner code is as follows: #! /usr/bin/env python3

import rospy
from gazebo_msgs.srv import DeleteModel, SpawnModel
from geometry_msgs.msg import *
from random import randint,uniform
from pathlib import Path
home = str(Path.home())


command = '0'
obj_dict = {'construction_barrel':0,'jersey_barrier':0}
keys = list(obj_dict)



def grid_spawn(command):
    obj_count = 0
    if command == '0':
        for i in range(-5,6,1):
            for j in range(-5,6,1):
                if abs(i)<1 and abs(j)<1:
                    pass
                else:
                    idx = randint(0,len(keys)-1)
                    # print(idx)
                    selected_obj = keys[idx]
                    with open(home+"/.gazebo/models/{}/model.sdf".format(selected_obj), "r") as f:
                        product_xml = f.read()
                        
                        print(obj_dict[selected_obj])

                        item_name   =   "obj_{}".format(obj_count)
                        obj_count+=1
                        # print("Spawning model:{}".format(item_name))
                        item_pose   =   Pose()
                        if selected_obj == 'dumpster':
                            item_pose_position = Point(i*8,j*8,0.5)
                            item_pose_orientation = Quaternion(uniform(0,1),0,0,uniform(0,1))
                        else:
                            item_pose_position = Point(i*8,j*8,0)
                            item_pose_orientation = Quaternion(0,0,uniform(0,1),uniform(0,1))
                        item_pose.position = item_pose_position
                        item_pose.orientation = item_pose_orientation

                        spawn_model(item_name, product_xml, "", item_pose, "world")

    else:
        for i in range(1000):
            name = "obj_{}".format(i)
            delete_model(name)



if __name__ == '__main__':
    print("Waiting for gazebo services...")
    rospy.init_node("spawn_products_in_bins")
    rospy.wait_for_service("gazebo/delete_model")
    rospy.wait_for_service("gazebo/spawn_sdf_model")
    print("Got it.")
    delete_model = rospy.ServiceProxy("gazebo/delete_model", DeleteModel)
    spawn_model = rospy.ServiceProxy("gazebo/spawn_sdf_model", SpawnModel)
    obj_count = 0
    grid_spawn(command)