# Deep-Reinforcement-Learning-Based Planner for City Tours for Cruise Passengers +
## 5 Methods and Methodology  
### The Intelligent Planner  
The planner was implemented as a self-learning agent utilizing the deep Q-network (DQN) algorithm. To train this agent, an environment was created that simulates the dynamics of the city. This environment considers essential parameters for evaluating the reward and planning touristic itineraries. 1. _Stay time (T)_ represents the time passing from the disembarkation of the tourists from the ship up to the boarding after the tour. In our scenario, it was assumed that the tourists have up to 480 min (8 h) for their city tour;
2. _Current residual capacity_--\(CRC_{i},\quad i\in(0,N)\)--is the residual reception capacity of the \(i^{th}\) touristic attraction is determined by subtracting the current occupancy from the maximum nominal capacity. In our case, Naples has 14 suggested tourist attraction locations, denoted as \(N\). We collected data on the average occupancy of each location, representing the mean number of museum visitors at each hour. These mean values are varied on an episode-by-episode basis using a uniform distribution with a variance of \(\sigma^{2}=0.1\). 3. _Traveling time_--\(D(i,j),\quad i,j\in(0,N)\)--is the time needed by a bus to move from the \(i^{th}\) to the \(j^{th}\) location. Starting from a fixed baseline, for modeling the distances between each PoI, these values are varied according to a casual distribution with \(scale=2\) to emulate different traffic conditions (Figure 2);
4. _Visiting time_--\(TV_{i},\quad i\in(0,N)\)--is the mean time required for a tourist to visit the \(i^{th}\) touristic attraction. In our case, these values are kept constant during the agent's training. 5. _Current position (Bpos)_ is the current position of the bus;  
Figure 1: Dynamically planning a novel sub-optional route. _Cruise status_ (_Cstatus_) is the cruise ship status that may be _on time_ or _delayed_ if tourists are brought back to the port on time for boarding or not;
* _Bus tourists_ (_NTourists_) are tourists that will take the cruise bus for the city tour. * _Time to visit_ (_Time_) is the available time after a PoI is chosen. * _City Pols_ (_C_Pols_) is a list of relevant points of interest of the city to be visited. * _Pols visited_ (_PoI_Visited_) is a list that takes into account the current visited PoIs. The main characteristics underlying the behavior of the proposed adaptive planner are founded on the pseudo-code shown in Algorithm 1 and encompass the following ones:  
* A set of states \(s\in S\) where \(s\) is a vector containing the following information: \[s=[Time,NTourists,Bpos,PoI\_Visited,Cstatus]\] (5)
* A set of actions \(a\in A\) consisting of selecting a PoI from a list of Pols: \[a=pick\,a\,\text{\it Pol}\,\text{\it from}\,\,\text{\it C}\_Pols\]
* A reward \(r\) the agent receives for its choice evaluated according to the following rule: \[r_{t}= \begin{cases}w\in\mathbb{R}^{+}&\text{if a new PoI is chosen}\\ -10&\text{if an overcrowded PoI is chosen}\\ -100&\text{if the itinerary chosen causes a boarding delay}\end{cases} \] (6) where \(w\in\mathbb{R}^{+}\) represents the importance of a given city's tourist attraction. This reward function allows the agent to obtain a positive cumulative reward if and only if it avoids overcrowded places and composes an itinerary that brings the tourists back to the port before the ship's departure. Conversely, the agent is subjected to a penalty commensurate with the adverse effects generated by its decision. As the penalty's severity increases, the agent's motivation to engage in specific actions is diminished. * _Experience replay memory (RM)_ allows the agent to store its experiences, for performing Q-learning and updating its neural network. * _Target Network \(Q^{*}\)_: As is common in the DQN algorithm, the agent uses two DNNs to stabilize its learning process. The first one is the main neural network, represented by the weight vector \(\theta\), and it is used to estimate the Q-values for the current state s and action a: \(Q(s_{t},a;\theta)\).