# Application of Reinforcement Learning in Decision Systems: Lift Control Case Study
## 4 The Proposed Solution
### Passengers Distribution  
A raw simulation is not sufficient to create an environment model that represents the real lift control problem. What is also needed is the distribution of passengers' arrivals depending on their route and time. To approximate such a distribution, real-world data from the lift control system reading in the intelligent building of the ASTOR sp. z o.o. company were used. With a mature approach to data and knowledge of their increasing value over time, ASTOR began archiving sensor readings in the entire building (over300,000 parameters) over a decade ago, including those from the lift control system using an advanced AVEVA Historian 2023 tool. The considered building comprises seven floors (from floor \(-1\) to \(5\)) and one cabin with stop buttons for every floor. On each floor there are two buttons, one associated with the intention to travel to a higher floor and the other to a lower floor (except for the highest and lowest floors, which have only one button). The available parameters for further analysis included the current cabin position and the current state of all the buttons. To count and classify the passengers, we used data ranging from August 2018 to August 2019. The assumption made for this purpose was that when a user enters a lift standing on floor \(A\), he immediately selects and presses the button \(Z_{B}\) inside the cabin corresponding to his destination floor \(B\). Such an assumption allowed, by sequentially analyzing the recorded parameter changes in time, for the association of each change in the activity of the \(Z_{x}\) stop button inside the cabin with a specific passenger travelling to floor \(x\) from floor \(y\), where the lift's last stop was registered. In this way, 65,142 passengers were counted and categorized based on their travel route, considering both the starting and destination floors, the specific day of the week, and the travel time. The passenger counts were averaged over four-second intervals, aligning with the typical duration of lift travel between two floors. This process resulted in the calculation of the average number of passenger appearances for each route within specific time slots. The derived distribution revealed that, on average, a passenger appeared once every 120 steps. To increase the dynamics of the environment and introduce a higher level of challenge for the learning agents, we reduced this time interval to 10 steps by multiplying the distribution values by 12. This adjusted distribution maintains real-world characteristics while providing a more demanding problem for the agents to learn a more interesting case for this study. The distribution, illustrated in Figure 1, indicates that almost all of the trips are the longer ones, either starting from the lower floors or concluding at the higher floors, represented by values in lower left and upper right corners. Additionally, the higher values are present in the upper right corner, meaning that a majority of trips start on either floor \(-1\) or \(0\) and head up to one of floors 3, 4 or 5. This is a common pattern, as people are more likely to call and wait for the cabin when going up than when going down [18]. An analysis of the summed values along the destination floor axis, presented in the last'sum' column, reveals that over half of the passengers start on either floor \(-1\) or \(0\)  
Figure 1: The distribution of the average number of passenger appearances for each possible route. Aggregated values for the starting and destination floors are presented in the last column and the last row, respectively. Regardless of the chosen route, the average number of passenger appearances at each step is indicated in the bottom-right corner. The values in the matrix are colour-coded according to a logarithmic colour map, with the colour bar located on the right side of the figure. At the same time, the remaining part is mostly concentrated around floor 4.