# Enhanced Routing Algorithm Based on Reinforcement Machine Learning--A Case of VoIP Service
## 1 Introduction  
Nowadays, there is a great demand for internet application services, such as video [1] and audio streaming, Voice-over-IP (VoIP) [2, 3], online games [4] among others. Multimedia services represent more than 50% of current Internet traffic [5]. VoIP service is one of the most popular communication services due to the low phone call rate compared to conventional telephony [6], but also due to the high speech quality level achieved in recent years [7]. Thus, network providers need to perform monitoring and operation tasks to ensure an acceptable end-user's Quality-of-Experience (QoE). In ad-hoc wireless networks, to ensure a reliable network performance is a great challenge due to the characteristic of this kind of network [8]. Dynamic topology, shared wireless channels, and limited node capabilities are factors that need to be considered in order to provide a high quality VoIP service. For instance, device batteries are limited resources that can lead to link losses connected to that nodes during power failures [9]. In a VoIP communication, end-user's QoE is determined by the user's perception [10, 11, 12]. In general, speech quality assessment methods can be divided in subjective and objectivemethods. Subjective methods are performed in a laboratory environment using a standardized test procedure [13]. Several listeners score an audio sample and the average value is computed and named Mean Opinion Score (MOS). However, subjective methods are time-consuming and expensive [14]. Another manner to predict the quality of a VoIP call is through a parametric method, such as the E-model algorithms [15; 16] which provides a conversation quality index estimated through different parameters related to acoustic environment, speech codec characteristics and network performance parameters. Several factors, such as channel transmission capacity, node processing capacity, and routing protocols affect network performance parameters [17]. Conventional routing protocols in ad-hoc networks, such as Optimized Link State Routing (OLSR), are unable to learn from abnormal network events that occurred several times in the past [18]; then, those protocols can choose a path that in the past had recurrent problems. For example, let us consider a path \(P\) where a given node \(N\) presents recurrent shut downs due to either device failures or programmed power-offs to save energy [19]. If a conventional protocol chooses this path \(P\), network degradation can occur, such as packet losses [20]. A routing protocol that is able to learn from previous network failure events could avoid this path improving the network performance. Hence, there is a need for protocols capable to learn from network data history. Therefore, it is important that routing protocols use strategies that make them learn from past experiences to choose optimal routing paths [21]. In the latest decades, Machine Learning algorithms have come to be used in several applications [22; 23; 24; 25; 26; 27; 28]. Thus, these algorithms can be applied into routing control protocols [29; 30; 31], specifically Reinforcement Learning (RL) is increasingly being used to solve routing problems [32; 33; 34]. In RL, an agent must be able to learn how to behave in a dynamic environment through iterations [35]. For instance, an agent who makes a choice receives a reward or a punishment whether the choice was good or bad, respectively. Hence, the RL technique can improve the steps along the decision making of path choice process, leading to better network performance, and consequently improved applications services, such as a VoIP communication [36]. In Reference [37], the authors introduce a generic model based on RL for ad-hoc networks focusing on routing strategies. Some works use RL for routing in urban vehicular ad-hoc networks (VANETs) [32]. Other works focus on wireless sensor networks and their characteristics [38] or unmanned robotic systems [39].