# HTTP Adaptive Streaming Framework with Online Reinforcement Learning
## 3 Framework Design
### Neural Network Model  
To learn the optimal policy, the neural network model must be properly configured. Neural network models should be large enough to generate optimal policies and low in complexity to minimize the load from training. We added a dropout layer and an early stopping technique to a simple structure using only a convolution layer and a fully-connected layer to increase the convergence speed of the model and reduce the variance of training [22; 23]. The proposed scheme uses a learning algorithm based on the actor-critic algorithm. This method trains the actor network and the critic network simultaneously. The actor network plays a role in determining the action given a state, and the critic network evaluates the current value of the state and passes this information to the actor network to help update the weights. The existing RL algorithm takes a large amount of time to learn and has a problem in that the training result does not converge. This is due to the fact that when training episodes are long or the optimization target is complex, trainingbecomes more distributed. On the other hand, the actor-critic algorithm has the advantage of reducing the variance of training through a complementary training process. Whenever a video segment is downloaded, the agent passes the state to the neural network model. Figure 3 shows how the state is passed to the neural network model, as well as the structure and output of the actor-critic network. A non-vector state is transmitted to a normal neuron, and a vector-type state is transmitted to a 1D-convolutional neural network (1D-CNN) neuron. The 1D-CNN layer is effective in extracting features for multiple inputs. The extracted features are transferred to the fully connected layer and used in the hidden layer. According to the parameters calculated in the hidden layer, the output layer of the actor network calculates the selection probability for video quality. As the calculated probability value selects a higher quality, the reward value increases, and a policy is created and trained based on it. The output layer of the critic network computes the current value of the state. This value is expressed as an integer and is used to update parameters in the actor network. The proposed scheme trains the policy using the policy gradient method. The policy gradient method estimates the gradient of the expected total reward by observing the actions taken in accordance with the policy [24]. By updating the parameters of the neural network model, the agent adjusts the neural network model to select high-reward tasks more frequently. The slope of the expected cumulative reward is calculated as in Equation (8) by considering the policy parameters. \[\
abla_{\theta}\mathrm{E}_{\pi\theta}\!\left[\sum_{l=0}^{\infty}\gamma^{t}r_{ t}\right]=\mathrm{E}_{\pi\theta}\!\left[\
abla_{\theta}\log\ \pi_{\theta}\!\left(s,a\right)A^{\pi_{\theta}}\!\left(s,a \right)\right] \tag{8}\]  
\(\gamma\) is a discount factor to control the effect of future reward, and \(r_{t}\) is the reward value at time step t. \(\theta\) means policy parameters, \(\ \pi_{\theta}\!\left(s,a\right)\) is a policy that performs action \(a\) according to state \(s\), and \(A^{\pi_{\theta}}\!\left(s,a\right)\) is an advantage function, meaning the difference between the predicted reward when selecting action \(a\) in state \(s\) and the predicted reward from the policy \(\ \pi_{\theta}\). That is, the advantage function compares how much better a specific action is compared to the average action taken according to the policy. Figure 3: Architecture of actor network and critic network. For the purpose of maximizing the agent's predicted cumulative reward, it is important to update the parameters for the actor network and the critic network. The expression for updating the parameters in the actor network is shown in Equation (9).