# DDPG-MPCC: An Experience Driven Multipath Performance Oriented Congestion Control
## 2 Experience Driven MPCC Design  
We first formulate a Markov Decision Process (MDP) to model the learning process of the proposed DDPG-MPCC. MDP formalizes the environment for DDPG learning. The learning process starts by observing the state of the environment \(s^{j}(t)\) from the MPTCP connection of the subflow \(j\) and decides by taking an action according to the policy \(\pi(a\mid s)\). The action of the DDPG agent changes the environment state to \(s^{j}(t+1)\), and the model receives a reward \(R(t)\). The state of the environment is represented by the tuple: sending rate \(x^{j}(t)\), round trip time \(RTT^{j}(t)\), loss ratio \(L^{j}(t)\) of each MPTCP subflow. Actions are represented by sending rate adjustments (increments/decrements) over an MPTCP subflow. The proposed DDPG agent consists of three main components, (i) State space: \(s_{t}=\left[x^{j}(t),RTT^{j}(t),L(t)\right]\) (ii) Action space: \(a_{t}=\left[x_{1},x_{2},\ldots,x^{j}\right]\); and (iii) Reward: for simplicity, reward \(R=U^{j}\) (given by (1)). For simplicity, we use the MPCC utility as a reward after each M.  
DDPG is explicitly adapted for continuous action spaces and therefore has the potential to transform the MPCC. In our design of DDPG-MPCC, we have flexibly coupled a deterministic policy network (_Actor_) and Q-network (_Critic_). The underlying idea is to capture the network dynamics inherently by using a policy iterating mechanism, which is applied to alternate between policy improvement (actor) and policy evaluation (critic). It is worth noting that the actor is meant to improve the policy using a policy gradient, while the critic evaluates the policy for the current parameters. DDPG allows the implementation of an experienced pool and two target networks to improve stability and convergence. For higher performance and better convergence, we adopt replay buffers, where a set of previous experiences is stored. It contains a wide array of experiences, though it may not be suitable to keep everything as using too much experience may decelerate the learning process. Due to time-varying network dynamics, the target function fluctuates, and the training needs to cope with such fluctuations. Therefore, we adopt the target network that fixes the parameters of the target function for a fixed number of epochs and then replaces them with a newly updated target network. We formulate an actor and critic model by representing policy function and Q-network parameters as:  
\[policy\Bigg{\{}\begin{aligned} & online:\mu(s|\theta^{\mu})\; gradient\;update\;\theta^{\mu}\\ & target:\mu\left(s|\theta^{\mu^{\prime}}\right)\;soft\;update\; \theta^{\mu^{\prime}}\\ & Q\;network\Bigg{\{} \begin{aligned} & online:Q(s,a|\theta^{Q})\; gradient\;update\;\theta^{Q}\\ & target:Q\left(s,a|\theta^{Q^{\prime}}\right)\;soft\;update\; \theta^{Q^{\prime}}\\ \end{aligned} \]  
The policy objective function, \(J(\theta^{\mu})\), for the actor-network is computed as a weighted cumulative expected reward from all subflows:  
\[J(\theta^{\mu})=\mathbb{E}_{\theta^{\mu}}\Big{[}R_{1}+\gamma R_{2}+\gamma^{2 }R_{3}+\ldots\Big{]}\]  
with \(\gamma\) exponentially decreasing the future rewards' weight. With relevant insights from [19], we assume that the change in the policy gradient with respect to \(\theta^{\mu}\) is approximately equal to the considered gradient of the Q-valued function. Therefore, the policy gradient of \(J(\theta^{\mu})\) can be estimated by computing the expectation  
\[\frac{\partial J(\theta^{\mu})}{\partial\theta^{\mu}}=\mathbb{E}_{s}\left[ \frac{\partial Q\big{(}s,a|\theta^{Q}\big{)}}{\partial a}\frac{\partial\mu(s| \theta^{\mu})}{\partial\theta^{\mu}}\right]\]Similarly, the critic network receives the action and state spaces and computes \(Q(s,a)\) for evaluating the impacts of taking action \(a\). It updates its parameters on the basis of the target value.