# Collaborative Caching in Edge Computing via Federated Learning and Deep Reinforcement Learning

Authors: Yali Wang; Jiachao Chen
Journal: Wireless Communications and Mobile Computing
 (2022)

---

## Abstract

By deploying resources in the vicinity of users, edge caching can substantially reduce the latency for users to retrieve content and relieve the pressure on the backbone network. Due to the capacity limitation of caching and the dynamic nature of user requests, how to allocate caching resources reasonably must be considered. Some edge caching studies improve network performance by predicting content popularity and actively caching the most popular content, thereby ignoring the privacy and security issues caused by the need to collect user information at the central unit. To this end, a collaborative caching strategy based on federated learning is proposed. First, federated learning is used to make distributed predictions of the preferences of users in the nodes to develop an effective content caching policy. Then, the problem of allocating caching resources to optimize the cost of video providers is formulated as a Markov decision process, and a reinforcement learning method is used to optimize the caching decisions. Compared with several basic caching strategies in terms of cache hit rate, transmission delay, and cost, the simulation results show that the proposed content caching strategy reduces the cost of video providers, and has higher cache hit rate and lower average transmission delay.


---

## Body

## 1. Introduction

The explosive growth of mobile devices, including cell phones, wearable devices, connected cars, and Internet-of-Things (IoT) devices, has led to exponential growth in data traffic. As a result, great pressure has been exerted on the backhaul network, with resultant increases in network latency. Resource-constrained mobile devices also face considerable challenges in supporting computation-intensive and time-critical applications, such as video services, voice control, gesture recognition, 3D modeling, natural language processing, and online interactive games. On the other hand, video data is also growing explosively. With substantial storage space and powerful computing ability, cloud data centers are the best content repository for some video vendors, such as YouTube and TikTok. However, mobile devices suffer performance degradation when retrieving all data from the cloud.To address these issues, edge computing (EC), which is a promising paradigm, has been introduced to provide a service environment with computing and caching capacity. By hiring computing and storage resources on edge servers, mobile apps and content vendors (referred to as vendors hereafter) can host their apps and content on edge servers to provide low-latency and high-quality services for users [1]. In this way, EC can greatly alleviate the congestion of the backhaul network, improve the quality of user experience (QoE) by meeting the strict requirement of response delay, and enhance location awareness.Compared to cloud computing, MEC is still constrained by limited storage capacity [2]. Edge servers can cache only part of the content, and video users can retrieve data from nearby edge servers instead of from remote cloud servers if the data are already cached on those edge servers [3]. Edge servers are prone to storing popular content to obtain higher hit ratios. Thus, content popularity prediction [4] and cache cooperation strategies between edge servers are important to improve caching performance.In recent years, traditional caching strategies, such as the least recent use (LRU) and least frequent use (LFU) approaches, have been extensively studied [5, 6]. However, these methods are not very efficient because they do not take content popularity into account. Most of the existing work on edge caching handles the popularity based on the assumption that the content popularity is known in advance, which is impractical. Therefore, it is imperative to improve the caching efficiency by properly predicting content popularity [7]. Recently, centralized machine learning can learn on its own from available data and make accurate decisions or predictions on unseen data with the help of algorithms [8, 9]. In literature [10], an optimal cache resource allocation scheme during the next update period is drawn using a neural network from the content history requests collected from all users. In literature [11], a deep learning-based content popularity prediction scheme is proposed. In literature [12], the authors propose a popularity prediction model for each content category in terms of historical popularity by training a simplified bidirectional long and short-term memory (Bi-LSTM) network. In literature [13], the authors propose an evolutionary learning-based content caching strategy that adaptively learns content popularity over time. In literature [14], a user preference model is proposed to predict content popularity and track popularity changes based on user preferences and the features of the requested content.However, most existing caching schemes require users to share their private preference data with a central server, which may pose privacy and security risks [15]. Federated learning (FL) is a distributed framework that learns global models on a server while protecting participants’ privacy by allowing each participant to share locally trained model parameters with the server instead of their local data. A mechanism to protect user privacy is introduced in the literature [16], where the content popularity obtained by the user’s local model is weighted with user preferences. The authors in the literature [17] propose an FL-based model aggregation method that divides all users into multiple subspaces based on their contextual information.In addition, most of the above articles consider the user’s perspective to optimize user request latency and improve the QoE. However, for content providers, how to increase profit and reduce costs are additional factors to consider. In literature [18], the multicell collaborative caching problem is studied to minimize the total cost for content providers. The literature [19] considers the caching resource allocation problem in a scenario where a network operator coexists with multiple content providers. This problem is modeled as a multileader multifollower Stackelberg game, and the optimal caching strategy for the content providers is obtained.In this paper, we propose a cooperative caching strategy in edge caching by considering the cost of optimizing content providers and the privacy security of users. Aiming to ensure the privacy security of users, we adopt FL to predict user preferences in different edge nodes in a distributed manner and apply the approach to the design of the caching policy. For the cost problem for content providers, we use a deep reinforcement learning (DRL) based collaborative caching (Dueling DDQN) algorithm, which combines DDQN and Dueling DQN to make caching decisions for edge nodes. The contributions of this paper are as follows:(i)
Due to the dynamic changes in video popularity on the network, the FL framework is used to accurately predict the content popularity in a given region. The proposed model can learn the influence of multidimensional content features to formulate an effective caching strategy(ii)
To reduce the cost of video providers, a video collaborative caching strategy based on deep reinforcement learning is proposed. We construct a collaborative edge cache model based on transforming storage and energy consumption into costs. By combining DDQN and Dueling DQN, the proposed algorithm can effectively reduce the high valuation of DQN and accelerate the convergence speed. The experimental results verifies the validity of our approach(iii)
Experimental verification and comparative analysis are conducted on the proposed cache strategy, and our strategy is compared with other baseline algorithms. The experimental results show that our strategy optimizes the cost of video providers and has a high cache hit ratioThe rest of this paper is organized as follows: Section2 summarizes the relevant work; Section 3 defines the system model and problem description; Section 4 details the proposed cache collaboration scheme; Section 5 uses simulation results to evaluate the cache performance; finally, Section 6 summarizes this paper.


## 2. Related Work

Many different strategies and algorithms have been for edge caching. In [20, 21], the user can only obtain the requested content from the local base station or cloud data center, while the noncooperative content caching scheme that cannot be obtained from the nonlocal base station is proposed. Since the cache capacity of a single base station is limited, to alleviate the storage capacity limitation of edge cache nodes, a collaborative cache strategy is adopted to solve the problem that limited cache node capacity affects cache efficiency and degrades system performance. Literature [22] proposes a cooperative cache strategy for heterogeneous cellular networks by transforming the optimal strategy design of content caching into an integer linear programming problem that is solved by the subgradient method. Literature [23] describes the problem of optimal collaborative content caching as a 0-1 integer program to minimize the average download delay. A greedy algorithm based on content popularity is used to solve the problem. Compared with the popular cache strategy, this strategy can substantially improve the content cache hit ratio and reduce the average content delivery delay.Due to the dynamic nature of the network and the complexity of the environment, an efficient cache strategy must take user requirements into account. However, accurate user requirements are difficult to obtain. Literature [10–14] design cache strategies based on machine learning technology, estimating the future needs of users according to the content popularity and user preference data set with dynamic characteristics of the time series. Literature [24] proposes an online active cache scheme using the bidirectional deep recursive neural network (BRNN) model to predict time series content requests and update the edge cache accordingly. Literature [25] proposes a collaborative cache strategy of edge servers based on the software-defined network (SDN), in which multilayer sensory neural networks are used to predict video content request probability by mobile users and to construct an objective minimization function to maximize the utilization of edge servers’ resources. The approach adopts a branch-and-bound algorithm to determine the optimal global solution. Literature [26] proposes a deep reinforcement learning approach and an improved branch delimitation strategy to solve the problem of jointly optimizing cooperative edge caching and wireless resource allocation in IoT networks, respectively. However, centralized access to user data may lead to user privacy exposure. Federated learning is a distributed machine learning framework that can effectively solve this problem [16, 17]. Thus, it is necessary to extract, analyze, and identify content popularity, user preferences, and user movement patterns while respecting user privacy. Literature [27] proposes a federated K-means scheme for privacy protection that is used for active caching in the next-generation cellular network. This scheme protects user privacy by means of two privacy-protection technologies: FL and secret sharing. Literature [28, 29] propose an intelligent F-RANS framework based on FL that can accurately predict the content popularity distribution in a network by applying FL to user demand prediction.Due to the limited storage capacity of edge servers, it is impossible to guarantee that all content provided by the content provider (CP) will be cached to the edge server, and a user’s request will directly affect the income of the CP. Therefore, the study of collaborative cache strategy must also consider maximizing the income of the CP [18, 19]. To reduce the delay and cost in the cloud-side collaboration environment, literature [30] proposes a content collaboration caching strategy. The strategy considers the delay gain and cache cost gain brought by the cached content to the cooperative domain and designs the cached content according to the gain. Because of the considerable request delay problems and high operation costs in the current video service-based caching strategy, literature [31] proposes a moving edge computing video caching strategy with coordinated optimization of delay and energy consumption. The method adopts a branch-and-bound algorithm to solve the optimization problem with the aim of reducing request delays for users and lowering costs for suppliers. This paper proposes an edge cache strategy algorithm based on reinforcement learning that considers cache placement and privacy security while protecting user privacy and reducing the costs of video providers. The recent work summary can be seen in Table 1.Table 1
Comparison of existing papers addressing edge caching problems.
ReferenceOptimization objectiveMethodDisadvantages[22]Download latencyHungarian algorithmNo quantitative benefits[23]Cache hit ratioGreedy algorithmHigh complexity[24]Cache hit ratioBidirectional recurrent neural networksPrivacy security[25]Energy consumptionBranch and bound algorithmPrivacy security[26]Estimating content popularityFederatedk-means schemeHigh complexity[27]Minimize traffic costFederated learningLower model accuracy[29]User response latencyHeuristic algorithmHomogeneous user demand distribution[30]The cost of the video providerBranch and bound algorithmHigh time consuming


## 3. System Architecture

In this section, we introduce a collaborative edge caching system architecture that supports FL frameworks. The architecture consists of a network model, content request model, local popularity model, cooperative cache model, and request cost model. The main parameter symbols are listed in Table2.Table 2
Key notations and descriptions.
NotationDescriptionMSet of BSsUSet of UEFSet of contentUmtSet of UE requesting content fromBSm in period tRu,mtRequest vector for each UE in slottPmtLocal content popularity for all content inBSm in time slot txu,m,ft,xu,i,ft,xu,c,ftContent delivery variables via local, collaborative and clouddm,fCaching decisions for contentfzfSize of contentpEnergy consumption of each bit of data cached by MEC serverpu,m,gu,mThe transmission power fromBSm to UEu, the channel gain between BSm and UEuσ2The variance of additive Gaussian white noiseϕo,m, ϕc,oTransmission rate fromBSm to UEu, transfer rate from CM to cloud serverEm,f,Ei,f,Ec,fCosts delivered via on-premises, collaboration and cloudcosttotal,costca,costtranTotal cost, cache cost, transmission costxfnThe feature vector of the contentfyfThe category label of the contentfpt,u,f,p^t,u,fThe probability that useru in time slot t requests content f, the predicted probability that user u in time slot t requests content fRuThe cumulative number of requested samples for useruwu,wur+1wmThe user preference model parameter vector of useru, the user preference model parameter vector learned by user u at the r-th iteration. The parameter vector of the regional integrated modelLosswu,xf,yfThe logistic loss of userugrg1:rThe gradient vector of ther-th sample with respect to wu’s logistic loss, the sum of the gradient vectors of the logistic loss of the first r samplesλ1,λ2; α,βPositive regularization parameter; tuning parametersηr,ηr,n,σThe nonincreasing learning rate, the learning rate of then-th feature, and the parameter related to the learning rate ηrNFeature dimension


### 3.1. Network Model

Figure1 shows the system scenario. This scenario contains a set of BSs M1,2,⋯,f,⋯,M, and UEs U1,2,⋯,u,⋯,U. Each BS can serve multiple UEs that have disjoint coverage areas. Mobile edge servers are deployed at each BS to provide edge caching service for UEs and to make wireless resource allocation decisions. Let sM be the caching capacity of BSm; then, all BSs capacity sets can be defined as S=s1,s2,⋯,sM. All the MEC servers can exchange cache information and share data through backhaul links and the cache manager (CM). Mobile devices send content requests at the beginning of each time slot t.Figure 1
Illustration of the edge cache scenarios.


### 3.2. Content Request Model

The content repository set located in the cloud server is denoted asF=1,2,⋯,F, and the size of content f∈F is denoted as zf. The set of requesting users at BSm in time slot t is defined as Umt, and the number of UEs requesting content from BSm during time slot t is defined as Nmt. Each UE is assumed to be associated with only one BS in each time slot, i.e., each UE can be served by only one BS in a given time slot. The number of requests from all users can be defined as Ru,mt=R1,mt,R2,mt,⋯,Ru,mt,⋯,RU,mt.


### 3.3. Local Popularity Model

Due to the diversity of the content preferences in different BSs, the local content popularity of all content inBSm at time slot t is defined as a content popularity vector Pmt=Pm,1t,Pm,1t,⋯,Pm,ft,⋯,Pm,Ft. In addition, considering the privacy and security of users, FL is applied to accurately predict content popularity without the UE uploading all individual user preference data to the BS.


### 3.4. Cooperative Cache Model

The content requests of mobile devices are first received by BSs. If the requested content has been cached in the local BS, it will be pushed to users immediately. A binary local content delivery variablexu,m,ft∈0,1 indicates whether the local BS provides services for the UE: xu,m,ft=1 if a response is requested and xu,m,ft=0 otherwise. If the requested content is not cached in the local BS, the BS will obtain the requested content from other BSs through the CM. Let xu,i,ft∈0,1i∈M,i≠m denote a nonlocal BS serving the UE: xu,i,ft=1 if another BS responds to the request and xu,i,ft=0 otherwise. If the CM cannot find the requested content in any BS, the local BS will obtain the requested content from the cloud server and deliver it to the UE. Let xu,c,ft∈0,1 denote whether the UE obtains the content from cloud server c in time slot t: xu,i,ft=1 if the UE obtains content f from the cloud server and xu,i,ft=0 otherwise.


### 3.5. Request Cost Model

The cost of the system is composed of two main parts: the storage energy consumption and the transmission energy consumption of the content on the MEC. If the content is cached in the BS, there will be additional storage energy consumption. Assuming that all BS servers have the same performance, the storage cost of BS caching content [31] within period T can be expressed as
(1)costca=∑f=1Fdm,f·zf·p·Twhere the caching decision dm,f∈0,1 indicates whether content f is cached in BSm and p is the energy consumption of the MEC server to cache each bit of data. The wireless transmission rate between BSm and UEu can be obtained by the Shannon formula:
(2)ϕu,m=Blog21+pu,mgu,mσ2,where B represents the bandwidth of the base station, pu,m denotes the transmission power from BSm to UEu, gu,m is the channel gain between BSm and UEu, and σ2 is the variance of additive Gaussian white noise. Then, the transmission energy consumption of UEu to download content f from its local BSm is
(3)Em,f=zfϕu,m·pe,where zf/ϕu,m denotes the time taken for file f to be transmitted from the user to the local base station, pe is the transmission power between BSs. We define the transmission rate between a BS and the CM as ϕo,m and that between the CM and the cloud server as ϕc,o. The transmission energy consumption of UE acquiring content from nonlocal BSs is expressed as:
(4)Ei,f=2zfϕc,m·pe+Em,f,where zf/ϕc,m denotes the time taken for file f to be transmitted from CM to the nonlocal base station. The energy consumption of UE downloading content f from cloud server c can be expressed as:
(5)Ec,f=zfϕc,o·pc+zfϕo,m·pe+Em,f,where pc is the transmission power of the cloud server. ϕo,m>ϕc,o can be seen from the transmission rate, so the transmission energy consumption is Ec,f>Ei,f>Em,f. Therefore, the transmission cost can be expressed as follows:
(6)costtran=∑u=1Nmt∑f=1Fxu,m,ft·Em,f+xu,i,ft·Ei,f+xu,c,ft·Ec,f.Thus, the total cost can be expressed as(7)costtotal=costca+costtran.


### 3.6. Formalization

In this paper, the provider cost-based edge cocaching approach aims to minimize the total cost by efficiently caching content on edge servers. This problem can be expressed mathematically as:(8)min∑m=1Mcosttotal,s.t.C1:dm,f∈0,1,C2:xu,m,ft,xu,i,ft,xu,c,ft∈0,1,C3:∑f=1Fdm,f·zf≤sm.where constraints C1 and C2 indicate that the cache decision and content delivery variables are binary. C3 indicates that the data in each BS should not exceed its storage capacity.


## 3.1. Network Model

Figure1 shows the system scenario. This scenario contains a set of BSs M1,2,⋯,f,⋯,M, and UEs U1,2,⋯,u,⋯,U. Each BS can serve multiple UEs that have disjoint coverage areas. Mobile edge servers are deployed at each BS to provide edge caching service for UEs and to make wireless resource allocation decisions. Let sM be the caching capacity of BSm; then, all BSs capacity sets can be defined as S=s1,s2,⋯,sM. All the MEC servers can exchange cache information and share data through backhaul links and the cache manager (CM). Mobile devices send content requests at the beginning of each time slot t.Figure 1
Illustration of the edge cache scenarios.


## 3.2. Content Request Model

The content repository set located in the cloud server is denoted asF=1,2,⋯,F, and the size of content f∈F is denoted as zf. The set of requesting users at BSm in time slot t is defined as Umt, and the number of UEs requesting content from BSm during time slot t is defined as Nmt. Each UE is assumed to be associated with only one BS in each time slot, i.e., each UE can be served by only one BS in a given time slot. The number of requests from all users can be defined as Ru,mt=R1,mt,R2,mt,⋯,Ru,mt,⋯,RU,mt.


## 3.3. Local Popularity Model

Due to the diversity of the content preferences in different BSs, the local content popularity of all content inBSm at time slot t is defined as a content popularity vector Pmt=Pm,1t,Pm,1t,⋯,Pm,ft,⋯,Pm,Ft. In addition, considering the privacy and security of users, FL is applied to accurately predict content popularity without the UE uploading all individual user preference data to the BS.


## 3.4. Cooperative Cache Model

The content requests of mobile devices are first received by BSs. If the requested content has been cached in the local BS, it will be pushed to users immediately. A binary local content delivery variablexu,m,ft∈0,1 indicates whether the local BS provides services for the UE: xu,m,ft=1 if a response is requested and xu,m,ft=0 otherwise. If the requested content is not cached in the local BS, the BS will obtain the requested content from other BSs through the CM. Let xu,i,ft∈0,1i∈M,i≠m denote a nonlocal BS serving the UE: xu,i,ft=1 if another BS responds to the request and xu,i,ft=0 otherwise. If the CM cannot find the requested content in any BS, the local BS will obtain the requested content from the cloud server and deliver it to the UE. Let xu,c,ft∈0,1 denote whether the UE obtains the content from cloud server c in time slot t: xu,i,ft=1 if the UE obtains content f from the cloud server and xu,i,ft=0 otherwise.


## 3.5. Request Cost Model

The cost of the system is composed of two main parts: the storage energy consumption and the transmission energy consumption of the content on the MEC. If the content is cached in the BS, there will be additional storage energy consumption. Assuming that all BS servers have the same performance, the storage cost of BS caching content [31] within period T can be expressed as
(1)costca=∑f=1Fdm,f·zf·p·Twhere the caching decision dm,f∈0,1 indicates whether content f is cached in BSm and p is the energy consumption of the MEC server to cache each bit of data. The wireless transmission rate between BSm and UEu can be obtained by the Shannon formula:
(2)ϕu,m=Blog21+pu,mgu,mσ2,where B represents the bandwidth of the base station, pu,m denotes the transmission power from BSm to UEu, gu,m is the channel gain between BSm and UEu, and σ2 is the variance of additive Gaussian white noise. Then, the transmission energy consumption of UEu to download content f from its local BSm is
(3)Em,f=zfϕu,m·pe,where zf/ϕu,m denotes the time taken for file f to be transmitted from the user to the local base station, pe is the transmission power between BSs. We define the transmission rate between a BS and the CM as ϕo,m and that between the CM and the cloud server as ϕc,o. The transmission energy consumption of UE acquiring content from nonlocal BSs is expressed as:
(4)Ei,f=2zfϕc,m·pe+Em,f,where zf/ϕc,m denotes the time taken for file f to be transmitted from CM to the nonlocal base station. The energy consumption of UE downloading content f from cloud server c can be expressed as:
(5)Ec,f=zfϕc,o·pc+zfϕo,m·pe+Em,f,where pc is the transmission power of the cloud server. ϕo,m>ϕc,o can be seen from the transmission rate, so the transmission energy consumption is Ec,f>Ei,f>Em,f. Therefore, the transmission cost can be expressed as follows:
(6)costtran=∑u=1Nmt∑f=1Fxu,m,ft·Em,f+xu,i,ft·Ei,f+xu,c,ft·Ec,f.Thus, the total cost can be expressed as(7)costtotal=costca+costtran.


## 3.6. Formalization

In this paper, the provider cost-based edge cocaching approach aims to minimize the total cost by efficiently caching content on edge servers. This problem can be expressed mathematically as:(8)min∑m=1Mcosttotal,s.t.C1:dm,f∈0,1,C2:xu,m,ft,xu,i,ft,xu,c,ft∈0,1,C3:∑f=1Fdm,f·zf≤sm.where constraints C1 and C2 indicate that the cache decision and content delivery variables are binary. C3 indicates that the data in each BS should not exceed its storage capacity.


## 4. Problem Solution

In this section, we predict the user’s request behavior via the factor machine (FM) algorithm [32] to account for different users’ personality preferences in different scenarios. The FM algorithm can solve the feature combination problem under sparse data conditions. In addition, considering users’ privacy security, we accurately predict the content popularity by applying FL to the content popularity prediction algorithm, which does not require the UE to upload all individual user preference data to the BS.


### 4.1. FL-Based Content Popularity Prediction Model

#### 4.1.1. Creation of the Local Model

For each contentf, define xfn=xf1,xf2,⋯,xfd as its feature vector. yf is the category tag. If the content is requested, yf=1; otherwise, yf=0. We define pt,u,f as the probability of requesting content f for UEu in time slot t. The correspondence between the feature vector for the requested content and the category label is approximated based on the sigmoid function, and the FM model represents the user preferences. The formula is expressed as follows:
(9)p^t,u,f=11−ey∧f,(10)y^f=wu,0+∑i=1dwu,ixf,i+∑i=1d∑j=i+1dwu,ijxf,ixf,j,where d denotes the sample feature dimension and xf,i is the value of the i-th feature of content f. wu,0wu,iwu,ij are model parameters. In the case of sparse data, very few samples will satisfy the nonzero cross term. When the number of training samples is insufficient, insufficient and inaccurate training of the parameters wu,ij is likely, which affects the model’s effectiveness. Therefore, an e-dimensional auxiliary vector vi=vi,1,vi,2⋯vi,l⋯vi,e,l∈1,e is introduced for each feature xf,i. Then, the second-order parameter can be expressed as wu,ij=vi·vjT. The above equation (10) can be converted to
(11)y^f=wu,0+∑i=1dwu,ixf,i+∑i=1d∑j=i+1dvi,vjxf,ixf,j,where vi,vj=∑l=1evi,l·vj,l. The training parameters are integrated into wu=wu,0,wu,1,wu,2,⋯,wu,d,v1,1,v1,2,⋯,vd,e, and the quadratic term learning parameters are d∗e.


#### 4.1.2. Training of the Local Model

To track the changes in user preferences and protect user privacy, we design a local model training process using user request records as the input data for model training. To measure the learning performance of the model, we use the cross-entropy loss function to represent the loss ofUEu for the binary classification problem. The formula is as follows:
(12)Losswu,xf,yf=−yflogp^t,u,f−1−yflog1−p^t,u,f.When the user preference model update starts, we assume thatUEu receives Ru requests. Based on the collected samples, we iteratively learn the user preference model parameters by minimizing the logistic loss for each sample, denoted as
(13)wur+1=argminLosswu,xf,yfr=1,2,⋯,Ru,where wur+1 is defined as the model parameters learned by UEu in the k-th iteration.Due to the constructed dataset’s extensive feature dimensionality and sparseness, overfitting may occur. Follow-the-regularized-leader (FTRL) [14] is used to solve Equation (14). FTRL is an online optimization method based on the online gradient descent (OGD) method, and Equation (15) is the iterative strategy of OGD. FTRL introduces both L1 and L2 mixed regularization terms into the optimization process. The L1 regularization term increases the sparsity of the model solution, and the L2 regularization term helps to prevent the model from overfitting. The update strategy of FTRL is
(14)wur+1=wur−ηrgr,r=1,2,⋯,Ru,(15)wur+1=argming1:r·wu+12∑s=1rσswu−ws,u22+λ1wu1+12λ2wu22,r=1,2,⋯,Ru,where ηr denotes a nonincreasing learning rate. σ is a parameter related to ηr that satisfies ∑s=1rσs=1/ηr. λ1 and λ2 denote regularization parameters with positive values, and g1:r=∑s=1rgs is the sum of the gradient vectors of the first r samples. The gradient vector gr of the r-th sample is represented as follows
(16)gr=∇Losswu,xk,yk=p^u−yrwu=wu,0,p^u−yrxr,iwu=wu,i,p^u−yrxr,i∑j=1dvj,lxr,j−vi,lxr,i2wu=vi,l.By continuing the expansion of Equation (16), we can obtain:
(17)wur+1=argming1:r·wu+12∑s=1rσswuTwu−2wuTws,u+ws,uTws,u+λ1wu1+12λ2wu22=argming1:r−∑s=1rσs·wu+12∑s=1rσsws,uTws,u+12∑s=1rσs+λ2wuTwu+λ1wu1=argminzrT·wu+12∑s=1rσs+λ2wu22+λ1wu1+12∑s=1rσs·ws,u22where 1/2∑s=1rσsws,u22 is a constant that does not affect the problem. Let zr=g1:r−∑s=1rσsws,u; we can then obtain the zr iteration relationship as follows:
(18)zr=zr−1+gr+σrwr,uFor the requested content, there is a difference in the weight change rate of each feature dimension, and the gradient value of each feature dimension reflects this change rate. Therefore, different learning rates are used for different feature dimensions:(19)gr=gu,0r,gu,1r,gu,2r,⋯,gu,dr,g1,1r,g1,2r⋯gd,erT,zr=zu,0r,zu,1r,zu,2r,⋯,zu,dr,z1,1r,z1,2r⋯zd,erT,wu=wu,0,wu,1,wu,2⋯wu,d,v1,1,v1,2⋯vd,eT,wur+1=wu,0r+1,wu,1r+1,wu,2r+1⋯wu,dr+1,v1,1r+1,v1,2r+1⋯vd,er+1T.The feature dimension isN=1+d+d∗e, and the n-th feature learning rate can be denoted as ηr,n=α/β+∑s=1rgs,n2=α/β+qs,n, where α and β are tuning parameters chosen to yield good learning performance. These two parameters are FTRL optimization parameters. Equation (18) can be split into subproblems for each feature
(20)wu,nr+1=argminzr,nwu,n+λ1wu,n+12∑s=1rσs+λ2wu,n2r=1,2,⋯,Ruwhere the L1 norm is nondifferentiable at wu,n=0. Define ∂wu,n as the subgradient of the L1 norm when wu,n=wr+1,u,n. The optimal solution should satisfy that the derivative is 0, so zr,n+λ1∂wu,n+1/2∑s=1rσs+λ2wu,n=0 is obtained by derivation of the above formula. Thus, we have:
(21)wur+1=0,fzr,n<λλ1sgnzr,n−zr,nλ2+β+qs,n/α,otherwiser=1,2,⋯,Ru.


#### 4.1.3. Model Aggregation

After each UE completes the local model training process, to obtain the overall content popularity,BSm must aggregate the models trained by all UEs, using the federated average method for aggregation. The parameters of the global model are formulated as:
(22)wm=∑u=1UmtRu·wu∑u∈UmtRu.At this time, the regional popularityPmt of BSm can be obtained by Formula (10). The time complexity of the proposed prediction model algorithm is OUmtRuN, and the specific algorithm is shown in Algorithm 1.Algorithm 1: The proposed content popularity prediction algorithm based on FL.
1: Input:αu,βu,αv,βv,L1u,L2u,L1v,L2v2: Initialization:wu,zu,qu3:Foru=1,2,⋯,Umtdo4:Forr=1,2,⋯,Rudo5:  Calculategu,0r by (17)6:σu0r=1/αuqu,0r−1+gu,0r−12−qu,0r−17:zu,0r=zu,0r−1+gu,0r+σu,0r+wu,0r8:qu,0r=qu,0r−1+gu,0r29:Fori∈ddo10:   Calculategu,0r by (17)11:σuir=1/αuqu,ir−1+gu,ir−12−qu,ir−112:zu,ir=zu,ir−1+gu,ir+σu,ir+wu,ir13:qu,ir=qu,ir−1+gu,ir214:Forl∈edo15:    Calculategd,er by (17)16:σilr=1/αvqi,lr−1+gi,lr−12−qi,lr−117:zi,lr=zi,lr−1+gi,lr+σi,lr+wi,lr18:qi,lr=qi,lr−1+gi,lr219:End for20:End for21:End for22:End for23: Calculatewu by (21)24: Calculatewm by (22)25: Calculatep^t,u,f


### 4.2. Deep Reinforcement Learning-Based Content Caching Decision

After obtaining the content collection, it is necessary to determine which content must be placed in the edge server to minimize the cost to the video provider. Since edge cache environments usually have huge high-dimensional state spaces, it is difficult to manually determine all valuable features from the environment. Deep reinforcement learning can automatically obtain the optimal policy from the original high-dimensional state input to solve such problems. DQN is a general DRL framework, but it often overestimates theQ value of the possible actions in a given state. Additionally, DQN usually estimates Q values for all actions of each state, but this is not necessary for those states where actions have no effect on the environment or Q values. Therefore, we propose a content placement method based on Dueling-DDQN, which combines double DQN and Dueling DQN to effectively reduce the overestimation of DQN and accelerate the learning process. The main purpose of double DQN is to mitigate the overestimation problem. Dueling DQN decomposes the action-value function into a state-value function and a dominance function to speed up convergence without estimating the Q value of each action in each state.The method has the following three steps. (1) First, the cooperative content caching problem is formulated as a constrained Markov decision process (CMDP). (2) Second, the cache placement process is analyzed, and the reward function for the cache decision is constructed. (3) Finally, deep reinforcement learning is used to obtain the optimal content placement policy. The CM is considered a proxy in a given scenario, making caching decisions for all MEC servers. The CMDP element can be represented as a four-tuple consisting of (S, A, R, and C), whereS is the state space, A is the action space, R is the reward, and C is the constraint. The detailed definition is as follows:
(1)
State space:S denotes the set of edge cache node states, and St=S1t,S2t,⋯,SMt denotes the specific state of edge cache nodes in time slot t. The CM collects information such as content popularity vector and cache capacity of each edge base station(2)
Action space: action spaceA is defined as At=D1t,D2t,⋯,DMt, where At is the action space set of time slot t and represents the buffering decisions of DMt in time slot t. The CM will select at from the action space as the buffer decision of the BS according to the information received from each base station(3)
Reward:RSt,at represents the reward obtained by a BS for performing action at in state St. From Formula (8), we can see that the optimization goal of this paper is to minimize the cost of the video provider. Therefore, RSt,at, the profit gained from storing a single file, can be defined as:(23)RSt,at=−costf.(4)
Constraints: in making cache decisions, it is necessary to ensure that the files cached by each BS do not exceed its capacity. The capacity constraint is defined as follows:(24)∑f=1Fdm,f·zf≤sm.The Dueling-DDQN algorithm is a deep neural network algorithm used to predict the size of theQ value. The Q value can be understood as the state action value, i.e., the expected benefit of the agent acting in a certain state. The algorithm divides the entire model structure into two parts: the state value function and the advantage function. The state value function is used to estimate the value of a state, while the advantage function is used to estimate the advantage of an action taken in a state. The value function can be expressed as:
(25)QSt,at,θ=VSt,θ+ASt,at,θ.It is impossible to obtain a unique determination ofV and A based on a given value of Q. Therefore, Equation (25) is not identifiable. To solve this problem, a centralized treatment of the dominance function part Q function is given by
(26)QSt,at,θ=VSt,θ+ASt,at,θ−1A∑a′ASt,a′,θ,where A denotes the dimensionality of the vector ASt,at,θ. The weight parameters of the target Q network must be updated once per cycle during the training process. The parameters of the updated network are updated by stochastic gradient descent (SGD) to minimize the loss function.
(27)Lloss=ETargetQ−QSt,at,θ.The whole training process involves approximating theQ value to the target Q value, so the target Q value is expressed as:
(28)TargetQ=RSt,at+γQS′,argminQS′,a′,θ,θ′,where argminQS′,a′,θ represents the action corresponding to the maximum Q value in the current Q network. The selected action is then used to calculate the target Q value in the target network. The detailed process of the caching strategy based on Dueling-DDQN is shown in Algorithm 2.Algorithm 2: The edge cache algorithm based on Dueling-DDQN.
1: Input: The capacity of the experience replay pollN, train starts TR, size of minibatch K, discount factor γ, ε-greedy exploration ε, learning rate α, number of episodes E, target network update period C2: Initialization:w,w′,TargerQ3:For ep =1,2,⋯,Edo4: Input initial state spaceSt.5:Fort=1,2,⋯do6:  Choose an actionat via ε-greedy policy7:  Execute actionat and get the next status St+1 and reward RSt,at, judge whether St+1 is a terminal state.8:  Put the sampleSt,at,RSt,at,St+1 into the experience replay pool.9:Ift%TR==0then10:   Randomly sample training samples with a minibatch size K from the experience replay poolN.11:   Calculate the targetq value by formula (5)12:   Apply the SGD method to calculate equation (6) to update the weight w13:End if14:Ift%C==0then15:   Update target Q network parametersw′=w16:End if17:St=St+118:End for19:End for


## 4.1. FL-Based Content Popularity Prediction Model

### 4.1.1. Creation of the Local Model

For each contentf, define xfn=xf1,xf2,⋯,xfd as its feature vector. yf is the category tag. If the content is requested, yf=1; otherwise, yf=0. We define pt,u,f as the probability of requesting content f for UEu in time slot t. The correspondence between the feature vector for the requested content and the category label is approximated based on the sigmoid function, and the FM model represents the user preferences. The formula is expressed as follows:
(9)p^t,u,f=11−ey∧f,(10)y^f=wu,0+∑i=1dwu,ixf,i+∑i=1d∑j=i+1dwu,ijxf,ixf,j,where d denotes the sample feature dimension and xf,i is the value of the i-th feature of content f. wu,0wu,iwu,ij are model parameters. In the case of sparse data, very few samples will satisfy the nonzero cross term. When the number of training samples is insufficient, insufficient and inaccurate training of the parameters wu,ij is likely, which affects the model’s effectiveness. Therefore, an e-dimensional auxiliary vector vi=vi,1,vi,2⋯vi,l⋯vi,e,l∈1,e is introduced for each feature xf,i. Then, the second-order parameter can be expressed as wu,ij=vi·vjT. The above equation (10) can be converted to
(11)y^f=wu,0+∑i=1dwu,ixf,i+∑i=1d∑j=i+1dvi,vjxf,ixf,j,where vi,vj=∑l=1evi,l·vj,l. The training parameters are integrated into wu=wu,0,wu,1,wu,2,⋯,wu,d,v1,1,v1,2,⋯,vd,e, and the quadratic term learning parameters are d∗e.


### 4.1.2. Training of the Local Model

To track the changes in user preferences and protect user privacy, we design a local model training process using user request records as the input data for model training. To measure the learning performance of the model, we use the cross-entropy loss function to represent the loss ofUEu for the binary classification problem. The formula is as follows:
(12)Losswu,xf,yf=−yflogp^t,u,f−1−yflog1−p^t,u,f.When the user preference model update starts, we assume thatUEu receives Ru requests. Based on the collected samples, we iteratively learn the user preference model parameters by minimizing the logistic loss for each sample, denoted as
(13)wur+1=argminLosswu,xf,yfr=1,2,⋯,Ru,where wur+1 is defined as the model parameters learned by UEu in the k-th iteration.Due to the constructed dataset’s extensive feature dimensionality and sparseness, overfitting may occur. Follow-the-regularized-leader (FTRL) [14] is used to solve Equation (14). FTRL is an online optimization method based on the online gradient descent (OGD) method, and Equation (15) is the iterative strategy of OGD. FTRL introduces both L1 and L2 mixed regularization terms into the optimization process. The L1 regularization term increases the sparsity of the model solution, and the L2 regularization term helps to prevent the model from overfitting. The update strategy of FTRL is
(14)wur+1=wur−ηrgr,r=1,2,⋯,Ru,(15)wur+1=argming1:r·wu+12∑s=1rσswu−ws,u22+λ1wu1+12λ2wu22,r=1,2,⋯,Ru,where ηr denotes a nonincreasing learning rate. σ is a parameter related to ηr that satisfies ∑s=1rσs=1/ηr. λ1 and λ2 denote regularization parameters with positive values, and g1:r=∑s=1rgs is the sum of the gradient vectors of the first r samples. The gradient vector gr of the r-th sample is represented as follows
(16)gr=∇Losswu,xk,yk=p^u−yrwu=wu,0,p^u−yrxr,iwu=wu,i,p^u−yrxr,i∑j=1dvj,lxr,j−vi,lxr,i2wu=vi,l.By continuing the expansion of Equation (16), we can obtain:
(17)wur+1=argming1:r·wu+12∑s=1rσswuTwu−2wuTws,u+ws,uTws,u+λ1wu1+12λ2wu22=argming1:r−∑s=1rσs·wu+12∑s=1rσsws,uTws,u+12∑s=1rσs+λ2wuTwu+λ1wu1=argminzrT·wu+12∑s=1rσs+λ2wu22+λ1wu1+12∑s=1rσs·ws,u22where 1/2∑s=1rσsws,u22 is a constant that does not affect the problem. Let zr=g1:r−∑s=1rσsws,u; we can then obtain the zr iteration relationship as follows:
(18)zr=zr−1+gr+σrwr,uFor the requested content, there is a difference in the weight change rate of each feature dimension, and the gradient value of each feature dimension reflects this change rate. Therefore, different learning rates are used for different feature dimensions:(19)gr=gu,0r,gu,1r,gu,2r,⋯,gu,dr,g1,1r,g1,2r⋯gd,erT,zr=zu,0r,zu,1r,zu,2r,⋯,zu,dr,z1,1r,z1,2r⋯zd,erT,wu=wu,0,wu,1,wu,2⋯wu,d,v1,1,v1,2⋯vd,eT,wur+1=wu,0r+1,wu,1r+1,wu,2r+1⋯wu,dr+1,v1,1r+1,v1,2r+1⋯vd,er+1T.The feature dimension isN=1+d+d∗e, and the n-th feature learning rate can be denoted as ηr,n=α/β+∑s=1rgs,n2=α/β+qs,n, where α and β are tuning parameters chosen to yield good learning performance. These two parameters are FTRL optimization parameters. Equation (18) can be split into subproblems for each feature
(20)wu,nr+1=argminzr,nwu,n+λ1wu,n+12∑s=1rσs+λ2wu,n2r=1,2,⋯,Ruwhere the L1 norm is nondifferentiable at wu,n=0. Define ∂wu,n as the subgradient of the L1 norm when wu,n=wr+1,u,n. The optimal solution should satisfy that the derivative is 0, so zr,n+λ1∂wu,n+1/2∑s=1rσs+λ2wu,n=0 is obtained by derivation of the above formula. Thus, we have:
(21)wur+1=0,fzr,n<λλ1sgnzr,n−zr,nλ2+β+qs,n/α,otherwiser=1,2,⋯,Ru.


### 4.1.3. Model Aggregation

After each UE completes the local model training process, to obtain the overall content popularity,BSm must aggregate the models trained by all UEs, using the federated average method for aggregation. The parameters of the global model are formulated as:
(22)wm=∑u=1UmtRu·wu∑u∈UmtRu.At this time, the regional popularityPmt of BSm can be obtained by Formula (10). The time complexity of the proposed prediction model algorithm is OUmtRuN, and the specific algorithm is shown in Algorithm 1.Algorithm 1: The proposed content popularity prediction algorithm based on FL.
1: Input:αu,βu,αv,βv,L1u,L2u,L1v,L2v2: Initialization:wu,zu,qu3:Foru=1,2,⋯,Umtdo4:Forr=1,2,⋯,Rudo5:  Calculategu,0r by (17)6:σu0r=1/αuqu,0r−1+gu,0r−12−qu,0r−17:zu,0r=zu,0r−1+gu,0r+σu,0r+wu,0r8:qu,0r=qu,0r−1+gu,0r29:Fori∈ddo10:   Calculategu,0r by (17)11:σuir=1/αuqu,ir−1+gu,ir−12−qu,ir−112:zu,ir=zu,ir−1+gu,ir+σu,ir+wu,ir13:qu,ir=qu,ir−1+gu,ir214:Forl∈edo15:    Calculategd,er by (17)16:σilr=1/αvqi,lr−1+gi,lr−12−qi,lr−117:zi,lr=zi,lr−1+gi,lr+σi,lr+wi,lr18:qi,lr=qi,lr−1+gi,lr219:End for20:End for21:End for22:End for23: Calculatewu by (21)24: Calculatewm by (22)25: Calculatep^t,u,f


## 4.1.1. Creation of the Local Model

For each contentf, define xfn=xf1,xf2,⋯,xfd as its feature vector. yf is the category tag. If the content is requested, yf=1; otherwise, yf=0. We define pt,u,f as the probability of requesting content f for UEu in time slot t. The correspondence between the feature vector for the requested content and the category label is approximated based on the sigmoid function, and the FM model represents the user preferences. The formula is expressed as follows:
(9)p^t,u,f=11−ey∧f,(10)y^f=wu,0+∑i=1dwu,ixf,i+∑i=1d∑j=i+1dwu,ijxf,ixf,j,where d denotes the sample feature dimension and xf,i is the value of the i-th feature of content f. wu,0wu,iwu,ij are model parameters. In the case of sparse data, very few samples will satisfy the nonzero cross term. When the number of training samples is insufficient, insufficient and inaccurate training of the parameters wu,ij is likely, which affects the model’s effectiveness. Therefore, an e-dimensional auxiliary vector vi=vi,1,vi,2⋯vi,l⋯vi,e,l∈1,e is introduced for each feature xf,i. Then, the second-order parameter can be expressed as wu,ij=vi·vjT. The above equation (10) can be converted to
(11)y^f=wu,0+∑i=1dwu,ixf,i+∑i=1d∑j=i+1dvi,vjxf,ixf,j,where vi,vj=∑l=1evi,l·vj,l. The training parameters are integrated into wu=wu,0,wu,1,wu,2,⋯,wu,d,v1,1,v1,2,⋯,vd,e, and the quadratic term learning parameters are d∗e.


## 4.1.2. Training of the Local Model

To track the changes in user preferences and protect user privacy, we design a local model training process using user request records as the input data for model training. To measure the learning performance of the model, we use the cross-entropy loss function to represent the loss ofUEu for the binary classification problem. The formula is as follows:
(12)Losswu,xf,yf=−yflogp^t,u,f−1−yflog1−p^t,u,f.When the user preference model update starts, we assume thatUEu receives Ru requests. Based on the collected samples, we iteratively learn the user preference model parameters by minimizing the logistic loss for each sample, denoted as
(13)wur+1=argminLosswu,xf,yfr=1,2,⋯,Ru,where wur+1 is defined as the model parameters learned by UEu in the k-th iteration.Due to the constructed dataset’s extensive feature dimensionality and sparseness, overfitting may occur. Follow-the-regularized-leader (FTRL) [14] is used to solve Equation (14). FTRL is an online optimization method based on the online gradient descent (OGD) method, and Equation (15) is the iterative strategy of OGD. FTRL introduces both L1 and L2 mixed regularization terms into the optimization process. The L1 regularization term increases the sparsity of the model solution, and the L2 regularization term helps to prevent the model from overfitting. The update strategy of FTRL is
(14)wur+1=wur−ηrgr,r=1,2,⋯,Ru,(15)wur+1=argming1:r·wu+12∑s=1rσswu−ws,u22+λ1wu1+12λ2wu22,r=1,2,⋯,Ru,where ηr denotes a nonincreasing learning rate. σ is a parameter related to ηr that satisfies ∑s=1rσs=1/ηr. λ1 and λ2 denote regularization parameters with positive values, and g1:r=∑s=1rgs is the sum of the gradient vectors of the first r samples. The gradient vector gr of the r-th sample is represented as follows
(16)gr=∇Losswu,xk,yk=p^u−yrwu=wu,0,p^u−yrxr,iwu=wu,i,p^u−yrxr,i∑j=1dvj,lxr,j−vi,lxr,i2wu=vi,l.By continuing the expansion of Equation (16), we can obtain:
(17)wur+1=argming1:r·wu+12∑s=1rσswuTwu−2wuTws,u+ws,uTws,u+λ1wu1+12λ2wu22=argming1:r−∑s=1rσs·wu+12∑s=1rσsws,uTws,u+12∑s=1rσs+λ2wuTwu+λ1wu1=argminzrT·wu+12∑s=1rσs+λ2wu22+λ1wu1+12∑s=1rσs·ws,u22where 1/2∑s=1rσsws,u22 is a constant that does not affect the problem. Let zr=g1:r−∑s=1rσsws,u; we can then obtain the zr iteration relationship as follows:
(18)zr=zr−1+gr+σrwr,uFor the requested content, there is a difference in the weight change rate of each feature dimension, and the gradient value of each feature dimension reflects this change rate. Therefore, different learning rates are used for different feature dimensions:(19)gr=gu,0r,gu,1r,gu,2r,⋯,gu,dr,g1,1r,g1,2r⋯gd,erT,zr=zu,0r,zu,1r,zu,2r,⋯,zu,dr,z1,1r,z1,2r⋯zd,erT,wu=wu,0,wu,1,wu,2⋯wu,d,v1,1,v1,2⋯vd,eT,wur+1=wu,0r+1,wu,1r+1,wu,2r+1⋯wu,dr+1,v1,1r+1,v1,2r+1⋯vd,er+1T.The feature dimension isN=1+d+d∗e, and the n-th feature learning rate can be denoted as ηr,n=α/β+∑s=1rgs,n2=α/β+qs,n, where α and β are tuning parameters chosen to yield good learning performance. These two parameters are FTRL optimization parameters. Equation (18) can be split into subproblems for each feature
(20)wu,nr+1=argminzr,nwu,n+λ1wu,n+12∑s=1rσs+λ2wu,n2r=1,2,⋯,Ruwhere the L1 norm is nondifferentiable at wu,n=0. Define ∂wu,n as the subgradient of the L1 norm when wu,n=wr+1,u,n. The optimal solution should satisfy that the derivative is 0, so zr,n+λ1∂wu,n+1/2∑s=1rσs+λ2wu,n=0 is obtained by derivation of the above formula. Thus, we have:
(21)wur+1=0,fzr,n<λλ1sgnzr,n−zr,nλ2+β+qs,n/α,otherwiser=1,2,⋯,Ru.


## 4.1.3. Model Aggregation

After each UE completes the local model training process, to obtain the overall content popularity,BSm must aggregate the models trained by all UEs, using the federated average method for aggregation. The parameters of the global model are formulated as:
(22)wm=∑u=1UmtRu·wu∑u∈UmtRu.At this time, the regional popularityPmt of BSm can be obtained by Formula (10). The time complexity of the proposed prediction model algorithm is OUmtRuN, and the specific algorithm is shown in Algorithm 1.Algorithm 1: The proposed content popularity prediction algorithm based on FL.
1: Input:αu,βu,αv,βv,L1u,L2u,L1v,L2v2: Initialization:wu,zu,qu3:Foru=1,2,⋯,Umtdo4:Forr=1,2,⋯,Rudo5:  Calculategu,0r by (17)6:σu0r=1/αuqu,0r−1+gu,0r−12−qu,0r−17:zu,0r=zu,0r−1+gu,0r+σu,0r+wu,0r8:qu,0r=qu,0r−1+gu,0r29:Fori∈ddo10:   Calculategu,0r by (17)11:σuir=1/αuqu,ir−1+gu,ir−12−qu,ir−112:zu,ir=zu,ir−1+gu,ir+σu,ir+wu,ir13:qu,ir=qu,ir−1+gu,ir214:Forl∈edo15:    Calculategd,er by (17)16:σilr=1/αvqi,lr−1+gi,lr−12−qi,lr−117:zi,lr=zi,lr−1+gi,lr+σi,lr+wi,lr18:qi,lr=qi,lr−1+gi,lr219:End for20:End for21:End for22:End for23: Calculatewu by (21)24: Calculatewm by (22)25: Calculatep^t,u,f


## 4.2. Deep Reinforcement Learning-Based Content Caching Decision

After obtaining the content collection, it is necessary to determine which content must be placed in the edge server to minimize the cost to the video provider. Since edge cache environments usually have huge high-dimensional state spaces, it is difficult to manually determine all valuable features from the environment. Deep reinforcement learning can automatically obtain the optimal policy from the original high-dimensional state input to solve such problems. DQN is a general DRL framework, but it often overestimates theQ value of the possible actions in a given state. Additionally, DQN usually estimates Q values for all actions of each state, but this is not necessary for those states where actions have no effect on the environment or Q values. Therefore, we propose a content placement method based on Dueling-DDQN, which combines double DQN and Dueling DQN to effectively reduce the overestimation of DQN and accelerate the learning process. The main purpose of double DQN is to mitigate the overestimation problem. Dueling DQN decomposes the action-value function into a state-value function and a dominance function to speed up convergence without estimating the Q value of each action in each state.The method has the following three steps. (1) First, the cooperative content caching problem is formulated as a constrained Markov decision process (CMDP). (2) Second, the cache placement process is analyzed, and the reward function for the cache decision is constructed. (3) Finally, deep reinforcement learning is used to obtain the optimal content placement policy. The CM is considered a proxy in a given scenario, making caching decisions for all MEC servers. The CMDP element can be represented as a four-tuple consisting of (S, A, R, and C), whereS is the state space, A is the action space, R is the reward, and C is the constraint. The detailed definition is as follows:
(1)
State space:S denotes the set of edge cache node states, and St=S1t,S2t,⋯,SMt denotes the specific state of edge cache nodes in time slot t. The CM collects information such as content popularity vector and cache capacity of each edge base station(2)
Action space: action spaceA is defined as At=D1t,D2t,⋯,DMt, where At is the action space set of time slot t and represents the buffering decisions of DMt in time slot t. The CM will select at from the action space as the buffer decision of the BS according to the information received from each base station(3)
Reward:RSt,at represents the reward obtained by a BS for performing action at in state St. From Formula (8), we can see that the optimization goal of this paper is to minimize the cost of the video provider. Therefore, RSt,at, the profit gained from storing a single file, can be defined as:(23)RSt,at=−costf.(4)
Constraints: in making cache decisions, it is necessary to ensure that the files cached by each BS do not exceed its capacity. The capacity constraint is defined as follows:(24)∑f=1Fdm,f·zf≤sm.The Dueling-DDQN algorithm is a deep neural network algorithm used to predict the size of theQ value. The Q value can be understood as the state action value, i.e., the expected benefit of the agent acting in a certain state. The algorithm divides the entire model structure into two parts: the state value function and the advantage function. The state value function is used to estimate the value of a state, while the advantage function is used to estimate the advantage of an action taken in a state. The value function can be expressed as:
(25)QSt,at,θ=VSt,θ+ASt,at,θ.It is impossible to obtain a unique determination ofV and A based on a given value of Q. Therefore, Equation (25) is not identifiable. To solve this problem, a centralized treatment of the dominance function part Q function is given by
(26)QSt,at,θ=VSt,θ+ASt,at,θ−1A∑a′ASt,a′,θ,where A denotes the dimensionality of the vector ASt,at,θ. The weight parameters of the target Q network must be updated once per cycle during the training process. The parameters of the updated network are updated by stochastic gradient descent (SGD) to minimize the loss function.
(27)Lloss=ETargetQ−QSt,at,θ.The whole training process involves approximating theQ value to the target Q value, so the target Q value is expressed as:
(28)TargetQ=RSt,at+γQS′,argminQS′,a′,θ,θ′,where argminQS′,a′,θ represents the action corresponding to the maximum Q value in the current Q network. The selected action is then used to calculate the target Q value in the target network. The detailed process of the caching strategy based on Dueling-DDQN is shown in Algorithm 2.Algorithm 2: The edge cache algorithm based on Dueling-DDQN.
1: Input: The capacity of the experience replay pollN, train starts TR, size of minibatch K, discount factor γ, ε-greedy exploration ε, learning rate α, number of episodes E, target network update period C2: Initialization:w,w′,TargerQ3:For ep =1,2,⋯,Edo4: Input initial state spaceSt.5:Fort=1,2,⋯do6:  Choose an actionat via ε-greedy policy7:  Execute actionat and get the next status St+1 and reward RSt,at, judge whether St+1 is a terminal state.8:  Put the sampleSt,at,RSt,at,St+1 into the experience replay pool.9:Ift%TR==0then10:   Randomly sample training samples with a minibatch size K from the experience replay poolN.11:   Calculate the targetq value by formula (5)12:   Apply the SGD method to calculate equation (6) to update the weight w13:End if14:Ift%C==0then15:   Update target Q network parametersw′=w16:End if17:St=St+118:End for19:End for


## 5. Simulation Results

In this section, the experimental results of the proposed algorithms are investigated, and the performance of four other algorithms, namely, content caching algorithm based on marginal gain, FL-based caching strategy, popularity-based caching algorithm, and noncooperative caching strategy, is taken as a reference.


### 5.1. Simulation Parameters

In the experiment, it is assumed that the number of base stationsM is [3,6], and the capacity of base stations sm is set to [100,300] MB identically. The number of users is 30, the users are randomly distributed under different BSs, and the content size zf is set to [5,10] MB. The data rate between each BS m and the CM is set to 128 MB/s, and the data rate between the CM and cloud server c is set to 32 MB/s. The parameters used in the simulation experiment are shown in Table 3.Table 3
Key notations and values.
NotationDescriptionValueMNumber of BSs[3,6]FNumber of contents1682UNumber of users[30,60]smCapacity of BSs[100,300]MBzfContent size[8,16]MBϕo,mData rate between each BSm and CM128 MB/sϕc,oData rate between CM and cloud serverc32 MB/s


### 5.2. Datasets

To evaluate the performance of the proposed edge caching strategy, we use a real-world dataset-MovieLens [33]. The MovieLens ([https://grouplens.org/datasets/movielens/]) dataset contains rating data for multiple movies by multiple users, movie metadata information, and user attribute information. The MovieLens 100 K dataset contains 100,000 ratings for 1682 movies by 943 users. Each user has reviewed at least 20 movies with ratings on a 5-star scale, from 0 to 5. This paper simulates the process of users requesting content. We assume that the movie participation score is the content requested by the user, and each movie score corresponds to a content download. Literature [14, 28] take a similar approach to simulate the process of a user requesting content.


### 5.3. Performance Metrics

This paper considers three performance metrics: cache hit ratio (hit), average transmission delay (time), and cost. The cache hit ratio represents the ratio of satisfied requests to the total number of requests at the edge node. It is defined as:(29)hit=Rrequest−RmissRrequest,where Rrequest is the total number of requests received by the edge server in each time slot, and Rmiss is the number of missed requests. The average transmission delay represents the average delay to transmit the content from the edge server or the central server to the user. It is expressed as:
(30)time=∑u=1Nmt∑f=1Fxu,m,ft·zfϕu,m+xu,i,ft·2zfϕc,m+zfϕu,m+xu,c,ft·zf/ϕc,o+zf/ϕc,m+zf/ϕu,mRrequest,where the above equation is the ratio of the transmission delay of all requests and the number of all requests between time slot t. The cost is the optimization objective function in this paper.


### 5.4. Results

To evaluate the performance, the proposed algorithm is compared with the following four algorithms:(1)
Content caching strategy based on marginal gain [30](CCBG): this strategy analyzes the marginal gain in latency and cache cost to cache the content in the edge server(2)
FL-based caching policy [28](FLBC): this algorithm applies FL to user demand prediction and formulates the caching problem as an integer linear programming (ILP) problem(3)
Popularity-based caching algorithm [34](PBC): this is a caching update scheme based on content popularity, replacing low-popularity content with high-popularity content(4)
Noncooperative caching strategy (no collaboration): this algorithm does not consider cooperation between base stations for content storageIn Figure2, the predicted popularity is shown using the proposed request prediction strategy. The predicted popularity is very close to the actual popularity, which indicates that our proposed popularity prediction strategy can effectively predict user requests.Figure 2
Comparison of the predicted popularity and the real popularity in the finite time horizonT.Figure3 shows the impact of the number of BSs on the cache hit rate, average transmission delay, and cost. The capacity of BSs is set to 100 MB, and the number of BSs is increased from 3 to 6. From Figures 3(a) and 3(b), it can be seen that the cache hit ratio increases and the average transfer latency decreases for the four algorithms as the number of BSs increases. In addition, the proposed algorithm consistently outperforms the four baseline algorithms. The reason is that as the number of BSs increases, the amount of content that the collaborative baseline can cache increases; however, the number of users in the collaborative range also increases, and the types of content requests from users become more diverse. This scenario leads to a gradual slowing of the hit rate variation and a gradual decrease in the delay variation. The increase in the number of users leads to an increase in the number of requests, so the hit rate and latency still show an improving trend. From Figure 3(c), the comparison shows that the proposed service placement method outperforms the other cache placement methods. The overall cost of the comparison algorithm is increasing, and the total cost of the proposed algorithm is decreasing. There is a downward trend in transport costs as most requests can be responded to locally. However, as the number of BSs increases, storage costs also increase. The proposed algorithm is more optimized for the transfer cost.Figure 3
Effect of the number of BSs on cache performance.
(a)
Effect of the number of BSs on hit rate(b)
Effect of the number of BSs on average latency(c)
Effect of the number of BSs on energy costFigure4 shows the impact of the BS capacity on the cache hit ratio, average transmission delay, and cost. The number of BSs is set to 3, and the capacity of the BSs is set to 100 MB-300 MB. As shown in Figure 4(a), as the storage capacity of the BSs increases, the cache hit rate gradually increases. That is because as the cache capacity of the BSs increases, the BS can cache more files, so the cache hit rate increases. As shown in Figure 4(b), as the storage capacity of the BSs increases, the delay gradually decreases the number of times files are obtained from the cloud server decreases, so the delay decreases. As shown in Figure 4(c), as the BS storage capacity increases, the video provider’s cost decreases. Because more content can be requested locally without going through the cloud center, the content with the most significant profit gain will be cached first as the cache capacity increases. Hence, the hit rate, latency, and profit curve tend to change faster initially and then more slowly.Figure 4
Effect of the caching capacity for each BS on cache performance.
(a)
Effect of the caching capacity for each BS on hit rate(b)
Effect of the caching capacity for each BS on average latency(c)
Effect of the caching capacity for each BS on energy costFigure5 shows the impact of the UE number on the cache hit rate, average transmission delay, and cost. The number of BSs is set to 3, the capacity of the BSs is set to 200 MB, and the number of UEs served is increased from 30 to 60. As shown in Figure 5(a), as the number of UEs increases, the cache hit rate shows a downward trend. This is because as the number of UEs increases, the content requested by users becomes diverse, and content requests become scattered, so the hit rate decreases accordingly. As shown in Figure 5(b), the delay increases gradually as the number of UEs increases. It is because as the cache hit rate of the BSs decreases, the number of times files are obtained from the cloud server increases, so the delay also decreases. As shown in Figure 5(c), the video provider cost increases as the number of UEs increases. Because the number of requests passing through the cloud server increases, the overall cost of video providers increases.Figure 5
Effect of the number of UEs on cache performance.
(a)
Effect of the number of UEs on hit rate(b)
Effect of the number of UEs on average latency(c)
Effect of the number of UEs on energy cost


## 5.1. Simulation Parameters

In the experiment, it is assumed that the number of base stationsM is [3,6], and the capacity of base stations sm is set to [100,300] MB identically. The number of users is 30, the users are randomly distributed under different BSs, and the content size zf is set to [5,10] MB. The data rate between each BS m and the CM is set to 128 MB/s, and the data rate between the CM and cloud server c is set to 32 MB/s. The parameters used in the simulation experiment are shown in Table 3.Table 3
Key notations and values.
NotationDescriptionValueMNumber of BSs[3,6]FNumber of contents1682UNumber of users[30,60]smCapacity of BSs[100,300]MBzfContent size[8,16]MBϕo,mData rate between each BSm and CM128 MB/sϕc,oData rate between CM and cloud serverc32 MB/s


## 5.2. Datasets

To evaluate the performance of the proposed edge caching strategy, we use a real-world dataset-MovieLens [33]. The MovieLens ([https://grouplens.org/datasets/movielens/]) dataset contains rating data for multiple movies by multiple users, movie metadata information, and user attribute information. The MovieLens 100 K dataset contains 100,000 ratings for 1682 movies by 943 users. Each user has reviewed at least 20 movies with ratings on a 5-star scale, from 0 to 5. This paper simulates the process of users requesting content. We assume that the movie participation score is the content requested by the user, and each movie score corresponds to a content download. Literature [14, 28] take a similar approach to simulate the process of a user requesting content.


## 5.3. Performance Metrics

This paper considers three performance metrics: cache hit ratio (hit), average transmission delay (time), and cost. The cache hit ratio represents the ratio of satisfied requests to the total number of requests at the edge node. It is defined as:(29)hit=Rrequest−RmissRrequest,where Rrequest is the total number of requests received by the edge server in each time slot, and Rmiss is the number of missed requests. The average transmission delay represents the average delay to transmit the content from the edge server or the central server to the user. It is expressed as:
(30)time=∑u=1Nmt∑f=1Fxu,m,ft·zfϕu,m+xu,i,ft·2zfϕc,m+zfϕu,m+xu,c,ft·zf/ϕc,o+zf/ϕc,m+zf/ϕu,mRrequest,where the above equation is the ratio of the transmission delay of all requests and the number of all requests between time slot t. The cost is the optimization objective function in this paper.


## 5.4. Results

To evaluate the performance, the proposed algorithm is compared with the following four algorithms:(1)
Content caching strategy based on marginal gain [30](CCBG): this strategy analyzes the marginal gain in latency and cache cost to cache the content in the edge server(2)
FL-based caching policy [28](FLBC): this algorithm applies FL to user demand prediction and formulates the caching problem as an integer linear programming (ILP) problem(3)
Popularity-based caching algorithm [34](PBC): this is a caching update scheme based on content popularity, replacing low-popularity content with high-popularity content(4)
Noncooperative caching strategy (no collaboration): this algorithm does not consider cooperation between base stations for content storageIn Figure2, the predicted popularity is shown using the proposed request prediction strategy. The predicted popularity is very close to the actual popularity, which indicates that our proposed popularity prediction strategy can effectively predict user requests.Figure 2
Comparison of the predicted popularity and the real popularity in the finite time horizonT.Figure3 shows the impact of the number of BSs on the cache hit rate, average transmission delay, and cost. The capacity of BSs is set to 100 MB, and the number of BSs is increased from 3 to 6. From Figures 3(a) and 3(b), it can be seen that the cache hit ratio increases and the average transfer latency decreases for the four algorithms as the number of BSs increases. In addition, the proposed algorithm consistently outperforms the four baseline algorithms. The reason is that as the number of BSs increases, the amount of content that the collaborative baseline can cache increases; however, the number of users in the collaborative range also increases, and the types of content requests from users become more diverse. This scenario leads to a gradual slowing of the hit rate variation and a gradual decrease in the delay variation. The increase in the number of users leads to an increase in the number of requests, so the hit rate and latency still show an improving trend. From Figure 3(c), the comparison shows that the proposed service placement method outperforms the other cache placement methods. The overall cost of the comparison algorithm is increasing, and the total cost of the proposed algorithm is decreasing. There is a downward trend in transport costs as most requests can be responded to locally. However, as the number of BSs increases, storage costs also increase. The proposed algorithm is more optimized for the transfer cost.Figure 3
Effect of the number of BSs on cache performance.
(a)
Effect of the number of BSs on hit rate(b)
Effect of the number of BSs on average latency(c)
Effect of the number of BSs on energy costFigure4 shows the impact of the BS capacity on the cache hit ratio, average transmission delay, and cost. The number of BSs is set to 3, and the capacity of the BSs is set to 100 MB-300 MB. As shown in Figure 4(a), as the storage capacity of the BSs increases, the cache hit rate gradually increases. That is because as the cache capacity of the BSs increases, the BS can cache more files, so the cache hit rate increases. As shown in Figure 4(b), as the storage capacity of the BSs increases, the delay gradually decreases the number of times files are obtained from the cloud server decreases, so the delay decreases. As shown in Figure 4(c), as the BS storage capacity increases, the video provider’s cost decreases. Because more content can be requested locally without going through the cloud center, the content with the most significant profit gain will be cached first as the cache capacity increases. Hence, the hit rate, latency, and profit curve tend to change faster initially and then more slowly.Figure 4
Effect of the caching capacity for each BS on cache performance.
(a)
Effect of the caching capacity for each BS on hit rate(b)
Effect of the caching capacity for each BS on average latency(c)
Effect of the caching capacity for each BS on energy costFigure5 shows the impact of the UE number on the cache hit rate, average transmission delay, and cost. The number of BSs is set to 3, the capacity of the BSs is set to 200 MB, and the number of UEs served is increased from 30 to 60. As shown in Figure 5(a), as the number of UEs increases, the cache hit rate shows a downward trend. This is because as the number of UEs increases, the content requested by users becomes diverse, and content requests become scattered, so the hit rate decreases accordingly. As shown in Figure 5(b), the delay increases gradually as the number of UEs increases. It is because as the cache hit rate of the BSs decreases, the number of times files are obtained from the cloud server increases, so the delay also decreases. As shown in Figure 5(c), the video provider cost increases as the number of UEs increases. Because the number of requests passing through the cloud server increases, the overall cost of video providers increases.Figure 5
Effect of the number of UEs on cache performance.
(a)
Effect of the number of UEs on hit rate(b)
Effect of the number of UEs on average latency(c)
Effect of the number of UEs on energy cost


## 6. Conclusion

In this paper, we propose a deep reinforcement learning-based approach to collaborative content caching to optimize video providers’ costs. First, the content caching problem is represented as a CMDP. Then, the content caching process is analyzed to construct a caching reward function. Finally, deep Q-learning is used to obtain the optimal content caching strategy. In addition, considering the user’s content request and privacy security, federated learning is used to design the caching strategy to make distributed predictions for the users in the nodes. Simulation results based on real datasets show that the proposed algorithm optimizes the cost of the video provider while achieving a high cache hit rate.Although the cooperative caching strategy proposed in this paper has achieved good results, there are still some shortcomings. The proposed algorithm is a centralized algorithm and may not be suitable for network scenarios with a large number of base stations. Future work will aim at designing a scheme for multiagent proxy DRL that can learn the optimal caching policy where each BS acts as an agent and makes its own caching decisions.



---

*Source: 7212984-2022-12-22.xml*