# Social-Aware Caching Strategy Based on Joint Action Deep Reinforcement Learning

Authors: Jing Yang; Zhuowei Song; Peng He; Yaping Cui; Dapeng Wu; Ruyan Wang
Journal: Wireless Communications and Mobile Computing
 (2021)

---

## Abstract

Caching in device-to-device (D2D) networks is emerging a promising trend, which enables to reduce backhaul traffic. Moreover, social interaction among users influences the performance of overall system network. Therefore, it is crucial to consider social attributes in the D2D networks to develop a caching strategy to resolve the problem of unbalanced content distributed. In this paper, we consider two types of users according to their activeness, i.e., active users and inactive users. Inactive users assist active users cache contents during off-peak periods and provide the contents to the active users during peak periods to relieve the pressure of base station (BS). In addition, caching system model is divided into physical domain model and social domain model. In physical domain, the quality of communication links is judged by the delay between D2D users. In social domain, based on a real-world dataset, CiaoDVD, we calculate user similarity in three dimensions and obtain user trust by a trust topology to measure user relationships. Finally, in order to maximize the cache hit ratio, a joint action deepQ-networks (JADQN) framework is proposed to pair the active users with inactive users and distribute the contents to inactive users. Simulation results indicate that the proposed strategy improves the cache hit ratio by 42.9% and reduces the download delay by 48.8% compared with least frequency used (LFU) algorithm, which validates the effectiveness of our method.


---

## Body

## 1. Introduction

Recently, the data traffic is exploding with plenty of mobile devices connecting in the wireless network, which brings a heavy burden to the cellular infrastructures. In addition, due to the ever-increasing video applications, the multimedia video services play an important role in our lives and become the majority of Internet traffic [1]. Besides, the unprecedented amount of demand imposes stringent requirements in terms of bandwidth, transmission delay, and communication quality [2]. Caching utilizes the storage capacity of wireless device-assisted traffic offloading [3], which is regard as one of the most promising methods to cope with these problems.Device-to-device (D2D) communication is recognized as a key technology which can establish communication links without passing through cellular infrastructure [4]. By this way, there are plenty of advantages including high data rates, short delays, and low power consumption [5]. Moreover, D2D communications could extend network capacity, which can increase the cache space of the system. Thus, D2D-assisted caching, which is combined with the advantages of caching and D2D communications, can exploit the cache capacity of user devices during the off-peak periods. In detail, D2D-assisted caching can prefetch contents in the local cache from the adjacent devices during off-peak hours when the network has abundant resources. In this way, the spectrum resources can be efficiently utilized, and the throughput of network can be remarkably improved. In this case, the pressure of base station (BS) can be relieved during the peak periods. However, due to the huge number of contents, the cache resources and the communication resources of users are limited. Moreover, different requests of users enable the unbalanced content distribution and different cache capacity of users. Thus, it is particularly significant to design a caching allocation policy to enhance the cache resources utilization. Some approaches have considered the channel state of the wireless links and the distance of nodes to improve caching performance like the average download delay, throughput, and cache hit ratio. Nevertheless, user relationships influence the quality of communication because the communication link may be broken if the users are not willing to establish the communication links to share their resources.With the development and popularity of social applications such as Wechat, YouTube, and Facebook, the social relationships among people are extensively broadened and obviously enhanced [6]. In addition, the social interactions among users and mobile devices have a significant impact on wireless network because the devices are carried by users. Thus, the social relationship is regarded as a perspective tool to address the problem in caching, and some efforts on the user social attributes are researched and utilized to resolve the problems on caching. [7, 8]. The social characteristics consist of ties, community, centrality, and bridge. By using the close social ties, the users could easily find the people who have the same interests with them in the MSN. In addition, the users prefer to share contents with their familiar or some people who have common interests in the real world. However, how to match the users and allocate the contents to the users remained challenges. To solve the problem, users’ social attributes can be exploited for developing the caching strategy, which enables the users to provide assistance on caching and enhance the cache hit ratio of the system.In this paper, we divide the users into active users and inactive users depending on their activeness. We establish two layers of networks, which consists of physical layer and social layer, to design the caching strategy. In physical domain, we consider the quality of communication links. Meanwhile, in social domain, we analyze the users social relationships which include users similarity and users’ trust. In this case, we present a caching strategy based on a joint action deepQ-networks (JADQN) framework to choose cache nodes and cache contents. The proposed strategy is aiming to maximize the cache hit ratio and reduce the average download delay. We summarize the main contributions of this paper as follows.We consider the attributes of physical domain and social domain. In the social domain, the user similarity is calculated by three dimensions (i.e., user content rating, user preference for the genre of content, and the number of common friends between users). The trust topology consists of users’ direct trust and indirect trust which can be obtained by users’ interaction and trust transitivity.We propose a JADQN framework to match the users and allocate the contents based on users’ preference in inactive users remain space. Based on CiaoDVD, the proposed strategy performs a higher cache hit ratio and lower average download delay compared with the classical algorithm.The rest of this paper is structured as follows. The related works are introduced by us in Section2. In Section 3, the system model including physical domain and social domain is described. After that, we formulate the problem in Section 4. Then, we present a JADQN framework to design caching strategy in Section 5. Next, we analyze the CiaoDVD dataset and discuss the simulation results in Section 6. Finally, we give the conclusion and the future work in Section 7. The important notations used in this paper are listed in Table 1.Table 1
Summary of notations.
NotationInterpretationMThe number of active usersNThe number of inactive usersSnRemaining space size of inactive usernτDelay of download contentsΓDelay thresholdTrm,nTrust degree between userm and user nSimm,nSimilarity between userm and user nHCache hit ratiostState at decision epochtat=xt,ytA joint action at a decision epochtrtReward at a decision epochtγDiscount rateϵExplore rate


## 2. Related Works

A huge number of researches combine D2D-assisted caching and social attributes in the existing literatures. In [9], the authors have utilized the social relationship between users to design the caching scheme at the network edge. In [10], a genetic algorithm-based collaborate caching strategy has been used to motivate the collaboration nodes; further, a submodular function has been used to optimize energy consumption. In [11], an incentive mechanism has been designed to minimize the cost of the BS, and the users who have shared contents with others by D2D communications could obtain a reward. However, these works have modeled the request probability of contents as a Zipf distribution. In fact, users send the requests based on their preference rather than the assumed Zipf distribution. Therefore, a prior knowledge-based learning algorithm for user preference has been exploited to enhance the performance of the caching policy [12]. In order to maximize the caching utility and improve the probability of content sharing between the devices, a caching algorithm based on user preference has been proposed [13]. Nevertheless, user relationships influence the quality of communication because the devices are carried by users. Besides, users are reluctant to share their resources and provide assistance to others who have weak social relationship with them due to the selfish nature.The problem can be solved by considering the social relationships between users including user intimacy, user similarity, and user trust. All of them can be used to select the nodes as a helper in caching contents. In addition, the users are willing to share the contents to the others in the same community with them. By this way, the cache resources of overall system can be utilized more effectively. In [14], the authors have investigated the caching strategy and cooperative distance to optimize the throughput of the system and improve energy efficiency, where user similarity is calculated by the set of common neighbors among users. In [15], the different time spans have been divided by temporal score information. A new measurement which weights similarity by the network adjacent matrices of diverse time spans has been presented. In [16], the authors have considered the similarity of user interest and adjusted some parameters to control the proportion of social attributes to generate stabler community. However, we should consider not only user similarity but also user trust to ensure transmission reliability. In [17], the authors have argued that the users trust each other if the connection exists between them. Besides, the trust of users can be divided into direct trust and indirect trust. Zhang and Zhong [18] have proposed a method to design trust model by trust transitivity feature. In [19], the authors have combined the direct trust of users and the indirect trust to obtain the trust similarity by the Pearson similarity formula.In addition, reinforcement learning (RL) is applied to solve the complex communication problems since the environment is becoming more and more complex [20]. Compared with traditional optimization theory, using RL to build environment model can resolve this kind of problem and achieve a significantly effect. In [21], the authors have used a JADQN framework to design an optimal offloading strategy. In [22], the authors have presented a joint framework which composed of mobile edge computing (MEC) and cached-enabled D2D to optimize the energy cost. In [23], the authors have formulated the caching problem as a Markov decision process (MDP) and proposed the RL-based cooperative caching strategy to learn the optimal policy to minimize the delay. In [24], the authors have proposed a model-free reinforcement learning- (RL-) based algorithm. Furthermore, the algorithm has weighted a large set of features including the object size, recency, and frequency of access to maximize the cache hit ratio. However, these works only have considered the influence of physical layer on the communication problems. It is complicated to integrate the social domain and physical domain because the conditions of the network environment (i.e., user similarity, user trust value, and wireless communication links) are dynamic. Thus, the complexity of the integrated network is very high, and it is difficult to solve the formulation optimization problem. Motivated by the previous analysis, we present a model consists of two layers including the physical factors and social attributes. Then, we design a strategy based on a JADQN framework to choose cache nodes and cache contents.


## 3. System Model

Figure1 illustrates the considered system model which is divided into physical domain model and social domain model. The social domain and the physical domain are modeled separately; if two users have low social relationship, they may not establish the communication link. In physical domain, we obtain the communication link quality by calculating the download delay. Besides, we introduce the dataset used in this paper. In social domain, we utilize the data from CiaoDVD dataset to judge user relationship by calculating user similarity and user trust. Then, the contents are allotted according to the active user preference.Figure 1
System model.


### 3.1. Physical Domain Model

In the physical domain, we consider a scenario of D2D-assisted networks, which is composed of mobile users and a base station (BS). In addition, we assume that the BS with huge cache ability could store all contents which can be represented byF=1,2,⋯,f,⋯,F. The users in the system are denoted by U=1,2,⋯,u,⋯,U. We suppose that each user takes a mobile smart device with D2D communication capability. The users will be divided into two categories based on their activeness: active users M=1,2,⋯,m,⋯,M and and inactive users which expressed by N=1,2,⋯,n,⋯,N. The inactive users have extra memory size, and they can provide assistance for the active users who are close to them.The active users send a large number of requests which cause them have no extra memory space. During off-peak periods, the inactive users with different cache size store the contents that active users are interested in, which are represented byP=1,2,⋯,p,⋯,P, P⊂F. In more details, P is the order of rating contents by active users. Then, inactive users provide the contents to their neighbor during peak periods.Assuming that the spectrum resources are orthogonality, which means that there is no interference between the inactive users and their neighbors. The active users communicate with the inactive users via D2D links which is subjected to Shannon bound. Thus, when the active userm gets content from the provider πj, the download delay can be calculated as
(1)τm,πj=BW·log21+Pπj/N0·gm,πj,where B denotes content size, W represents the channel bandwidth, Pπj is the transmitted power of πj, N0 is the noise power between πj and active user m, and gm,πj is the channel gain which consists of small-scale fading due to multipath effect and shadow fading in large-scale fading. Therefore, gm,πj can be expressed as follows:
(2)gm,πj=υϑm,πjξm,πjdm,πj−α,where υ represents the path loss constant which will be provided at Section 4. ϑm,πj represents the small-scale fading, and ξm,πj denotes the large-scale fading. dm,πj−α represents the distance from user m to πj, and α is the path loss factor.When an active userm sends a content request to the BS, the neighbor of user m has the precedence to provide the contents. The D2D links can be established between the active user m and many inactive users. Thus, the BS selects the inactive user n with the minimum communication delay τm,πn from the set of inactive users. Besides, the download delay between the inactive user n and m must satisfy the delay threshold Γ; otherwise, the inactive user n will be discarded. If none of the inactive users adjacent to the active user m can provide the requested content, the user m downloads the content from the BS and the delay denoted by τm,MBS.


### 3.2. Dataset Analysis

In this subsection, CiaoDVD dataset is introduced and analyzed. Based on CiaoDVD, Chen et al., utilized temporal score information to calculate user interest similarity [25]. In this paper, the similarity between users is calculated in three dimensions: user content rating, user preference for the genre of content, and the number of common friends between users. Besides, according to the CiaoDVD dataset, we can establish a trust topology for users.In the dataset, all users rate their favorite contents on a five-point scale: 1, 2, 3, 4, and 5. Moreover, 0 represents the content is not rated by the user. Thus, the score matrix of users is expressed as(3)Rv=U1U2U3⋮Um+nf1f2f3f4f5⋯fF53430⋯352405⋯004352⋯405430⋯055440⋯5.From (3), it can be found that the score of U1 for the content f1 is 5, and the score of U3 for the content f3 is 3.Each content in the dataset corresponds to a content type. There are 17 genres of movies in the dataset, and all of them have a fixed label. In addition, users may collect more than one content for the same type. By the statistics, the number of content that userm and user n collected can be expressed as
(4)Kv=J1J2J3J4J5⋯J17mn01321⋯5114112⋯0.From (4), it can be found that m collects three contents which belong to type j3 and two contents which belongs to type j4. In addition, type j17 is the favorite genre of m, and type j4 is the favorite genre of n.There are the user trust relationships in the dataset, and the following matrix denotes the trust degree of first five users.(5)Rv=U1U2U3U4U4U1U2U3U4U51110011010101110111000101.As shown in (5), if a trust relationship exists between two users, the trust value is 1; if there is no direct trust relationship between two users, the trust value is 0. Moreover, the trust relationship between users and users themselves is represented by 1.


### 3.3. Social Domain Model

In this subsection, CiaoDVD dataset is introduced and analyzed. Based on CiaoDVD, Khan et al., utilized temporal score information calculated user interest similarity [9]. In this paper, the similarity between users is calculated in three dimensions: user content rating, user preference for the genre of content, and the number of common friends between users. Besides, according to the CiaoDVD dataset, we can establish a trust topology for users.


#### 3.3.1. User Similarity Analysis

The Jaccard similarity algorithm is utilized to calculate the similarity in this paper. We consider three dimensions different from other literatures [26, 27], including user collected contents, user favorite content genre, and the number of common friends of the users. By this way, the characteristic set of user m and user n is expressed by Fm,n=fm,ni,Jm,n,frm,nu. In detail, fm,ni is the favorite list of the mth and the nth user, Jm,n is the movie genre that both user m and user n collected, and frm,nu denotes the common friend list of user m and user n.We suppose that once a user has rated the content, the content is considered to be put on their favorite lists. In this case, the content matric of userm and user n can be expressed as
(6)Rm=frm,n1frm,n2frm,n3frm,n4frm,n5⋯frm,ni01011⋯1,(7)Rn=frm,n1frm,n2frm,n3frm,n4frm,n5⋯frm,ni01011⋯1,where Rm is the set of favorite contents for user m and Rn is the set of favorite contents of user n. From (6) and (7), we can see that the contents collected by user m are IDs 1, 2, 3, 4, and 5. For user n, the content IDs with 2, 4, and 5 are collected.Km represents the movie genre list rated by the user m, and there are 17 genres in total dataset. KmJ1=1 represents user m prefers the J1; otherwise, KmJ1=0. It is noted that each user has only one favorite content genre.
(8)Km=J1J2J3J4J5⋯J1700100⋯0,(9)Kn=J1J2J3J4J5⋯J1701000⋯0.From (8), it is obvious that the favorite content of user m is genre J3. In the same way, from (9), it can be seen that user n is genre J2.Tm and Tn denote the friend list of user m and user n, respectively. frm,nu denotes the user list involving the friends of user m and the friends of user n, i.e., frm,nu=frmA⋃frnB, where frmA is the friend list of user m who has A friends and frnB is the friend list of user n who has B friends. If user u is the friend of user m, frmu=1; otherwise, frmu=0. Similarly, if user u is the friend of user n, frnu=1. Otherwise, frnu=0. Thus, the friend matrix of user m and user n can be expressed as follows:
(10)Tm=frm,n1frm,n2frm,n3frm,n4⋯frm,nu1011⋯0,(11)Tn=frm,n1frm,n2frm,n3frm,n4⋯frm,nu1100⋯0.In (10), obviously, user 1, user 3, and user 4 are familiar with user m, and user 2 is not the friend of user m. Similarly, in (11), user 1 and user 2 are familiar with user n, and user 3 and user 4 are not the friend of user n. Moreover, user 1 is the common friend between m and n.In that way, the feature set of userm can be expressed as Fm=Rm,Km,Tm. Similarly, the feature set of user n can be expressed as Fn=Rn,Kn,Tn. Thus, Jaccard similarity algorithm can be used as
(12)Simm,n=Fm⋂FnFm∪Fn.


#### 3.3.2. User Trust Analysis

Ifm and n contact directly m↔n, it is believed that there is a direct trust relationship, which can be denoted as DTm,n. However, not all users exist direct trust, which contributes to the trust matrix sparse. In fact, although there is no direct contact between the two users, it does not mean that users do not trust each other, which can be denoted as m∼n. At this point, the indirect trust can be calculated by using the transitivity of trust, denoted by PTm,n. In other words, if there is at least one set of intermediate users between m and n, O⊂U=o1,o2,⋯,ov, which enables the users trust each other: m⟶o1⟶o2⟶⋯⟶ov⟶n. Nevertheless, there are more than one intermediate user groups, and the trust network is regarded as a resistance network [26].
(13)1PTm,n=1DTm,o1+1DTo1,o2+1DTo1,ov+1DTov,n,where PTm,n∈0,1. According to six degrees of separation theory [27], a person can be able to know a stranger through six friends. In case of intermediate users more than six, PTm,n=0. In details, based on the data which expressed like Equation (5), we can know whether the two users trust each other directly. If two users have no direct trust and the intermediate users of them are less than six, we can utilize Equation (13) to calculate the indirect trust value between them. Otherwise, we regard the two users trust value is 0. Thus, the trust calculation formula can be given by
(14)Trm,n=DTm,n,ifm↔n,PTm,n,ifm∼n0,otherwise.&∃O&O≤6,BS selects usern from the set of inactive users adjacent to the active user m by weighing the link quality in physical domain and the relationship in social domain.


## 3.1. Physical Domain Model

In the physical domain, we consider a scenario of D2D-assisted networks, which is composed of mobile users and a base station (BS). In addition, we assume that the BS with huge cache ability could store all contents which can be represented byF=1,2,⋯,f,⋯,F. The users in the system are denoted by U=1,2,⋯,u,⋯,U. We suppose that each user takes a mobile smart device with D2D communication capability. The users will be divided into two categories based on their activeness: active users M=1,2,⋯,m,⋯,M and and inactive users which expressed by N=1,2,⋯,n,⋯,N. The inactive users have extra memory size, and they can provide assistance for the active users who are close to them.The active users send a large number of requests which cause them have no extra memory space. During off-peak periods, the inactive users with different cache size store the contents that active users are interested in, which are represented byP=1,2,⋯,p,⋯,P, P⊂F. In more details, P is the order of rating contents by active users. Then, inactive users provide the contents to their neighbor during peak periods.Assuming that the spectrum resources are orthogonality, which means that there is no interference between the inactive users and their neighbors. The active users communicate with the inactive users via D2D links which is subjected to Shannon bound. Thus, when the active userm gets content from the provider πj, the download delay can be calculated as
(1)τm,πj=BW·log21+Pπj/N0·gm,πj,where B denotes content size, W represents the channel bandwidth, Pπj is the transmitted power of πj, N0 is the noise power between πj and active user m, and gm,πj is the channel gain which consists of small-scale fading due to multipath effect and shadow fading in large-scale fading. Therefore, gm,πj can be expressed as follows:
(2)gm,πj=υϑm,πjξm,πjdm,πj−α,where υ represents the path loss constant which will be provided at Section 4. ϑm,πj represents the small-scale fading, and ξm,πj denotes the large-scale fading. dm,πj−α represents the distance from user m to πj, and α is the path loss factor.When an active userm sends a content request to the BS, the neighbor of user m has the precedence to provide the contents. The D2D links can be established between the active user m and many inactive users. Thus, the BS selects the inactive user n with the minimum communication delay τm,πn from the set of inactive users. Besides, the download delay between the inactive user n and m must satisfy the delay threshold Γ; otherwise, the inactive user n will be discarded. If none of the inactive users adjacent to the active user m can provide the requested content, the user m downloads the content from the BS and the delay denoted by τm,MBS.


## 3.2. Dataset Analysis

In this subsection, CiaoDVD dataset is introduced and analyzed. Based on CiaoDVD, Chen et al., utilized temporal score information to calculate user interest similarity [25]. In this paper, the similarity between users is calculated in three dimensions: user content rating, user preference for the genre of content, and the number of common friends between users. Besides, according to the CiaoDVD dataset, we can establish a trust topology for users.In the dataset, all users rate their favorite contents on a five-point scale: 1, 2, 3, 4, and 5. Moreover, 0 represents the content is not rated by the user. Thus, the score matrix of users is expressed as(3)Rv=U1U2U3⋮Um+nf1f2f3f4f5⋯fF53430⋯352405⋯004352⋯405430⋯055440⋯5.From (3), it can be found that the score of U1 for the content f1 is 5, and the score of U3 for the content f3 is 3.Each content in the dataset corresponds to a content type. There are 17 genres of movies in the dataset, and all of them have a fixed label. In addition, users may collect more than one content for the same type. By the statistics, the number of content that userm and user n collected can be expressed as
(4)Kv=J1J2J3J4J5⋯J17mn01321⋯5114112⋯0.From (4), it can be found that m collects three contents which belong to type j3 and two contents which belongs to type j4. In addition, type j17 is the favorite genre of m, and type j4 is the favorite genre of n.There are the user trust relationships in the dataset, and the following matrix denotes the trust degree of first five users.(5)Rv=U1U2U3U4U4U1U2U3U4U51110011010101110111000101.As shown in (5), if a trust relationship exists between two users, the trust value is 1; if there is no direct trust relationship between two users, the trust value is 0. Moreover, the trust relationship between users and users themselves is represented by 1.


## 3.3. Social Domain Model

In this subsection, CiaoDVD dataset is introduced and analyzed. Based on CiaoDVD, Khan et al., utilized temporal score information calculated user interest similarity [9]. In this paper, the similarity between users is calculated in three dimensions: user content rating, user preference for the genre of content, and the number of common friends between users. Besides, according to the CiaoDVD dataset, we can establish a trust topology for users.


### 3.3.1. User Similarity Analysis

The Jaccard similarity algorithm is utilized to calculate the similarity in this paper. We consider three dimensions different from other literatures [26, 27], including user collected contents, user favorite content genre, and the number of common friends of the users. By this way, the characteristic set of user m and user n is expressed by Fm,n=fm,ni,Jm,n,frm,nu. In detail, fm,ni is the favorite list of the mth and the nth user, Jm,n is the movie genre that both user m and user n collected, and frm,nu denotes the common friend list of user m and user n.We suppose that once a user has rated the content, the content is considered to be put on their favorite lists. In this case, the content matric of userm and user n can be expressed as
(6)Rm=frm,n1frm,n2frm,n3frm,n4frm,n5⋯frm,ni01011⋯1,(7)Rn=frm,n1frm,n2frm,n3frm,n4frm,n5⋯frm,ni01011⋯1,where Rm is the set of favorite contents for user m and Rn is the set of favorite contents of user n. From (6) and (7), we can see that the contents collected by user m are IDs 1, 2, 3, 4, and 5. For user n, the content IDs with 2, 4, and 5 are collected.Km represents the movie genre list rated by the user m, and there are 17 genres in total dataset. KmJ1=1 represents user m prefers the J1; otherwise, KmJ1=0. It is noted that each user has only one favorite content genre.
(8)Km=J1J2J3J4J5⋯J1700100⋯0,(9)Kn=J1J2J3J4J5⋯J1701000⋯0.From (8), it is obvious that the favorite content of user m is genre J3. In the same way, from (9), it can be seen that user n is genre J2.Tm and Tn denote the friend list of user m and user n, respectively. frm,nu denotes the user list involving the friends of user m and the friends of user n, i.e., frm,nu=frmA⋃frnB, where frmA is the friend list of user m who has A friends and frnB is the friend list of user n who has B friends. If user u is the friend of user m, frmu=1; otherwise, frmu=0. Similarly, if user u is the friend of user n, frnu=1. Otherwise, frnu=0. Thus, the friend matrix of user m and user n can be expressed as follows:
(10)Tm=frm,n1frm,n2frm,n3frm,n4⋯frm,nu1011⋯0,(11)Tn=frm,n1frm,n2frm,n3frm,n4⋯frm,nu1100⋯0.In (10), obviously, user 1, user 3, and user 4 are familiar with user m, and user 2 is not the friend of user m. Similarly, in (11), user 1 and user 2 are familiar with user n, and user 3 and user 4 are not the friend of user n. Moreover, user 1 is the common friend between m and n.In that way, the feature set of userm can be expressed as Fm=Rm,Km,Tm. Similarly, the feature set of user n can be expressed as Fn=Rn,Kn,Tn. Thus, Jaccard similarity algorithm can be used as
(12)Simm,n=Fm⋂FnFm∪Fn.


### 3.3.2. User Trust Analysis

Ifm and n contact directly m↔n, it is believed that there is a direct trust relationship, which can be denoted as DTm,n. However, not all users exist direct trust, which contributes to the trust matrix sparse. In fact, although there is no direct contact between the two users, it does not mean that users do not trust each other, which can be denoted as m∼n. At this point, the indirect trust can be calculated by using the transitivity of trust, denoted by PTm,n. In other words, if there is at least one set of intermediate users between m and n, O⊂U=o1,o2,⋯,ov, which enables the users trust each other: m⟶o1⟶o2⟶⋯⟶ov⟶n. Nevertheless, there are more than one intermediate user groups, and the trust network is regarded as a resistance network [26].
(13)1PTm,n=1DTm,o1+1DTo1,o2+1DTo1,ov+1DTov,n,where PTm,n∈0,1. According to six degrees of separation theory [27], a person can be able to know a stranger through six friends. In case of intermediate users more than six, PTm,n=0. In details, based on the data which expressed like Equation (5), we can know whether the two users trust each other directly. If two users have no direct trust and the intermediate users of them are less than six, we can utilize Equation (13) to calculate the indirect trust value between them. Otherwise, we regard the two users trust value is 0. Thus, the trust calculation formula can be given by
(14)Trm,n=DTm,n,ifm↔n,PTm,n,ifm∼n0,otherwise.&∃O&O≤6,BS selects usern from the set of inactive users adjacent to the active user m by weighing the link quality in physical domain and the relationship in social domain.


## 3.3.1. User Similarity Analysis

The Jaccard similarity algorithm is utilized to calculate the similarity in this paper. We consider three dimensions different from other literatures [26, 27], including user collected contents, user favorite content genre, and the number of common friends of the users. By this way, the characteristic set of user m and user n is expressed by Fm,n=fm,ni,Jm,n,frm,nu. In detail, fm,ni is the favorite list of the mth and the nth user, Jm,n is the movie genre that both user m and user n collected, and frm,nu denotes the common friend list of user m and user n.We suppose that once a user has rated the content, the content is considered to be put on their favorite lists. In this case, the content matric of userm and user n can be expressed as
(6)Rm=frm,n1frm,n2frm,n3frm,n4frm,n5⋯frm,ni01011⋯1,(7)Rn=frm,n1frm,n2frm,n3frm,n4frm,n5⋯frm,ni01011⋯1,where Rm is the set of favorite contents for user m and Rn is the set of favorite contents of user n. From (6) and (7), we can see that the contents collected by user m are IDs 1, 2, 3, 4, and 5. For user n, the content IDs with 2, 4, and 5 are collected.Km represents the movie genre list rated by the user m, and there are 17 genres in total dataset. KmJ1=1 represents user m prefers the J1; otherwise, KmJ1=0. It is noted that each user has only one favorite content genre.
(8)Km=J1J2J3J4J5⋯J1700100⋯0,(9)Kn=J1J2J3J4J5⋯J1701000⋯0.From (8), it is obvious that the favorite content of user m is genre J3. In the same way, from (9), it can be seen that user n is genre J2.Tm and Tn denote the friend list of user m and user n, respectively. frm,nu denotes the user list involving the friends of user m and the friends of user n, i.e., frm,nu=frmA⋃frnB, where frmA is the friend list of user m who has A friends and frnB is the friend list of user n who has B friends. If user u is the friend of user m, frmu=1; otherwise, frmu=0. Similarly, if user u is the friend of user n, frnu=1. Otherwise, frnu=0. Thus, the friend matrix of user m and user n can be expressed as follows:
(10)Tm=frm,n1frm,n2frm,n3frm,n4⋯frm,nu1011⋯0,(11)Tn=frm,n1frm,n2frm,n3frm,n4⋯frm,nu1100⋯0.In (10), obviously, user 1, user 3, and user 4 are familiar with user m, and user 2 is not the friend of user m. Similarly, in (11), user 1 and user 2 are familiar with user n, and user 3 and user 4 are not the friend of user n. Moreover, user 1 is the common friend between m and n.In that way, the feature set of userm can be expressed as Fm=Rm,Km,Tm. Similarly, the feature set of user n can be expressed as Fn=Rn,Kn,Tn. Thus, Jaccard similarity algorithm can be used as
(12)Simm,n=Fm⋂FnFm∪Fn.


## 3.3.2. User Trust Analysis

Ifm and n contact directly m↔n, it is believed that there is a direct trust relationship, which can be denoted as DTm,n. However, not all users exist direct trust, which contributes to the trust matrix sparse. In fact, although there is no direct contact between the two users, it does not mean that users do not trust each other, which can be denoted as m∼n. At this point, the indirect trust can be calculated by using the transitivity of trust, denoted by PTm,n. In other words, if there is at least one set of intermediate users between m and n, O⊂U=o1,o2,⋯,ov, which enables the users trust each other: m⟶o1⟶o2⟶⋯⟶ov⟶n. Nevertheless, there are more than one intermediate user groups, and the trust network is regarded as a resistance network [26].
(13)1PTm,n=1DTm,o1+1DTo1,o2+1DTo1,ov+1DTov,n,where PTm,n∈0,1. According to six degrees of separation theory [27], a person can be able to know a stranger through six friends. In case of intermediate users more than six, PTm,n=0. In details, based on the data which expressed like Equation (5), we can know whether the two users trust each other directly. If two users have no direct trust and the intermediate users of them are less than six, we can utilize Equation (13) to calculate the indirect trust value between them. Otherwise, we regard the two users trust value is 0. Thus, the trust calculation formula can be given by
(14)Trm,n=DTm,n,ifm↔n,PTm,n,ifm∼n0,otherwise.&∃O&O≤6,BS selects usern from the set of inactive users adjacent to the active user m by weighing the link quality in physical domain and the relationship in social domain.


## 4. Problem Formulation

The caching strategy can be modeled as a Markov decision process (MDP) which is regarded as the states in the future depending only on the present state rather than the past history. In this paper, we use deepQ-networks (DQN) framework to design the caching strategy, including matching the users and allotting the contents to the helper nodes, which can be expressed by ψ=umn,yn, where umn denotes the inactive user m paired with active user n. That is, BS puts the contents in the remaining cache space of user n. yn is the content list which cached in user n.The cache hit ratio in this paper is the ratio of the hit times of the active user all requested during a period of timeT, which can be formulated as
(15)H=∑t=1T∑p=1Pthm,ntp∑t=1TPt=1P∑t=1T∑p=1Pthm,ntp,where hm,nt∈0,1 indicates the cache status by the active user m at time slot t. It is noted that each content has a corresponding content genre. We suppose that the active user m is paired with the inactive user n and sends a request to download the content p of genre Jf. It is called a hit if Jf exists in the cache space of the inactive user n, denoted by hm,nt=1. In addition, Pt expresses the count of request during the period.Both matching users and choosing contents which should be cached in the remaining space of inactive users can affect the cache hit rate. Therefore, the goal is to obtain an optimal policyψ∗ to find the proper content placement location and cache the active user prefer contents for maximizing the cache hit rate H.
(16)maxψ1P∑t=1T∑p=1Pthm,ntp,s.t.C1:∑pcn,p≤Sn,C2:τm,πj<Γ,C3:hm,ntpϵ0,1,C4:0≤Simm,n≤1,C5:0≤Trm,n≤1,where p represents the content that user n cached and cn,p is the size of p. Furthermore, C1 denotes that cn,p shall be smaller than the remain storage space Sn of inactive user n. C2 denotes that the delay of caching content for user m should be less than the delay threshold Γ. Otherwise, the communication link cannot be established. C3 represents two cache states, hm,ntp=1 means the content can be provided by the inactive users nearly, and hm,ntp=0 means the content is not cached by the adjacent inactive users. C4 means the trust value located in the range of 0,1, and C5 means the similarity value located in the range of 0,1.


## 5. Joint Action DQN-Based Caching Strategy

In this section, we propose a caching strategy based on a JADQN framework. We treated the BS as an agent who knows the cache status of all users. When an active user sends a content request, the agent observes the state space to make a joint action, including selecting inactive users and allotting the caching content placement process. Finally, the goal of maximizing cache hit is achieved.It is well known thatQ-learning is a widely used model-free reinforcement learning. However, due to the complex environment, the state space is huge, and one of the challenges in Q-learning is to store the state in Q table. Thus, DQN who combines the advantages of Q-leaning and neural network is proposed. The DQN framework consists of an agent, an environment, and three crucial elements including the state space S, the action space A, and a reward function R. The agent learns and makes decisions through interacting with the environment in discrete time steps. In each decision time slot, the agent observes the current state of the environment st∈S and takes an action at∈A, then the agent will obtain a reward rt∈R. We utilize the DQN framework to cope with the problem and define the key elements according to our system model as follows.


### 5.1. State Space

The environment is defined as the communication conditions of D2D users, which consists of physical condition and social condition. In this case, the statest includes the inactive users remaining cache space Snt which expresses the users caching ability in the physical domain, the favorite movie genre of user m is denoted by Jm; the trust degree and the similarity between user m and user n are denoted by Trm,n and Simm,n, respectively. It is noted that the Jm, Trm,n, and Simm,n express the characteristics of users in social domain. In addition, Snt, Trm,n, and Simm,n are detailed described in Section 4. At the beginning of each decision slot t, the agent selects different users whose remaining space is different and the social relationships between them change dynamically. Thus, st can be written as
(17)st=Simm,n,Trm,n,Snt,Jm.


### 5.2. Action Space

There areN inactive users and P contents to be cached. In order to restrict the size of the action space, it is assumed that only one inactive user can be selected to place cache contents when an active user requests content. The agent chooses a joint action at to decision where and what to cache based on the observed environment state st of time slot t. That is, the agent completes the pairing process between active user m and inactive user n. At the same time, the contents are allocated to the remaining cache space of the inactive user n paired with the active user m. Thus, the action can be expressed by
(18)at=x1t,y1t,x2t,y2t,⋯,xnt,ynt.The dimension of the action space can be expressed asN×P, where xnt∈0,1 represents the paring status of active user m. xnt=0 denotes that BS selects other inactive user rather than inactive user n. xnt=1 denotes that the inactive user n is selected by BS to help the active user cache contents.When the active userm is paired with the inactive user n, the agent will allocate contents to the user n according to the active user m preference and the remaining cache space of the selected user n. Besides, ynt=1,2,⋯,k, k∈P represents a cache list of the inactive user. However, the inactive users have limited cache space; the inactive user selected by the agent may not cache the content that the active user requests. If the content genre does not exist in the memory of the inactive users adjacent to the active user, the active user downloads the content from the BS.


### 5.3. Reward Design

In the caching strategy described in Section4, the objective is to maximize the cache hit ratio within a certain time constraint T. When we give a particular state st and a joint action at at time slot t, the agent obtains a larger reward when the selected action results in a higher cumulative hit times before the next decision epoch. Hence, the reward is defined as a sum of hit count on each cache slot, and it can be formulated as
(19)rt=∑iPhm,ntt+1−hm,ntt.The joint action DQN framework is shown in Figure2, which consists of two neural networks which have the same structure but their parameters are different, that is, target-Q network with θ and main network with θ−. The value of main network is represented as Q=st,xt,yt,θ obtained by at=xt,yt. Note that a joint action DQN and Double DQN are different. JADQN used in this paper obtains the target-Q value by greedy algorithm. Double DQN obtains the target-Q by two steps, that is, decouple the selection of target-Q and the calculation of target-Q value. The target-Q value of JADQN is calculated like the DQN apart from making a joint action in a decision slot, and the target-Q value can be calculated by
(20)Q′=rt+1+γmaxxt,ytQst+1,xt,yt;θ−.Figure 2
A joint action DQN framework.Since exploration and exploitation are contradictory, the most general way to balance the tradeoff between them is by usingϵ-greedy policy. Specially, the agent chooses the actions that have been tried and shown to be effective with probability 1−ϵ or the action is chosen randomly with probability ϵ to explore the state space and action space. We train the network through experience replay to learn the caching strategy. In detail, the agent observes the environment state st to make a joint action xt,yt, which obtains a reward and affects the state of next moment st+1. After that, the agent stores the transition tuple st,xt,yt,rt,st+1 in the replay memory D. The transition in D consists of the current state st, the current action at, the reward rt calculated by formula (19), and the state of the next decision epoch after the environment receives the action st+1. Furthermore, at each episode, minibatches are tokens from D. The network updates parameter θ and minimizes the mean square error as follows:
(21)L=Q′−Q.Algorithm 1: Caching strategy based on a JADQN.
Input: inactive users remaining space: SN, transmission distance of users: dN, delay threshold: Γ, the parameter of main Q network: θ, the parameter of target Q network: θ− and read the data from dataset.Output: the cache hit ratio H.Initialize:θ−⟵θ, memory size: D, leaning rate β, explore rate: ϵ, discount rate: γ, the number of iterations:E, the number of minibatches:M.ifτm,n<Γthenfor episode=1,2,⋯EdoUpdateSN;Pfor t =1,2,⋯TdoRandom generation probability:ρifρ<ϵ:Randomly generated action:xt,ytelse:Obtained a joint action by according toxt,yt=argmaxxt,ytQst,xt,yt,θend ifObservest+1Store the transactionst,at,st+1,at+1in replay memory D, uniformly sample minibatches from DOptimize error betweenQ-network and learning targets, using variant of stochastic gradient descentθ=θ−βdL/dθEach stepC updates the parameters of target Q-networkθ−⟵θend forend forend ifThe proposed solution is shown in Algorithm1, and the algorithm is divided into six steps as follows.Step 1.
Input the variables and initialize them including the remaining space set of inactive usersSN=S1,S2,⋯,Sn, the communication delay set between inactive users and the active user dN=d1,d2,⋯,dn, and the delay threshold Γ. Besides, we set the parameters of neural networks as θ and θ−, respectively, the number of iterations asE, the minibatch size as M, and the replay memory size as D. At the same time, we read the data from the CiaoDVD dataset to obtained the user preference P and calculate the user similarity Simm,n, user trust Trm,n.Step 2.
The communication delay between the inactive user and the active user determines whether to discard the inactive user. In detail, the agent discards the inactive user and chooses another one if the cache download delay is greater than the thresholdΓ.Step 3.
In each episode, update the users preferenceP and the remaining space of inactive users SN. The agent observes the current state space when the active user sends the content request based on their preference P. Then, in the each training step, the agent takes the joint action by the value network and ϵ-greedy policy. We used the replay memory D to store the transition tuple st,at,rt,st+1, in order to break up the correlation between the data.Step 4.
Sample minibatches fromD randomly and uniformly. The networks are training by utilizing variant of stochastic gradient descent.Step 5.
Update the target-Q network parameters for each C steps and train the network by calculating the loss in formula (21).Step 6.
Calculate the cache hit ratio once the active user sends requests until the algorithm converges. Then, output the cache hit ratioH.


## 5.1. State Space

The environment is defined as the communication conditions of D2D users, which consists of physical condition and social condition. In this case, the statest includes the inactive users remaining cache space Snt which expresses the users caching ability in the physical domain, the favorite movie genre of user m is denoted by Jm; the trust degree and the similarity between user m and user n are denoted by Trm,n and Simm,n, respectively. It is noted that the Jm, Trm,n, and Simm,n express the characteristics of users in social domain. In addition, Snt, Trm,n, and Simm,n are detailed described in Section 4. At the beginning of each decision slot t, the agent selects different users whose remaining space is different and the social relationships between them change dynamically. Thus, st can be written as
(17)st=Simm,n,Trm,n,Snt,Jm.


## 5.2. Action Space

There areN inactive users and P contents to be cached. In order to restrict the size of the action space, it is assumed that only one inactive user can be selected to place cache contents when an active user requests content. The agent chooses a joint action at to decision where and what to cache based on the observed environment state st of time slot t. That is, the agent completes the pairing process between active user m and inactive user n. At the same time, the contents are allocated to the remaining cache space of the inactive user n paired with the active user m. Thus, the action can be expressed by
(18)at=x1t,y1t,x2t,y2t,⋯,xnt,ynt.The dimension of the action space can be expressed asN×P, where xnt∈0,1 represents the paring status of active user m. xnt=0 denotes that BS selects other inactive user rather than inactive user n. xnt=1 denotes that the inactive user n is selected by BS to help the active user cache contents.When the active userm is paired with the inactive user n, the agent will allocate contents to the user n according to the active user m preference and the remaining cache space of the selected user n. Besides, ynt=1,2,⋯,k, k∈P represents a cache list of the inactive user. However, the inactive users have limited cache space; the inactive user selected by the agent may not cache the content that the active user requests. If the content genre does not exist in the memory of the inactive users adjacent to the active user, the active user downloads the content from the BS.


## 5.3. Reward Design

In the caching strategy described in Section4, the objective is to maximize the cache hit ratio within a certain time constraint T. When we give a particular state st and a joint action at at time slot t, the agent obtains a larger reward when the selected action results in a higher cumulative hit times before the next decision epoch. Hence, the reward is defined as a sum of hit count on each cache slot, and it can be formulated as
(19)rt=∑iPhm,ntt+1−hm,ntt.The joint action DQN framework is shown in Figure2, which consists of two neural networks which have the same structure but their parameters are different, that is, target-Q network with θ and main network with θ−. The value of main network is represented as Q=st,xt,yt,θ obtained by at=xt,yt. Note that a joint action DQN and Double DQN are different. JADQN used in this paper obtains the target-Q value by greedy algorithm. Double DQN obtains the target-Q by two steps, that is, decouple the selection of target-Q and the calculation of target-Q value. The target-Q value of JADQN is calculated like the DQN apart from making a joint action in a decision slot, and the target-Q value can be calculated by
(20)Q′=rt+1+γmaxxt,ytQst+1,xt,yt;θ−.Figure 2
A joint action DQN framework.Since exploration and exploitation are contradictory, the most general way to balance the tradeoff between them is by usingϵ-greedy policy. Specially, the agent chooses the actions that have been tried and shown to be effective with probability 1−ϵ or the action is chosen randomly with probability ϵ to explore the state space and action space. We train the network through experience replay to learn the caching strategy. In detail, the agent observes the environment state st to make a joint action xt,yt, which obtains a reward and affects the state of next moment st+1. After that, the agent stores the transition tuple st,xt,yt,rt,st+1 in the replay memory D. The transition in D consists of the current state st, the current action at, the reward rt calculated by formula (19), and the state of the next decision epoch after the environment receives the action st+1. Furthermore, at each episode, minibatches are tokens from D. The network updates parameter θ and minimizes the mean square error as follows:
(21)L=Q′−Q.Algorithm 1: Caching strategy based on a JADQN.
Input: inactive users remaining space: SN, transmission distance of users: dN, delay threshold: Γ, the parameter of main Q network: θ, the parameter of target Q network: θ− and read the data from dataset.Output: the cache hit ratio H.Initialize:θ−⟵θ, memory size: D, leaning rate β, explore rate: ϵ, discount rate: γ, the number of iterations:E, the number of minibatches:M.ifτm,n<Γthenfor episode=1,2,⋯EdoUpdateSN;Pfor t =1,2,⋯TdoRandom generation probability:ρifρ<ϵ:Randomly generated action:xt,ytelse:Obtained a joint action by according toxt,yt=argmaxxt,ytQst,xt,yt,θend ifObservest+1Store the transactionst,at,st+1,at+1in replay memory D, uniformly sample minibatches from DOptimize error betweenQ-network and learning targets, using variant of stochastic gradient descentθ=θ−βdL/dθEach stepC updates the parameters of target Q-networkθ−⟵θend forend forend ifThe proposed solution is shown in Algorithm1, and the algorithm is divided into six steps as follows.Step 1.
Input the variables and initialize them including the remaining space set of inactive usersSN=S1,S2,⋯,Sn, the communication delay set between inactive users and the active user dN=d1,d2,⋯,dn, and the delay threshold Γ. Besides, we set the parameters of neural networks as θ and θ−, respectively, the number of iterations asE, the minibatch size as M, and the replay memory size as D. At the same time, we read the data from the CiaoDVD dataset to obtained the user preference P and calculate the user similarity Simm,n, user trust Trm,n.Step 2.
The communication delay between the inactive user and the active user determines whether to discard the inactive user. In detail, the agent discards the inactive user and chooses another one if the cache download delay is greater than the thresholdΓ.Step 3.
In each episode, update the users preferenceP and the remaining space of inactive users SN. The agent observes the current state space when the active user sends the content request based on their preference P. Then, in the each training step, the agent takes the joint action by the value network and ϵ-greedy policy. We used the replay memory D to store the transition tuple st,at,rt,st+1, in order to break up the correlation between the data.Step 4.
Sample minibatches fromD randomly and uniformly. The networks are training by utilizing variant of stochastic gradient descent.Step 5.
Update the target-Q network parameters for each C steps and train the network by calculating the loss in formula (21).Step 6.
Calculate the cache hit ratio once the active user sends requests until the algorithm converges. Then, output the cache hit ratioH.


## 6. Simulation Results

In this section, we analyze the performance of the proposed strategy by utilizing the dataset CiaoDVD. We divide the dataset into two parts, one part is used as the historical request data, the other is regarded as the user requests. In addition, the preference of users is calculated based on the historical request data. We compare our strategy with several benchmark algorithms including least recently used (LRU) algorithm, least frequently used (LFU) algorithm, and first input first output (FIFO) algorithm to validate the effectiveness of the proposed caching strategy.In detail, when the cache space is filled with the contents, LRU algorithm evicts the earliest last used time content and caches the request content. For LFU algorithm, the least frequently request content is replaced. For FIFO algorithm, the content input first is replaced first. Moreover, we compare two algorithms (i.e., randomly select users and randomly cache: RU-RC, randomly select users and select users according to user preference: RU-PC) in order to show the performance advantages brought by selecting users. Table2 lists the main simulation parameters [28, 29].Table 2
Simulation parameters.
ParameterValueTransmission power of BS46 dBmTransmission power of users23 dBmNoise power-114 dBmPath-loss constantυ0.01Path-loss exponentα4.0Multipath fadingϑ8 dbShadowingξ-114 dBmSystem bandwidth10 MHzDiscount rateγ0.9Learning rateβ0.001Minibatch sizeM2000γDiscount rateExplore rateϵ0.02We select an active user and 11 inactive users from the CiaoDVD dataset randomly, which concludes 72665 users, then we analyze their social relationship. It is easy to get the trust value of an active user and eleven inactive users by trust topology network in (4). As shown in Figure 3, the value range of similarity and trust is between 0 and 1. It unable to establish trust relationship with the trust topology of six close friends between the user m and the user whose IDs are 3, 4, 5, 6, 8, and 9, respectively. Thus, the trust value of them is 0. Based on all users’ historical information, the preference for movie genres can be obtained. We calculate the similarity by formula (5) and obtain the value in the range from 0.1 to 0.2. Besides, it can be seen that the inactive user 2 has the highest similarity to the active user. In addition, the value of the similarity is about 0.16. The trust between user 11 and the selected active user is the highest, with a value of 1.Figure 3
User similarity and user trust analysis.Figure4 shows user preferences obtained from historical movie requests of different users. According to the dataset, we divide the users into active users and inactive users depending on their activeness. In detail, from user 1 to user 11 are inactive users, and user 12 is active user. Besides, the dataset includes a total of 17 movie types such as action, comedy, and love. In addition, the users in the dataset have different genre preference. The value represents the degree of user preference for the different genres which is defined as the ratio of contents collected by the user in this paper. That is, in all the contents that the user collected, the user preference is the proportion of the contents which belongs to the genre to the total amount of contents. It can be seen that user 2’s preference value for genre 17 is about 0.3. Moreover, user 2’s favorite movie genre is 17, and user 12’s preference value for movie genre 16 is about 0.2, and user 12’s favorite movie genre is 15. Based on the analysis of the users preference, we can utilize them in the caching strategy.Figure 4
User preference analysis.The DQN framework consists of 3 hidden layers, and the rectified linear unit (ReLU) is employed as an activation function. The learning rate is set as 0.001 by PRMProp optimizer to update network parameters. Besides, this is set as 0.02. We show the loss of each training episode as the number of training iterations increasing in Figure5. It can be observed that the neural network continues to explore actions to get better reward in the early stages of training. Therefore, it has a large loss. However, as the number of episodes increases, the loss decreases in each training episode which demonstrates the effectiveness of our algorithm. In addition, when the number of training episode reaches 2000, the performance gradually converges and the DQN framework explores all the possible states and enables the agent select the best joint action depended on the states and rewards. Based on such an observation, we set the training episode as 3000 to guarantee convergence of DQN.Figure 5
Loss for each training episode with increasing iterations.The average download delay performance of caching strategies with different cached genre number is shown in Figure6. It can be observed that the number of cache genre influences the average delay. With the cache genre number increases, the cache average delay decreases obviously. When the user space only caches fewer genres, the average download delay is higher. Since the number of user cache genres is limited, it is difficult for active users to obtain the requested content from their neighbor. If they want to get the content only cache from the BS where far away from them, it decides the higher cache download delay. As the genres of contents cached by the inactive users increase, the more contents that active users interested in are cached. In this case, the average download delay gradually decreases. For the genre of content that inactive user could cache is 5 and the delay threshold is 30 ms, the average download delay of presented strategy is 46 ms. When the delay threshold is set as 20 ms, the average download delay of presented strategy is 46 ms. When the delay threshold is set as 40 ms, the average download delay of presented strategy is 52 ms. Based on these results, it can be found that the delay threshold effects the cache download delay. For instance, the cache download delay decreases around 32.6% when the delay threshold is set as 30 ms compared with 40 ms. However, the cache download delay decreases around 13.0% when the delay threshold is set as 30 ms compared with 20 ms. Based on this observation, we believe that the larger delay threshold does not mean the lower cache delay, there may be an optimal value for the delay threshold.Figure 6
User cache genre vs. average delay.Figure7 depicts the cache download delay performance of the system under different delay threshold. With the delay threshold increases, the average download delay curve of active user cache contents shows decreases firstly, then the value increases over threshold, and the average download delay tends to be stable finally. The main reason is that our proposed caching strategy is affected by not only social attributes but also physical factors, i.e., the similarity and trust degree between users in social domain and the quality of communication links in physical domain. However, users with high similarity and trust may be far away from active users. When the value of delay threshold is set small, part of inactive users will be weeded out. This is because the distance between them to the active user is too far. They are unable to establish a communication link even if they have a high degree of similarity and trust to the active user. It can be seen that the delay threshold is 30 ms; a concave point is reached. When the delay threshold increases from 30 ms to 40 ms, the average download delay of the content requested by active users shows a trend of slow increase. The main reason is the number of users satisfying constraint in the physical domain. In this case, the agent prefers selecting inactive users who have closed relationships with the active user in the social domain to assist cache contents. In this way, the security and reliability of the communication can be ensured. However, the quality of communication links between the active user and the inactive users who are selected by the agent may not be optimal even if they satisfied the delay threshold. In addition, when the delay threshold is set as 45 ms, all inactive users who can provide services for active users meet the requirements of the physical domain. Therefore, if the download delay continues to rise, the average download delay of active users remains unchanged.Figure 7
Delay threshold vs. average delay.In Figure8, we show the tendency of cache hit ratio with the different cached genre number. It can be observed that the cache hit ratio is increasing when the cache genre number inactive users can cache additionally changes from 1 to 6. In addition, as the delay threshold increases, the cache hit ratio increases accordingly. As the delay threshold is set as 20 ms, only a small part of inactive users satisfied the physical condition. Thus, not all inactive users could provide services and cache contents for active users, which results in low cache hit ratio. However, with the delay threshold increases, more and more users meet the condition result that the cache hit ratio is improved. For example, when the delay threshold is set as 40 ms, and the inactive user can store 5 movie genres, the cache hit ratio is 0.78. When the delay threshold is set as 20 ms, the cache ratio is 0.47. Hence, we can know that the cache hit ratio is improved by about 47.7%, when the delay threshold increases from 20 ms to 30 ms.Figure 8
User cache genre vs. cache hit ratio.Figure9 demonstrates that the cache hit ratio brought by our caching strategy is significantly better than other caching strategies, when the leaning-based cache algorithm converges. The main reason is that the proposed strategy considers to select users according the social relationship between users and the quality of communication links. With the algorithm converges, the agent has the ability to choose the inactive user who is much proper than other users and provide the assistance to the active user. Thus, our strategy can obtain the better performance when the number of inactive users cached genre that active user prefers changes from 1 to 6. In addition, when the extra space of inactive user is small, the contents that the user cached are sparse. All the algorithms show the low cache hit ratio because the user cannot satisfy all demands. For example, when the delay threshold is set as 40 ms and the active users on the number of cache content genre is 3, the proposed caching strategy makes the cache hit ratio reach 0.51. The cache hit ratio of RU-PC caching strategy is 0.37, and RU-PC caching strategy is 0.16 under the same conditions. In this case, the cache hit ratio of a joint action based on DQN framework to select users and the contents is improved 37.8% than that of RU-PC.Figure 9
User cache genre vs. cache hit ratio.In Figure10, we show the impact of the genre number the inactive user could cache additionally on the average download delay. When the delay threshold is set as 40 ms, the average download delay of the algorithm proposed in this paper decreases gradually with the number of genres that inactive users cached increases. The main reason is that the neighboring inactive users provide assistant instead to obtain the contents from the BS is increasing. The content is placed closer and closer to active users; thus, the average download delay is reduced when the active users send the requests. However, the download delay performance fluctuates in the case of randomly cached content and randomly selected users. For RU-PC, the active users’ interested contents which are obtained by historical requests are more likely to be cached. Therefore, its performance is better than RU-RC, but worse than the proposed strategy in this paper. When the inactive users could extraly cache 4 genres, the average download delay of the proposed algorithm is 46 ms, and the average download delay of RU-PC algorithm is 63 ms. Compared with RU-PC algorithm, the average download delay of the proposed caching strategy is reduced by about 36.9%.Figure 10
User cache genre vs. average delay.Figure11 illustrates the trend of cache hit ratio with the increasing number of content requests. Obviously, the cache hit ratio of LRU, LFU, and FIFO appear a tendency to decline and then level off gradually. However, the cache hit ratio of the proposed strategy fluctuates with the number of content requests increases. If the users can only cache four content genres, these three benchmark algorithms update the content when the preset storage space is filled with contents. It can be seen that our strategy shows much better performance than other algorithms. In more details, it is shown that the cache hit ratio of our strategy floats around 0.9. When the request number is 125, the cache hit ratio of our strategy improves by 49.1% compared with LRU algorithm.Figure 11
Request number vs. cache hit ratio.As shown in Figure12, with the increasing of content requests, the average download latency of content appears a trend that climbs up and then declines. Compared with the benchmark algorithms, the proposed strategy achieves better performance. If the inactive users nearby have no cached content, the content will be obtained from the BS, which results in much higher communication delay. As the D2D communication delay threshold is 30 ms, the download delay of the proposed strategy fluctuates between 40 ms and 45 ms. When the active user sends 100 content requests, compared with FIFO algorithm, the delay is reduced by 38.2%.Figure 12
Request number vs. average delay.


## 7. Conclusions

In this paper, we studied a JADQN framework to resolve the caching problem that includes choosing the cache location and allotting cache contents. To utilize the limited resources rationally, all the users in the network were divided into active users and inactive users according to their activeness. In addition, the inactive users could help active users cache contents with their extra cache space during the peak periods and provide the contents in the off-peak periods. In more details, we considered the physical domain and the social domain to design our caching strategy. In the physical domain, we calculated the communication delay to judge the quality of D2D links. In the social domain, we obtained users’ social relationships from CiaoDVD dataset including user similarity and user trust. BS matched active users and inactive users who were willing to provide assistance. Furthermore, inactive users got information of active user preference from the dataset and used their remaining cache space to assist them cache contents. Numerical results showed that the cache hit rate of the proposed strategy was improved distinctly compared with the classical LFU caching strategy. In future work, we are going to design an incentive mechanism based on pricing with user service willingness to inspire more users as auxiliary nodes to participate in caching.



---

*Source: 4293625-2021-09-30.xml*