# Open-Source Based Testbed for Multioperator 4G/5G Infrastructure Sharing in Virtual Environments

Authors: Ricardo Marco Alaez; Jose M. Alcaraz Calero; Qi Wang; Fatna Belqasmi; May El Barachi; Mohamad Badra; Omar Alfandi
Journal: Wireless Communications and Mobile Computing
 (2017)

---

## Abstract

Fourth-Generation (4G) mobile networks are based on Long-Term Evolution (LTE) technologies and are being deployed worldwide, while research on further evolution towards the Fifth Generation (5G) has been recently initiated. 5G will be featured with advanced network infrastructure sharing capabilities among different operators. Therefore, an open-source implementation of 4G/5G networks with this capability is crucial to enable early research in this area. The main contribution of this paper is the design and implementation of such a 4G/5G open-source testbed to investigate multioperator infrastructure sharing capabilities executed in virtual architectures. The proposed design and implementation enable the virtualization and sharing of some of the components of the LTE architecture. A testbed has been implemented and validated with intensive empirical experiments conducted to validate the suitability of virtualizing LTE components in virtual infrastructures (i.e., infrastructures with multitenancy sharing capabilities). The impact of the proposed technologies can lead to significant saving of both capital and operational costs for mobile telecommunication operators.


---

## Body

## 1. Introduction

Mobile telecommunication operators (telcos), with a huge number of subscribers distributed in large-scale geographic areas, require the physical deployment of proprietary and expensive hardware to provide radio network coverage. They also aim to improve bandwidth and quality of experience of their subscribers to gain competitive advantage in the telecommunication market.Along the last years, telcos have been using different kinds of Radio Access Networks (RANs). GSM (2G) [1] was developed to carry on real-time services with very low data rates. To reach higher data rates, a new access technology was developed, namely, GERAN/UTRAN (3G/UMTS) [2], where packet-switching technologies were introduced. The next standardization step forward has been E-UTRAN (4G/LTE) [3] end-to-end all-IP networks. The LTE RAN is mainly composed of Enhanced Node Bs (eNBs). An eNB can be logically and physically split into two different architectural components: one or more Remote Radio Heads (RRHs) and a Base Band Unit (BBU). This separation helps in reducing capital and operational costs and is known as Cloud-RAN (C-RAN) deployment.In every step of the evolution of mobile networks, a significant investment has been required by telcos in order to deploy the new generation of mobile networks. This is where technologies for sharing physical infrastructures between different telcos can play a critical role, increasing competitiveness and fostering telco alliances.Nowadays, LTE-Advanced (LTE-A) is the fastest and latest mobile network technology commercially available. LTE components, standards, and specifications have been defined in the Third-Generation Partnership Project (3GPP) Releases 8–12 [4]. In LTE-A, radio interfaces have been completely redesigned focusing on boosting downlink and uplink capabilities in terms of bandwidth, flexible spectrum allocation, and scalability and flexibility for different environments and Quality of Service (QoS) requirements. Liu and Jiang [5] lay out the vision of 5G towards the year 2020 by a variety of scenarios with improved energy and cost efficiency.In a previous work, we have already proposed and validated an architecture to enable the sharing of some of the components of the LTE architecture in physical infrastructures [6]. This paper significantly extends the previous study to validate the suitability of the proposed architecture in virtualized infrastructures where not only LTE components but also virtual resources and networks are shared. The main contributions of this paper are threefold. Firstly, we design and develop a testbed for emulating LTE architectures with sharing capabilities suitable for virtual infrastructures. Secondly, we provide empirical results on the performance of virtualized LTE MOCN shared infrastructures among different telcos. Thirdly, we validate empirically the suitability of Network Simulator 3 (NS-3) [7] to run in virtual infrastructures in the emulation mode, interacting with commercial off-the-shelf Network Interface Cards (NICs).The rest of this paper is organized as follows. Section2 provides background information on the LTE architecture, the state of the art in LTE sharing infrastructures, and our previously proposed LTE sharing architecture. Section 3 describes the design and implementation aspects of the proposed architecture. Section 4 presents and analyzes the validation results. Section 5 concludes the paper and pinpoints some future work.


## 2. Background and Related Work

### 2.1. LTE and C-RAN Architecture

Figure1 shows an overview of the LTE architecture and its basic components. The RRH is in charge of transforming the signals between the air interface and the optical fiber. The air interface provides User(s) Equipment (UEs) with connectivity to both data and control planes. The BBU is in charge of the management of the different RRHs in order to provide interference management, air spectrum allocation. The Serving Gateway (SGW) and the Mobility Management Entity (MME) are in charge of data and control planes of the mobile networks, respectively, in order to provide UEs with access to the network. The Home Subscriber Server (HSS) is a database to store user information and network information, such as the International Mobile Subscriber Identity (IMSI) or mobile telephone number. The Policy and Charging Rules Function (PCRF) enables telcos to manage bandwidth allocation to users, for example, limiting the data rate for a UE in roaming. The Packet Data Network Gateway (PGW) is in charge of allowing roaming when the users are abroad and of giving an IP address to grant Internet access. Firmin [8] provided a comprehensive description on this topic.Figure 1
Overview of the LTE architecture.After an exhaustive investigation, it has been identified that there is no open-source software for LTE architectures supporting any type of capability for sharing architectures released. The closest available options are NS-3 and OpenAirInterface [9]. In fact, this research work is reporting a one-year implementation effort, and by the time this research was carried out, OpenAirInterface was not even yet published, so NS-3 was the only choice. C-RAN deployments are based on the usage of a centralized pool of BBUs serving a number of RRHs geographically distributed. Both architectural elements are connected using the Common Public Radio Interface (CPRI) [10]. The centralized base band processing enables very tight coordination between cells to maximize radio capacity. The main advantage is to be able to manage this coordination in a centralized way rather than doing so via the external X2 [11] interfaces between base stations. As a consequence, quicker handovers can be achieved due to the less signaling messaging exchanged, which in turn gives the system more robustness against interference. C-RAN makes the antenna site installation easier and provides footprint reduction. It leads to shorter installation times and lowers renting cost, thus saving both operational and capital costs. A C-RAN architecture is shown in Figure 2 where two telcos are deployed, each one with its own C-RAN deployment.Figure 2
Overview of the C-RAN LTE architecture.Figure2 shows the protocols used in 4G/5G networks:(i)
X2 manages handovers when the users are switching to a different BBU.(ii)
CPRI carries the signaling between the RRH and the BBU.(iii)
S1-U is in charge of the implementation of the data path between the BBU and the SGW.(iv)
Gx is in charge of QoS policies.(v)
S1-AP is the protocol for the control of users in the network between the BBU and the MME.(vi)
S6a is used for authentication and authorization of users.In order to exploit this novel C-RAN concept, a mechanism for sharing resources is critical. It will enable RRHs to share BBUs and telcos to share infrastructures to improve the efficiency in the usage of resources of the LTE architecture.


### 2.2. Existing LTE Sharing Alternatives

Two main options are available to perform the sharing of LTE resources: passive and active sharing [12]. In passive sharing, active coordination between telcos is not needed. Therefore, telcos equipment to share will be masts, power supplies, cabinets, towers, and so forth. Active sharing complements passive sharing with the sharing of active elements on the network such as BBUs. The radio spectrum can or cannot be shared depending on the kind of agreement decided between telcos.Infrastructure sharing in LTE can be achieved by different technologies. Multioperator Radio Access Network (MORAN) [13], Multioperator Core Network (MOCN) [13], and Gateway Core Network (GWCN) [14] are the main options. MORAN allows two or more telcos to share passive elements and may also share spectrum carrier and license, among other spectrum considerations. MOCN architectures typically imply the sharing of C-RAN. GWCN is another approach for sharing components of the LTE architecture where both C-RAN and MME components are shared between telcos. Figure 3 shows the different shared elements for the three approaches mentioned.Figure 3
Overview of different LTE sharing architectures.There are several studies related to LTE architecture and C-RAN. CloudIQ [15] works with OpenAirInterface focusing on the management of the backhaul LTE for C-RAN. It manages the sharing of homogeneous computations and performs the scheduling of resources from a set of base stations to show the advantages of using a resource pool. Bo et al. [16] carried out a research on the impact of the existing positioning technology in LTE and proposed two approaches based on the reference signal time difference (RSTD) in the Observed Time Difference of Arrival (OTDOA) together with UE receiving time subtracting transmitting time (UE Rx-Tx). Liu et al. [17] proposed locating users based on C-RAN using Time Difference of Arrival (TDOA) of Sounding Reference Signals (SRSs) from different antennas. He et al. [18] worked on the CPRI protocol to achieve a low-latency compression scheme of LTE downlink signal based on a clustering algorithm to reduce the required CPRI transmission bandwidth in C-RAN. Guo et al. [19] also focused on a low-latency base band signal compression algorithm to reduce bandwidth requirements in CPRI. However, neither network simulation/emulation nor software has been used to validate the claimed results, limiting the contributions only to theoretical approximations. Guangjie et al. [20] proposed a C-RAN BBU pool architecture with five key components: front-end RRH, BBU processing pool with cluster structure, control unit, raw data switch network, and traffic data switch network. Challenges such as computation resources and switch bandwidth can be utilized as needed, maximizing the usage of resources. However, again there is not any type of empirical validation based on software prototypes of the proposed work. It is very important in terms of QoS to have a well-balanced network where traffic flows with minimal congestion; hence, a load balance optimization approach has been provided by Jia et al. [21] using network flows in combination with the proposed novel algorithm. However, again the main drawback is that there is no empirical validation of the results since all the studies have been carried out based on theoretical formulas. There are some contributions focused on virtualized environments. For example, Chen et al. [22] investigated virtual resource allocation in small cell networks, developing an algorithm to face signaling overhead, outdated dynamics information, and scalability issues efficiently. He et al. [23] investigated the usage of Licensed Shared Access (LSA) on the downlink cell edge along with the Fractional Frequency Reuse scheme for resource allocation and analyzed the average capacity gain for bandwidth ratio. Tata and Kadoch [24] investigated the Shared Commercial Radio with offloading, for Public Safety Network (PSN) over LTE Heterogeneous Networks (HetNets). They proposed a solution where the PSN users have access to the commercial radio network resources. Despite the existing published work in this field, it is noted that only a scarce number of publications contain any kind of empirical validation with prototypes and go beyond traditional theoretical hypothesis. In fact, almost no validated results have been found even with more than 100 different references in the area analyzed. This is where this contribution provides a key differentiating point with respect to other work, by delivering an LTE architecture with MOCN capabilities suitable for virtual infrastructures and empirically validated by a prototype.


## 2.1. LTE and C-RAN Architecture

Figure1 shows an overview of the LTE architecture and its basic components. The RRH is in charge of transforming the signals between the air interface and the optical fiber. The air interface provides User(s) Equipment (UEs) with connectivity to both data and control planes. The BBU is in charge of the management of the different RRHs in order to provide interference management, air spectrum allocation. The Serving Gateway (SGW) and the Mobility Management Entity (MME) are in charge of data and control planes of the mobile networks, respectively, in order to provide UEs with access to the network. The Home Subscriber Server (HSS) is a database to store user information and network information, such as the International Mobile Subscriber Identity (IMSI) or mobile telephone number. The Policy and Charging Rules Function (PCRF) enables telcos to manage bandwidth allocation to users, for example, limiting the data rate for a UE in roaming. The Packet Data Network Gateway (PGW) is in charge of allowing roaming when the users are abroad and of giving an IP address to grant Internet access. Firmin [8] provided a comprehensive description on this topic.Figure 1
Overview of the LTE architecture.After an exhaustive investigation, it has been identified that there is no open-source software for LTE architectures supporting any type of capability for sharing architectures released. The closest available options are NS-3 and OpenAirInterface [9]. In fact, this research work is reporting a one-year implementation effort, and by the time this research was carried out, OpenAirInterface was not even yet published, so NS-3 was the only choice. C-RAN deployments are based on the usage of a centralized pool of BBUs serving a number of RRHs geographically distributed. Both architectural elements are connected using the Common Public Radio Interface (CPRI) [10]. The centralized base band processing enables very tight coordination between cells to maximize radio capacity. The main advantage is to be able to manage this coordination in a centralized way rather than doing so via the external X2 [11] interfaces between base stations. As a consequence, quicker handovers can be achieved due to the less signaling messaging exchanged, which in turn gives the system more robustness against interference. C-RAN makes the antenna site installation easier and provides footprint reduction. It leads to shorter installation times and lowers renting cost, thus saving both operational and capital costs. A C-RAN architecture is shown in Figure 2 where two telcos are deployed, each one with its own C-RAN deployment.Figure 2
Overview of the C-RAN LTE architecture.Figure2 shows the protocols used in 4G/5G networks:(i)
X2 manages handovers when the users are switching to a different BBU.(ii)
CPRI carries the signaling between the RRH and the BBU.(iii)
S1-U is in charge of the implementation of the data path between the BBU and the SGW.(iv)
Gx is in charge of QoS policies.(v)
S1-AP is the protocol for the control of users in the network between the BBU and the MME.(vi)
S6a is used for authentication and authorization of users.In order to exploit this novel C-RAN concept, a mechanism for sharing resources is critical. It will enable RRHs to share BBUs and telcos to share infrastructures to improve the efficiency in the usage of resources of the LTE architecture.


## 2.2. Existing LTE Sharing Alternatives

Two main options are available to perform the sharing of LTE resources: passive and active sharing [12]. In passive sharing, active coordination between telcos is not needed. Therefore, telcos equipment to share will be masts, power supplies, cabinets, towers, and so forth. Active sharing complements passive sharing with the sharing of active elements on the network such as BBUs. The radio spectrum can or cannot be shared depending on the kind of agreement decided between telcos.Infrastructure sharing in LTE can be achieved by different technologies. Multioperator Radio Access Network (MORAN) [13], Multioperator Core Network (MOCN) [13], and Gateway Core Network (GWCN) [14] are the main options. MORAN allows two or more telcos to share passive elements and may also share spectrum carrier and license, among other spectrum considerations. MOCN architectures typically imply the sharing of C-RAN. GWCN is another approach for sharing components of the LTE architecture where both C-RAN and MME components are shared between telcos. Figure 3 shows the different shared elements for the three approaches mentioned.Figure 3
Overview of different LTE sharing architectures.There are several studies related to LTE architecture and C-RAN. CloudIQ [15] works with OpenAirInterface focusing on the management of the backhaul LTE for C-RAN. It manages the sharing of homogeneous computations and performs the scheduling of resources from a set of base stations to show the advantages of using a resource pool. Bo et al. [16] carried out a research on the impact of the existing positioning technology in LTE and proposed two approaches based on the reference signal time difference (RSTD) in the Observed Time Difference of Arrival (OTDOA) together with UE receiving time subtracting transmitting time (UE Rx-Tx). Liu et al. [17] proposed locating users based on C-RAN using Time Difference of Arrival (TDOA) of Sounding Reference Signals (SRSs) from different antennas. He et al. [18] worked on the CPRI protocol to achieve a low-latency compression scheme of LTE downlink signal based on a clustering algorithm to reduce the required CPRI transmission bandwidth in C-RAN. Guo et al. [19] also focused on a low-latency base band signal compression algorithm to reduce bandwidth requirements in CPRI. However, neither network simulation/emulation nor software has been used to validate the claimed results, limiting the contributions only to theoretical approximations. Guangjie et al. [20] proposed a C-RAN BBU pool architecture with five key components: front-end RRH, BBU processing pool with cluster structure, control unit, raw data switch network, and traffic data switch network. Challenges such as computation resources and switch bandwidth can be utilized as needed, maximizing the usage of resources. However, again there is not any type of empirical validation based on software prototypes of the proposed work. It is very important in terms of QoS to have a well-balanced network where traffic flows with minimal congestion; hence, a load balance optimization approach has been provided by Jia et al. [21] using network flows in combination with the proposed novel algorithm. However, again the main drawback is that there is no empirical validation of the results since all the studies have been carried out based on theoretical formulas. There are some contributions focused on virtualized environments. For example, Chen et al. [22] investigated virtual resource allocation in small cell networks, developing an algorithm to face signaling overhead, outdated dynamics information, and scalability issues efficiently. He et al. [23] investigated the usage of Licensed Shared Access (LSA) on the downlink cell edge along with the Fractional Frequency Reuse scheme for resource allocation and analyzed the average capacity gain for bandwidth ratio. Tata and Kadoch [24] investigated the Shared Commercial Radio with offloading, for Public Safety Network (PSN) over LTE Heterogeneous Networks (HetNets). They proposed a solution where the PSN users have access to the commercial radio network resources. Despite the existing published work in this field, it is noted that only a scarce number of publications contain any kind of empirical validation with prototypes and go beyond traditional theoretical hypothesis. In fact, almost no validated results have been found even with more than 100 different references in the area analyzed. This is where this contribution provides a key differentiating point with respect to other work, by delivering an LTE architecture with MOCN capabilities suitable for virtual infrastructures and empirically validated by a prototype.


## 3. Design and Implementation

The design and implementation of the architecture proposed enable MOCN LTE sharing capabilities over both physical and virtual infrastructures. The basic prototype of the components available in the LTE infrastructure has been reused from the LENA project of the NS-3. Mainly, both SGW and BBU components have been significantly extended to allow support for MOCN LTE sharing capabilities. Among others, a new admission control algorithm has been designed and implemented; a multiconnectivity interface to allow multiple operators assigned to the same SGW and a novel LTE-based routing protocol has been designed and implemented to determine which data path needs to be selected according to the telco the UEs belong to. These extensions are described in Sections3.2 and 3.3.


### 3.1. Design Principles

Figure4 shows the proposed LTE MOCN sharing architecture, illustrated with the deployment of two different telcos. In the figure, telcos are presented with their respective complete network elements. It can be concluded that there are protocols that have to be enhanced in order to provide the MOCN LTE sharing capability. In fact, some extensions to the standard LTE architecture have been proposed, mainly in both BBU control and user planes.Figure 4
The proposed LTE MOCN sharing architecture.


### 3.2. BBU Control Plane Extension for Sharing Capabilities

In the control plane, the S1-AP interface has been extended to provide BBUs with an MME selection protocol to determine where the MME is associated with the UE that plans to be attached to a BBU. To achieve this capability, when a new UE starts the communication with the BBU, a series of messages are exchanged between them, that is,attach request/response, authentication request/response, Evolved Packet System (EPS) Session Manager (ESM) request/response, and so forth. When a UE sends an attach request, the response message (i.e., theattach accept) is sent by the MME through the BBU to the UE. The BBU then intercepts this message and obtains the IMSI. By using the IMSI, the BBU is able to extract the Public Land Mobile Network (PLMN) ID, which is an ID composed, among others, of two fields: Mobile Country Code (MCC) and Mobile Network Code (MNC). MCC and MNC are used to uniquely identify a telco. Then, this information is used to decide to which MME a UE should be attached. A table structure is maintained in memory where the PLMN is associated with the IP address of the MME associated with this PLMN. In LTE MOCN, BBUs are shared by two or more telcos. This can lead to a security concern due to the fact that a UE belonging to Telco A is connected to the same (shared) BBU of Telco B and this UE could try to access the other telco that it does not belong to (i.e., Telco B). Therefore, an extended admission control mechanism has been designed and implemented in order to make sure that only authorized UEs are allowed to be attached to the BBU they belong to.


### 3.3. BBU Data Plane Extension for Sharing Capabilities

In the user plane, the S1-U interface has been extended with a new switching protocol that helps the BBU to select the SGW used to deal with both uplink and downlink communications between UE and Internet (external servers). Every shared BBU has to deal with several S1-U connections. To manage the different connections, smart handling of the Tunnel Endpoint Identifier (TEID) has been used. This TEID is available in the GPRS Tunneling Protocol (GTP) established between BBU and SGW. TEID is a unique ID, which allows identifying a tunnel for a given UE. To handle many connections to different telcos, BBU has to be equipped with a switching mechanism. We hence propose to embed both MCC and MNC inside of the TEID field. This modification allows using this information in the data path to perform MCC/MNC-based switching of the data to the right S1-U connection.


### 3.4. Implementation

To achieve compatibility with standards, several changes have been made over the data plane and control plane (S1-U and S1-AP) in the LENA software [25], an existing LTE-emulator tool that has already been integrated and distributed in NS-3. LENA has sufficient accuracy in the implementation of the LTE stack to run in the emulation mode. Our main contribution has been focused on extending LENA to provide MOCN capabilities and to validate it in virtual infrastructures. All the innovations proposed in this contribution have been fully implemented and validated. The LENA source code has been modified to create a release with MOCN capabilities in virtualized environments. The source code is available at https://sourceforge.net/p/aslte/ in order to enable the reader to reproduce all the experiments available herein. We have achieved keeping the level of accuracy enough to enable the emulation mode in our prototype (i.e., interacting with real LTE devices). The code has been significantly extended to provide all the functionalities claimed in this paper. It is worth mentioning that there is a helper class in the LENA code(point-to-point-epc-helper) that has been redesigned to do all the setting up to achieve the deployment of the communication between different machines and different telcos within the shared virtualized infrastructure. This is the entry point for the users of our LTE architecture with the MOCN capabilities.Figure5 shows an excerpt of the relevant NS-3 code, which has been extended to include such MME selection capabilities.Figure 5
Bridging configuration in NS-3.The different extensions made through the code to achieve the MOCN implementation can be summarized as follows:(i)
S1-AP in control plane has been extended to allow the BBU to deal with more than one MME connection.(ii)
S1-U in data plane has been extended to perform switching to the SGW that the UE belongs to.(iii)
Different LENA classes have been modified, and the most important one is not onlypoint-to-point-epc-helper but also every other class that has a dependency on this former one.


## 3.1. Design Principles

Figure4 shows the proposed LTE MOCN sharing architecture, illustrated with the deployment of two different telcos. In the figure, telcos are presented with their respective complete network elements. It can be concluded that there are protocols that have to be enhanced in order to provide the MOCN LTE sharing capability. In fact, some extensions to the standard LTE architecture have been proposed, mainly in both BBU control and user planes.Figure 4
The proposed LTE MOCN sharing architecture.


## 3.2. BBU Control Plane Extension for Sharing Capabilities

In the control plane, the S1-AP interface has been extended to provide BBUs with an MME selection protocol to determine where the MME is associated with the UE that plans to be attached to a BBU. To achieve this capability, when a new UE starts the communication with the BBU, a series of messages are exchanged between them, that is,attach request/response, authentication request/response, Evolved Packet System (EPS) Session Manager (ESM) request/response, and so forth. When a UE sends an attach request, the response message (i.e., theattach accept) is sent by the MME through the BBU to the UE. The BBU then intercepts this message and obtains the IMSI. By using the IMSI, the BBU is able to extract the Public Land Mobile Network (PLMN) ID, which is an ID composed, among others, of two fields: Mobile Country Code (MCC) and Mobile Network Code (MNC). MCC and MNC are used to uniquely identify a telco. Then, this information is used to decide to which MME a UE should be attached. A table structure is maintained in memory where the PLMN is associated with the IP address of the MME associated with this PLMN. In LTE MOCN, BBUs are shared by two or more telcos. This can lead to a security concern due to the fact that a UE belonging to Telco A is connected to the same (shared) BBU of Telco B and this UE could try to access the other telco that it does not belong to (i.e., Telco B). Therefore, an extended admission control mechanism has been designed and implemented in order to make sure that only authorized UEs are allowed to be attached to the BBU they belong to.


## 3.3. BBU Data Plane Extension for Sharing Capabilities

In the user plane, the S1-U interface has been extended with a new switching protocol that helps the BBU to select the SGW used to deal with both uplink and downlink communications between UE and Internet (external servers). Every shared BBU has to deal with several S1-U connections. To manage the different connections, smart handling of the Tunnel Endpoint Identifier (TEID) has been used. This TEID is available in the GPRS Tunneling Protocol (GTP) established between BBU and SGW. TEID is a unique ID, which allows identifying a tunnel for a given UE. To handle many connections to different telcos, BBU has to be equipped with a switching mechanism. We hence propose to embed both MCC and MNC inside of the TEID field. This modification allows using this information in the data path to perform MCC/MNC-based switching of the data to the right S1-U connection.


## 3.4. Implementation

To achieve compatibility with standards, several changes have been made over the data plane and control plane (S1-U and S1-AP) in the LENA software [25], an existing LTE-emulator tool that has already been integrated and distributed in NS-3. LENA has sufficient accuracy in the implementation of the LTE stack to run in the emulation mode. Our main contribution has been focused on extending LENA to provide MOCN capabilities and to validate it in virtual infrastructures. All the innovations proposed in this contribution have been fully implemented and validated. The LENA source code has been modified to create a release with MOCN capabilities in virtualized environments. The source code is available at https://sourceforge.net/p/aslte/ in order to enable the reader to reproduce all the experiments available herein. We have achieved keeping the level of accuracy enough to enable the emulation mode in our prototype (i.e., interacting with real LTE devices). The code has been significantly extended to provide all the functionalities claimed in this paper. It is worth mentioning that there is a helper class in the LENA code(point-to-point-epc-helper) that has been redesigned to do all the setting up to achieve the deployment of the communication between different machines and different telcos within the shared virtualized infrastructure. This is the entry point for the users of our LTE architecture with the MOCN capabilities.Figure5 shows an excerpt of the relevant NS-3 code, which has been extended to include such MME selection capabilities.Figure 5
Bridging configuration in NS-3.The different extensions made through the code to achieve the MOCN implementation can be summarized as follows:(i)
S1-AP in control plane has been extended to allow the BBU to deal with more than one MME connection.(ii)
S1-U in data plane has been extended to perform switching to the SGW that the UE belongs to.(iii)
Different LENA classes have been modified, and the most important one is not onlypoint-to-point-epc-helper but also every other class that has a dependency on this former one.


## 4. Empirical Validation

This section describes the empirical analysis carried out over the proposed and prototype LTE MOCN sharing architecture to analyze the impact of virtualization on the performance of LTE MOCN. The idea is to analyze the suitability of this architecture for virtual infrastructures by providing a complete set of empirical results based on both physical and virtual infrastructures in order to understand how the novel data center workload affects the sharing capabilities of LTE architectures. The testbed description is presented in Section4.1, followed by Section 4.2 where the scenarios executed are described. The metrics used for the analysis of the performance are discussed in Section 4.3, and then the evaluation results are presented and analyzed in Section 4.4.


### 4.1. Testbed Description

Figure6 shows the setup of the testbed used to carry out the empirical analysis. Two SuperMicro FT4-E2616 servers, with 1x Xeon E5-2630V2, 6 cores, hyperthreading activated (12 cores), 32 GB RAM PC3-12800, and 3 TB SATA III with Fedora 21, are connected using Gigabit Ethernet interfaces and a Gigabit switch. Each computer is running a Kernel-based Virtual Machine (KVM [27]), a hypervisor that enables the creation of virtual machines (VMs). A VM is created in each physical computer, with the same operating and software configuration deployed in the physical host. Linux bridges have been created and configured to interconnect the VMs and physical machines to allow connectivity between all of them and also to the Internet. Computer A (and its VM A associated) is used to emulate the entire MOCN LTE architecture. It executes NS-3 release 3.22 with our extended version of LENA. Computer B (and its VM B? associated) is used to emulate a remote host on the Internet. It is running a custom-made UDP daemon that accepts packets from the UEs and collects statistics (e.g., it obtains times and sizes of the packets received) to enable performance analysis. UEs are in Computer A as part of the LTE architecture and are communicating with the remote host in Computer B. There is one remote host per SGW (i.e., per telco). The usage of both physical and virtual machines with the same software deployment enables the analysis of four different testing environments as indicated in Table 1: (a) a completely physical deployment environment where the LTE infrastructure is allocated in the physical Computer A, which is connected to a remote host running in the physical Computer B, being used as a baseline for the analysis of how virtualization affects performance; ((b) and (c)) a hybrid deployment environment, where either the LTE infrastructure or the remote host is running on a VM and the other one is deployed on a physical computer, respectively; (d) a fully virtualized deployment environment where both the LTE infrastructure and the remote host are running on VMs. The variables used in the scenarios created in the LENA NS-3 emulation environment are explained in Table 2.Table 1
Setup meaning MOCN architecture.
Telco ID
Remote host ID
Telco computer type
Remote host computer type
A, B
A, B
Physical
Physical
A, B
A, B
Physical
Virtual
A, B
A, B
Virtual
Physical
A, B
A, B
Virtual
VirtualTable 2
Fixed parameters used in NS-3.
Options
Value
Functionality
Simulation time
2 sec
Duration of the simulation (i.e., simulation time).
Distance between UEs
1 meter
All UEs are geographically located in the same vertical plane. The distance between each UE is defined by this parameter. All eNBs are located 1 meter far in the vertical plane and all of them are equally spaced between themselves and also between UEs.
Interval leaving packet (ITL)
1, 10, and 100 ms
Every 1, 10, or 100 ms, a packet is sent from the UE to the UDP server located in the other virtual/physical computer.
Start time
0.1 sec
Traffic starts once the emulation is working. This is the time used for stabilizing the emulation (bootstrapping).
Increment time
0.01 sec
Gap between the starting times of each of the UEs.
SRS periodicity
320
Maximum number of UEs supported by antenna/eNB.
Scheduler
PssFfMacScheduler
LTE MAC scheduler (see LENA documentation [26]).
S1-U link rate
10 Gb/s
Speed for S1 links.
S1 link delay
0.01 sec
Delay for S1 links in seconds.
S1 link MTU
1500 bytes
Maximum transfer unit for S1 links in bytes.
UE Tx power
20 dB
UE Tx power in dB.
eNB Tx power
50 dB
eNB TX power in dB.
Download link bandwidth
100 MBs
Maximum data rate for download in LTE interface.
Upload link bandwidth
100 MBs
Maximum data rate for upload in LTE interface.Figure 6
Testbed setup for the MOCN architecture.


### 4.2. Experiments

A significant number of scenarios have been executed for each of the three deployment cases earlier described, that is, physical, hybrid, and virtualized deployment environments. Each scenario is executed by ranging the following parameters in order to create different experiments: workload conditions, stress levels, and so forth. In total, 192 different scenarios have been executed for each of the deployment environments analyzed.(i)
Number of UEs deployed in the scenario:[1,2,4,8,16,32,64,128,256].(ii)
Interleaving packet rate:[1,10,100] ms; the sending packet rate of each UE (all the UEs have the same rate); the values represent three saturation levels, corresponding directly to [2000,200,20] packets sent in 2 secs.(iii)
Number of eNBs[1,2,4,8,16,32]: number of antennas available in the scenario; there is always a balanced distribution of UE connected to antennas (e.g., eNB = 32, UE = 256), meaning that 8 UEs connect to each of the available antennas.(iv)
Number of telecommunication operators sharing the infrastructure[1,2,4,8,16,32].As an example, a scenario defined as UEs = 256, eNBs = 32, telcos = 2, and sending packet = 1 ms means that the scenario includes 256 UEs and 32 antennas. Each antenna has 8 UEs connected, which are those closer to the antenna. The UEs are stationary. There are two telcos (i.e., 2 MME, 2 SGW, and 2 PGW). Each antenna is connected by default to all the operators of the scenario. Moreover, the assignment of each of the UEs to a given telco is completely balanced. In the example, 128 UEs will belong to Telco 1 and the other 128 will belong to Telco 2. Each antenna will have four UEs from Telco 1 and four UEs from Telco 2. The workload is generated in both ways; that is, there is a UDP flow from UE to the remote host and another one from the remote host to UE. Each flow generates packets at a constant rate of 1 packet/ms. All the experiments were executed for two seconds. This means that, in this example, each UE will send 1000 packets/sec for two seconds. Each UDP packet has a payload of 64 Kbytes; this size is always the same for all the experiments. The idea behind the experiments is to send packets from UEs to the remote host while exponentially varying the values of the different parameters to saturate the network. The main purpose is to see how the increasing of these parameters as well as the sharing rate affects the overall performance. We have used three network saturation levels to analyze the behavior of the proposed MOCN architecture at different levels of saturation. Real-time emulation mode using TAP [28] interfaces and checksum computations has been enabled to take a higher level of accuracy for the results.


### 4.3. Evaluation Metrics

Two metrics have been used for the analysis of the architecture: average UE transmission time and average throughput. The UE transmission time is defined as the time it takes for one UE to send all the packets in the scenario to the remote host and for the packets to be received by the destination (i.e., the end-to-end delay). The average UE transmission time is the average delay calculated for all the UEs in the executed test scenario. The individual throughput is calculated at the remote host UDP daemon for each of the UEs representing the number of bits received from that UE per second. It is calculated using the following formula:(1)NumberofBytesReceived∗8/LastPacketTime-FirstPacketTime1024=XKbps. The average throughput is an averaged value of the calculated individual throughput per UE.


### 4.4. Evaluation Results

Given the large number of executed scenarios (i.e., 576), it is impossible to render all of them in this paper. In addition, the analysis revealed that many of these scenarios exhibit a similar behavior. This section will, therefore, show the most interesting scenarios to illustrate relevant findings. These are summarized in Figure7.Figure 7
(a) 10 ms interleaving scenario running in physical machines. (b) 100 ms interleaving scenario running in physical machines. (c) 10 ms interleaving scenario running in virtual machines. (d) 100 ms interleaving scenario running in virtual machines.
(a)
(b)
(c)
(d)Figure7 shows four different subfigures. The subfigures of Figure 7 ((b) and (d)) represent the averaged throughput achieved by the UE for a nonsaturated scenario (i.e., interleaving packet rate 100 ms), while subfigures (a) and (c) represent a midsaturated scenario with interleaving packet rate of 10 ms. Figure 7 has a fixed number of UEs of 64, 128, and 256 to analyze large deployments with the effect of ranging the number of BBUs and telcos sharing these BBUs. This is why there are different three-dimensional surfaces plotted in the subfigures. It is noted that Figures 7(a) and 7(b) are scenarios deployed in physical environments while Figures 7(c) and 7(d) are executions in virtual environments. This layout in Figure 7 can be used to analyze the impact of virtualization by comparing top-down figures and the impact of stressed workload by comparing the left-right figures. The Maximum Nominal Bandwidth for both nonsaturated and midsaturated scenarios is (10 or 100 packets/sec ∗ 10560 packet size (bits) ∗  x UEs) 27 Mbits/s and 270 Mbits/s, respectively. The most stressed scenario with 1 ms of interleaving packet (2.7 Gbits/s) leads to a high number of packet drops due to the fact that the hardware cannot handle this level of saturation in real time; this is why the figure has not been shown. Different axes in Figures 7(a)–7(d) represent, respectively, the number of telcos (PLMNs) sharing the BBUs deployed in the scenario, the number of BBUs deployed, and the average throughput received in the remote host. Firstly, it is noticed how Figure 7(a) (physical machine to physical machine) has a high peak up to 900 kbits/s whereas Figure 7(c) (VM to VM) has a high peak up to 600 kbits/s. Secondly, it can be observed that the most relevant aspect when comparing all the surfaces plotted in each of the figures is that the system is always stable up to ~300 kbits/s regardless of the number of telcos sharing the BBUs. At higher rates, packet loss starts to increase significantly. Therefore, it can be considered as the maximum speedy throughput the system can handle. The high peak at 900 kbits/s per user (i.e., over 57 Mbits/s) is the empirical maximum validation due to limitations of the hardware used. It is noted that 256 full LTE stacks are being emulated in real time while sending and receiving traffic simultaneously. Fontes et al. [29] were able to reach a high peak of almost 20 Mbits/s due to the same reason, although in our case we are working with a full emulation of the physical layer radio air interface whereas they were only using Optimized Link State Routing (OSLR) [30] protocols. In the nonsaturated scenario, that is, Figures 7(b) and 7(d), the results show as well how the system becomes stable at ~100 kbits/s (i.e., over 20 Mbits/s), which is the expected result for this workload with almost no packet drops.Thirdly, it is noted how the throughput is decreasing when the number of BBUs is increased until they become stable. This is due to the fact that the emulation needs to handle more virtual NICs (and more complex scenarios) and thus the performance is being decreased. The results of the overhead associated with the execution of VMs for the midsaturated scenario (10 ms interleaving packet) are shown in Table3 while the results for the nonsaturated scenario are shown in Table 4.Table 3
Throughput overhead for the saturated scenario (10 ms of interleaving packet rate).
Number of UEs (Figures7(a) and 7(c))
Scenarios analyzed
Overhead (%)
64
PHY-PHY versus VM-VM
14
128
PHY-PHY versus VM-VM
10
256
PHY-PHY versus VM-VM
12Table 4
Throughput overhead for the nonsaturated scenario (100 ms of interleaving packet rate).
Number of UEs (Figures7(b) and 7(d))
Point-to-point
Overhead (%)
64
PHY-PHY versus VM-VM
12
128
PHY-PHY versus VM-VM
16
256
PHY-PHY versus VM-VM
1The overhead in the average of the performance associated with the execution of the MOCN LTE infrastructure in VMs could be set up in almost all the 576 scenarios analyzed close to 10%. This fact validates empirically the efficiency and feasibility of the proposed MOCN architecture running in virtual infrastructures. Another interesting aspect to analyze is how packets are lost when NS-3 interacts with real NIC interfaces in comparison with NS-3 interacting with virtual NICs. Table5 shows that, in all the cases, the difference of packet losses between physical-to-physical machine communication and virtual-to-virtual machine communication is 0%, which is also a good result of the suitability of running emulated MOCN LTE in virtual infrastructures.Table 5
Overhead in terms of packet loss.
Number of eNBs
Point-to-point
Overhead (%)
1
PHY-PHY to VM-VM
0
8
PHY-PHY to VM-VM
0
16
PHY-PHY to VM-VM
0
32
PHY-PHY to VM-VM
0Finally, it has been observed how the number of operators sharing the LTE infrastructure almost does not affect the performance of the network. This is a clear insight into the validation of the scalability of the proposed extension over LENA NS-3 LTE.The analysis of Figures7(a) and 7(c), physical machine to physical machine and VM-to-VM communication, respectively, unveils the best sharing ratio of a BBU by different telcos to achieve the best performance according to our experiments. It is plotted in Line A where eight telcos sharing the same BBU provide the best performance in saturated scenarios. It is noted that surfaces represent scenarios with different numbers of UEs and the trend in the behavior of Line A is similar to all the surfaces. For midsaturated scenarios, depicted in Figures 7(b) and 7(d), 16 telcos sharing the same BBU show the best results for all the scenarios analyzed. These two values provide a clear hint on the range of an acceptable sharing factor between BBUs and telcos. The analysis of Line B unveils the best number of BBUs set up when the best sharing factor is assumed. There are eight BBUs and 16 BBUs for the saturated and midsaturated scenarios, respectively. Points P1 and P2 are the combinations showing the best throughput achieved (685.25 and 548.2 averaged kbits/s per UE). In a saturated scenario, Figures 7(a) and 7(c) show a similar shape regardless of the number of users as Lines C, D, and E show that the throughput is increasing until the system reaches eight BBUs, and from that point, the system tends to stabilize regardless of the number of telcos. In a midsaturated scenario, Figures 7(b) and 7(d) plot P3 and P4 which are the combinations showing the best throughput achieved (84.8 kbits/s). Lines C, D, and E in Figures 7(b) and 7(d) show similar behaviors for different numbers of users and how the proposed architecture tends to stabilize after 16 BBUs have been reached regardless of the number of telcos sharing the BBUs. This analysis demonstrates the benefit of sharing BBUs between different telcos and provides empirical insights into ranging values to be used to achieve good performance settings.


## 4.1. Testbed Description

Figure6 shows the setup of the testbed used to carry out the empirical analysis. Two SuperMicro FT4-E2616 servers, with 1x Xeon E5-2630V2, 6 cores, hyperthreading activated (12 cores), 32 GB RAM PC3-12800, and 3 TB SATA III with Fedora 21, are connected using Gigabit Ethernet interfaces and a Gigabit switch. Each computer is running a Kernel-based Virtual Machine (KVM [27]), a hypervisor that enables the creation of virtual machines (VMs). A VM is created in each physical computer, with the same operating and software configuration deployed in the physical host. Linux bridges have been created and configured to interconnect the VMs and physical machines to allow connectivity between all of them and also to the Internet. Computer A (and its VM A associated) is used to emulate the entire MOCN LTE architecture. It executes NS-3 release 3.22 with our extended version of LENA. Computer B (and its VM B? associated) is used to emulate a remote host on the Internet. It is running a custom-made UDP daemon that accepts packets from the UEs and collects statistics (e.g., it obtains times and sizes of the packets received) to enable performance analysis. UEs are in Computer A as part of the LTE architecture and are communicating with the remote host in Computer B. There is one remote host per SGW (i.e., per telco). The usage of both physical and virtual machines with the same software deployment enables the analysis of four different testing environments as indicated in Table 1: (a) a completely physical deployment environment where the LTE infrastructure is allocated in the physical Computer A, which is connected to a remote host running in the physical Computer B, being used as a baseline for the analysis of how virtualization affects performance; ((b) and (c)) a hybrid deployment environment, where either the LTE infrastructure or the remote host is running on a VM and the other one is deployed on a physical computer, respectively; (d) a fully virtualized deployment environment where both the LTE infrastructure and the remote host are running on VMs. The variables used in the scenarios created in the LENA NS-3 emulation environment are explained in Table 2.Table 1
Setup meaning MOCN architecture.
Telco ID
Remote host ID
Telco computer type
Remote host computer type
A, B
A, B
Physical
Physical
A, B
A, B
Physical
Virtual
A, B
A, B
Virtual
Physical
A, B
A, B
Virtual
VirtualTable 2
Fixed parameters used in NS-3.
Options
Value
Functionality
Simulation time
2 sec
Duration of the simulation (i.e., simulation time).
Distance between UEs
1 meter
All UEs are geographically located in the same vertical plane. The distance between each UE is defined by this parameter. All eNBs are located 1 meter far in the vertical plane and all of them are equally spaced between themselves and also between UEs.
Interval leaving packet (ITL)
1, 10, and 100 ms
Every 1, 10, or 100 ms, a packet is sent from the UE to the UDP server located in the other virtual/physical computer.
Start time
0.1 sec
Traffic starts once the emulation is working. This is the time used for stabilizing the emulation (bootstrapping).
Increment time
0.01 sec
Gap between the starting times of each of the UEs.
SRS periodicity
320
Maximum number of UEs supported by antenna/eNB.
Scheduler
PssFfMacScheduler
LTE MAC scheduler (see LENA documentation [26]).
S1-U link rate
10 Gb/s
Speed for S1 links.
S1 link delay
0.01 sec
Delay for S1 links in seconds.
S1 link MTU
1500 bytes
Maximum transfer unit for S1 links in bytes.
UE Tx power
20 dB
UE Tx power in dB.
eNB Tx power
50 dB
eNB TX power in dB.
Download link bandwidth
100 MBs
Maximum data rate for download in LTE interface.
Upload link bandwidth
100 MBs
Maximum data rate for upload in LTE interface.Figure 6
Testbed setup for the MOCN architecture.


## 4.2. Experiments

A significant number of scenarios have been executed for each of the three deployment cases earlier described, that is, physical, hybrid, and virtualized deployment environments. Each scenario is executed by ranging the following parameters in order to create different experiments: workload conditions, stress levels, and so forth. In total, 192 different scenarios have been executed for each of the deployment environments analyzed.(i)
Number of UEs deployed in the scenario:[1,2,4,8,16,32,64,128,256].(ii)
Interleaving packet rate:[1,10,100] ms; the sending packet rate of each UE (all the UEs have the same rate); the values represent three saturation levels, corresponding directly to [2000,200,20] packets sent in 2 secs.(iii)
Number of eNBs[1,2,4,8,16,32]: number of antennas available in the scenario; there is always a balanced distribution of UE connected to antennas (e.g., eNB = 32, UE = 256), meaning that 8 UEs connect to each of the available antennas.(iv)
Number of telecommunication operators sharing the infrastructure[1,2,4,8,16,32].As an example, a scenario defined as UEs = 256, eNBs = 32, telcos = 2, and sending packet = 1 ms means that the scenario includes 256 UEs and 32 antennas. Each antenna has 8 UEs connected, which are those closer to the antenna. The UEs are stationary. There are two telcos (i.e., 2 MME, 2 SGW, and 2 PGW). Each antenna is connected by default to all the operators of the scenario. Moreover, the assignment of each of the UEs to a given telco is completely balanced. In the example, 128 UEs will belong to Telco 1 and the other 128 will belong to Telco 2. Each antenna will have four UEs from Telco 1 and four UEs from Telco 2. The workload is generated in both ways; that is, there is a UDP flow from UE to the remote host and another one from the remote host to UE. Each flow generates packets at a constant rate of 1 packet/ms. All the experiments were executed for two seconds. This means that, in this example, each UE will send 1000 packets/sec for two seconds. Each UDP packet has a payload of 64 Kbytes; this size is always the same for all the experiments. The idea behind the experiments is to send packets from UEs to the remote host while exponentially varying the values of the different parameters to saturate the network. The main purpose is to see how the increasing of these parameters as well as the sharing rate affects the overall performance. We have used three network saturation levels to analyze the behavior of the proposed MOCN architecture at different levels of saturation. Real-time emulation mode using TAP [28] interfaces and checksum computations has been enabled to take a higher level of accuracy for the results.


## 4.3. Evaluation Metrics

Two metrics have been used for the analysis of the architecture: average UE transmission time and average throughput. The UE transmission time is defined as the time it takes for one UE to send all the packets in the scenario to the remote host and for the packets to be received by the destination (i.e., the end-to-end delay). The average UE transmission time is the average delay calculated for all the UEs in the executed test scenario. The individual throughput is calculated at the remote host UDP daemon for each of the UEs representing the number of bits received from that UE per second. It is calculated using the following formula:(1)NumberofBytesReceived∗8/LastPacketTime-FirstPacketTime1024=XKbps. The average throughput is an averaged value of the calculated individual throughput per UE.


## 4.4. Evaluation Results

Given the large number of executed scenarios (i.e., 576), it is impossible to render all of them in this paper. In addition, the analysis revealed that many of these scenarios exhibit a similar behavior. This section will, therefore, show the most interesting scenarios to illustrate relevant findings. These are summarized in Figure7.Figure 7
(a) 10 ms interleaving scenario running in physical machines. (b) 100 ms interleaving scenario running in physical machines. (c) 10 ms interleaving scenario running in virtual machines. (d) 100 ms interleaving scenario running in virtual machines.
(a)
(b)
(c)
(d)Figure7 shows four different subfigures. The subfigures of Figure 7 ((b) and (d)) represent the averaged throughput achieved by the UE for a nonsaturated scenario (i.e., interleaving packet rate 100 ms), while subfigures (a) and (c) represent a midsaturated scenario with interleaving packet rate of 10 ms. Figure 7 has a fixed number of UEs of 64, 128, and 256 to analyze large deployments with the effect of ranging the number of BBUs and telcos sharing these BBUs. This is why there are different three-dimensional surfaces plotted in the subfigures. It is noted that Figures 7(a) and 7(b) are scenarios deployed in physical environments while Figures 7(c) and 7(d) are executions in virtual environments. This layout in Figure 7 can be used to analyze the impact of virtualization by comparing top-down figures and the impact of stressed workload by comparing the left-right figures. The Maximum Nominal Bandwidth for both nonsaturated and midsaturated scenarios is (10 or 100 packets/sec ∗ 10560 packet size (bits) ∗  x UEs) 27 Mbits/s and 270 Mbits/s, respectively. The most stressed scenario with 1 ms of interleaving packet (2.7 Gbits/s) leads to a high number of packet drops due to the fact that the hardware cannot handle this level of saturation in real time; this is why the figure has not been shown. Different axes in Figures 7(a)–7(d) represent, respectively, the number of telcos (PLMNs) sharing the BBUs deployed in the scenario, the number of BBUs deployed, and the average throughput received in the remote host. Firstly, it is noticed how Figure 7(a) (physical machine to physical machine) has a high peak up to 900 kbits/s whereas Figure 7(c) (VM to VM) has a high peak up to 600 kbits/s. Secondly, it can be observed that the most relevant aspect when comparing all the surfaces plotted in each of the figures is that the system is always stable up to ~300 kbits/s regardless of the number of telcos sharing the BBUs. At higher rates, packet loss starts to increase significantly. Therefore, it can be considered as the maximum speedy throughput the system can handle. The high peak at 900 kbits/s per user (i.e., over 57 Mbits/s) is the empirical maximum validation due to limitations of the hardware used. It is noted that 256 full LTE stacks are being emulated in real time while sending and receiving traffic simultaneously. Fontes et al. [29] were able to reach a high peak of almost 20 Mbits/s due to the same reason, although in our case we are working with a full emulation of the physical layer radio air interface whereas they were only using Optimized Link State Routing (OSLR) [30] protocols. In the nonsaturated scenario, that is, Figures 7(b) and 7(d), the results show as well how the system becomes stable at ~100 kbits/s (i.e., over 20 Mbits/s), which is the expected result for this workload with almost no packet drops.Thirdly, it is noted how the throughput is decreasing when the number of BBUs is increased until they become stable. This is due to the fact that the emulation needs to handle more virtual NICs (and more complex scenarios) and thus the performance is being decreased. The results of the overhead associated with the execution of VMs for the midsaturated scenario (10 ms interleaving packet) are shown in Table3 while the results for the nonsaturated scenario are shown in Table 4.Table 3
Throughput overhead for the saturated scenario (10 ms of interleaving packet rate).
Number of UEs (Figures7(a) and 7(c))
Scenarios analyzed
Overhead (%)
64
PHY-PHY versus VM-VM
14
128
PHY-PHY versus VM-VM
10
256
PHY-PHY versus VM-VM
12Table 4
Throughput overhead for the nonsaturated scenario (100 ms of interleaving packet rate).
Number of UEs (Figures7(b) and 7(d))
Point-to-point
Overhead (%)
64
PHY-PHY versus VM-VM
12
128
PHY-PHY versus VM-VM
16
256
PHY-PHY versus VM-VM
1The overhead in the average of the performance associated with the execution of the MOCN LTE infrastructure in VMs could be set up in almost all the 576 scenarios analyzed close to 10%. This fact validates empirically the efficiency and feasibility of the proposed MOCN architecture running in virtual infrastructures. Another interesting aspect to analyze is how packets are lost when NS-3 interacts with real NIC interfaces in comparison with NS-3 interacting with virtual NICs. Table5 shows that, in all the cases, the difference of packet losses between physical-to-physical machine communication and virtual-to-virtual machine communication is 0%, which is also a good result of the suitability of running emulated MOCN LTE in virtual infrastructures.Table 5
Overhead in terms of packet loss.
Number of eNBs
Point-to-point
Overhead (%)
1
PHY-PHY to VM-VM
0
8
PHY-PHY to VM-VM
0
16
PHY-PHY to VM-VM
0
32
PHY-PHY to VM-VM
0Finally, it has been observed how the number of operators sharing the LTE infrastructure almost does not affect the performance of the network. This is a clear insight into the validation of the scalability of the proposed extension over LENA NS-3 LTE.The analysis of Figures7(a) and 7(c), physical machine to physical machine and VM-to-VM communication, respectively, unveils the best sharing ratio of a BBU by different telcos to achieve the best performance according to our experiments. It is plotted in Line A where eight telcos sharing the same BBU provide the best performance in saturated scenarios. It is noted that surfaces represent scenarios with different numbers of UEs and the trend in the behavior of Line A is similar to all the surfaces. For midsaturated scenarios, depicted in Figures 7(b) and 7(d), 16 telcos sharing the same BBU show the best results for all the scenarios analyzed. These two values provide a clear hint on the range of an acceptable sharing factor between BBUs and telcos. The analysis of Line B unveils the best number of BBUs set up when the best sharing factor is assumed. There are eight BBUs and 16 BBUs for the saturated and midsaturated scenarios, respectively. Points P1 and P2 are the combinations showing the best throughput achieved (685.25 and 548.2 averaged kbits/s per UE). In a saturated scenario, Figures 7(a) and 7(c) show a similar shape regardless of the number of users as Lines C, D, and E show that the throughput is increasing until the system reaches eight BBUs, and from that point, the system tends to stabilize regardless of the number of telcos. In a midsaturated scenario, Figures 7(b) and 7(d) plot P3 and P4 which are the combinations showing the best throughput achieved (84.8 kbits/s). Lines C, D, and E in Figures 7(b) and 7(d) show similar behaviors for different numbers of users and how the proposed architecture tends to stabilize after 16 BBUs have been reached regardless of the number of telcos sharing the BBUs. This analysis demonstrates the benefit of sharing BBUs between different telcos and provides empirical insights into ranging values to be used to achieve good performance settings.


## 5. Conclusion

This paper has provided the first open-source design and implementation of an LTE architecture with MOCN infrastructure sharing capabilities and has validated its feasibility in virtual infrastructures. The design includes several extensions in both control plane and data plane to the protocols S1-AP and S1-U, respectively. The suitability of the implemented testbed has successfully demonstrated running such software in both physical and virtual infrastructures. The architecture has been validated by an extensive set of experiments. Scalability, overhead, and network saturation have been analyzed, providing very promising results.As to future work, there are some improvements to be made related to the LENA software. For example, it would be interesting to have an implementation of CPRI in LENA to be able to perform some tests in the future. Adding support to interact with real LTE hardware (antennas, BBU, and RRH) would be also a natural step.The porting of these MOCN capabilities to the open-source software OpenAirInterface would be another future step to allow implementations with hardware air interfaces. It is also expected to design and implement other alternatives for sharing infrastructures in LTE, for example, the GWCN mode proposed by 3GPP.



---

*Source: 1984314-2017-11-05.xml*