# Multilayer Architecture Model for Mobile Cloud Computing Paradigm

Authors: Higinio Mora; Francisco J. Mora Gimeno; María Teresa Signes-Pont; Bruno Volckaert
Journal: Complexity
 (2019)

---

## Abstract

Mobile Cloud Computing is one of today's more disruptive paradigms of computation due to its effects on the performance of mobile computing and the development of Internet of Things. It is able to enhance the capabilities of devices by outsourcing the workload to external computing platforms deployed along the network, such as cloud servers, cloudlets, or other edge platforms. The research described in this work presents a computational model of a multilayer architecture for increasing the performance of devices using the Mobile Cloud Computing paradigm. The main novelty of this work lies in defining a comprehensive model where all the available computing platforms along the network layers are involved to perform the outsourcing of the application workload. This proposal provides a generalization of the Mobile Cloud Computing paradigm which allows handling the complexity of scheduling tasks in such complex scenarios. The behaviour of the model and its ability of generalization of the paradigm are exemplified through simulations. The results show higher flexibility for making offloading decisions.


---

## Body

## 1. Introduction

Cloud Computing paradigm is one of the most disruptive technology advances of our times. This paradigm has made Information Technology (IT) resources available to the general public through Internet. In this way, any business, organization or particular user can access to computing infrastructures and services for a fee. The progress of communication technologies has also contributed to this end. Boosting the bandwidth and hence speed of all connections has enabled us to handle more traffic. In addition, the improvements on management of cloud centres thanks to virtualization and server consolidation methods allow us to give a rapid response to changing application demands [1]. These cloud-based services evolutions have resulted in an increased outsourcing work to the cloud in all areas. This paradigm is now growing many times faster than the rest of IT industry [2].Implementation of this paradigm to mobile computation has led to Mobile Cloud Computing (MCC) concept [3, 4]. It consists in outsourcing part of the processing load to the Cloud and getting back the results. The implications for mobile devices are evident: in the first place, they can increase their performance without any change in their hardware, and, secondly, they allow us to extend the life of battery-powered devices. These benefits have enhanced the potential of Internet of Things (IoT) paradigm and allow us to compute advanced applications on devices and embedded systems.However, this promise of providing new resources beyond mobile computing capabilities by accessing cloud servers can lead to uncertainty and delays in response times due to unpredictability in network operations through the Internet. Further, some applications cannot expect to receive the calculated results from offloading the source. In general, there is a lack of versatility in the cloud that makes it unable to adapt to specific task requirements and to internet operation conditions. Thus, the user-perceived quality is highly variable and depends on both the application’s degree of interactivity and the network’s end-to-end latency. There is a requirement to improve the ability of the MCC paradigm to meet the performance requirements for heterogeneous devices in dynamic environments where the available infrastructure can change.From these motivation aspects exposed by these problems, themain objective of this work consists of researching architecture models that are able to increase the resilience and flexibility of offloading the workload outside the device and to enable scheduling the tasks along the network. To advance this goal, it is assumed as aworking hypothesis that several network computing layers exist around the mobile device or embedded system and are available for outsourcing the application workload.Thekey contributions of this work can be summarized as follows.(i)
A study of the complexity and key issues of distributing the processing load among several computing layers along the network and a review of the architectures and network layers to perform this offloading process.(ii)
The proposal of a general framework for a multilayer architecture to formalize the processing of an application and the implications of the distributed configurations to the performance and the communications requirements.Thenovelty of this proposal is the extension of the MCC paradigm to the available computing layers by considering them in a comprehensive and integrated fashion with the distributed architecture.This paper is organized as follows: in Section2, we review most relevant models and frameworks of MCC paradigm; in Section 3 the formal framework of the computational model is introduced, and the key technical characteristics of offloading are described; Section 4 explains how the proposed multilayer computational model can handle real-world scenarios; and, finally, conclusions are drawn and some approaches for future work are also pointed out in Section 5.


## 2. Related Work

### 2.1. Outsourcing Options for MCC

The concept of MCC has been evolved in recent times with the goal of increasing the performance by offloading the workload outside the mobile devices. As a result, the outsourcing options have been increased along the communication network as shown in Figure1.Figure 1
Outsourcing options deployed along the communication network.These outsourcing options are deployed at different layers of the network forming a pool of computing platforms that are available depending on the application context.In the top, at the network core, thecloud layer remains the most powerful computing platform. This is the traditional way for outsourcing the workload and takes its origin from the Cloud Computing Paradigm. Much of the current research on MCC is using this option as a default for implementing the offloading process [5–7].Next, thecloudlet layer concept emerges. This infrastructure is a “little” data center with the objective of bringing the cloud closer to the users [8]. Usually, cloudlets are deployed inside the organizations that use it [9, 10]. The main advantages provided by cloudlets are twofold: in the first place, to reduce service disruptions and delays caused by the network to remote servers and secondly to ensure a high level of security of the data since it does not need to be sent through Internet [11]. Therefore, cloudlets are computing platforms similar to cloud servers but prepared and scaled according to the specific requirements of the organization.The same principle of cloudlets is applied to theMobile Edge Computing (MEC) Layer. A MEC server is a data center in a box, that, in this case, it is deployed by the mobile telecom operators in close proximity to mobile subscribers [12]. Thus, access to these computing services is made through telecommunication base stations using communication protocols such as 3G or LTE to offer a service environment with high bandwidth and low latency [13, 14]. In addition, the benefits of MEC paradigm will be enhanced by the advent of 5G communication protocol [15].These last two layers (cloudlet and MEC) can be enabled in a dynamic way to deliver computing support to the cloud servers. In this way, they are not replacing but complementing the Cloud Computing Paradigm by providing a flexible computing power when necessary [16]. This operation encourages the development of IoT solutions and enhances the mobile applications when devices are moving over time.Finally, a recent trend to develop the model of MCC is the Fog Computing Paradigm. Thefog layer is a set of devices such as routers, switches, or other networking devices closer or within the end users' local networks. Their aim is to provide processing efforts as close as possible where the data is generated [17]. Thus, these devices have some computing capabilities able to compute part of the workload of the “things” and other application devices [18, 19]. As a result, service delay is significantly reduced for end user applications.


### 2.2. MCC Frameworks

Realizing the vision of multilayer offloading architecture for implementing the MCC paradigm is a challenging task because of the complexity in handling the multiple aspects involved, especially those concerned with performance evaluation and tasks scheduling. In this issue, the motivation for offloading is diverse depending on the user requirements, device configuration or application constraints. Moreover, performance aspect can be of different nature such as power consumption, time delay, money cost of using external services, and network usage.There is a need to design suitable frameworks to formalize the performance components in a homogeneous way to make decisions on when and where to outsource the processing load. In this subsection, a review of recent frameworks of MCC paradigm is described in order to reach the knowledge border in this issue.There are several existing frameworks and architectures in the literature for outsourcing the tasks of the application workload to other computing platforms. The areas of application of these techniques cover many sectors and disciplines. In general, they develop the IoT and Cyber-Physical Systems (CPS) paradigms that are experiencing a great growth in recent years. The devices involved are a heterogeneous set which mainly consists of sensors, actuators, and embedded systems. In addition, the mobile phones and other mobile devices such as wearables are also in this kind of applications.As overview of the MCC proposals, they aim to carry out a collaborative work for distributing the processing load and meeting the application requirements [20, 21]. In most cases, these applications have special features and are high demanding of computing resources such as multimedia analysis [22], high volume of data [23], or real-time constraint [4]. By distributing the work, the cloud architecture bottleneck is overcome, and better delay and predictability can be achieved [24, 25].In general terms, offloading from the “things” or mobile devices to cloud, edge, or cloudlet server always produced significant increase in performance. It is clear that the slower the devices are, the sharpest these differences become. However, the offloading criteria can consider other aspects such as network usage or money cost of external computing services [26]. In these cases, an indirect relation exists between using outsourcing services and benefits. Thus, the benefit function needs to be redefined to consider a heterogeneous type of aspects in order to make decisions on where and when to outsource the application workload.The key parts of a framework for handling the offloading process are the architecture that defines the available outsourcing platforms, the decision method on where and when to offload, and the communication model that defines how to perform this process.Regarding the decision method for offloading, in general term, it is considered as a scheduler which decides when and where to offload taking into account the application constraints, the tasks’ features, and the performance aspects of the available computing platforms. Its main aim is how to reasonably allocate the tasks to available computing platforms to minimize the total cost and load. The optimal scheduling scheme belongs to NP-complete problem set and, therefore, traditional strategies cannot be applied in a suitable way for MCC applications.To address this issue, many approaches have been proposed based on different mathematical techniques and algorithms. In recent approaches, the scheduling method can be formulated as a constrained optimization problem where a suboptimal solution is usually the best choice. These methods can be used for multitask [27] and multiobjective [28] scheduling problems in several scenarios. In the same line, there are methods based on stochastic techniques and search algorithms to deal with the high complexity of this problem [29, 30]. Other proposals are based on dynamic programming methods in order to handle the runtime variations of tasks execution [31]. Also, a decision tree can be built to classify the conditions and decision parameters in order to get a fast offloading response [32]. Finally, other works propose innovative approaches to address this problem based on techniques used for processor scheduling area such as genetic algorithms [33], and imprecise computation strategies [4].In most cases, the offloading decisions are mainly focused on energy optimization [31] since it is a critical issue for mobile devices. However, other performance aspects such as response time and Quality of Service (QoS) are also being considered by these methods [4, 27, 29, 33].The frameworks can be implemented as a set of procedures, methods, and tools. These components can be part of the device itself or installed in a middleware layer in an external supervising device [34]. This layer is introduced as an intermediary between the devices and the computing platforms and it works as a smart gateway that monitors the underlying nodes and decides if offloading is needed or not [35, 36]. Table 1 summarizes the recent proposals on these aspects.Table 1
Recent frameworks for MCC.
Work Key aspects of framework Collaborative Working Architecture [20] General architecture and scheduler.OT: ad hoc cloud, fog, and Cloud Multilayered scheduling [37] Tasks specification, scheduling method.OT: ad hoc cloud of CPS nodes Cuckoo [38] Programming model and integration tools.OT: Cloud Data storage framework [39] Architecture, Database Management, and File Repository Model.OT: Cloud Flexible Framework [4] General architecture and Scheduling method.OT: Cloud Cyber-Manufacturing [40] Architecture, communication protocol and analysis.OT: Cloud, cloudlet, fog Federated IoT services [41] Management, problem formulation, and heuristic for tasks allocation over 5G.OT: MEC, ad hoc clouds VM migration framework [42] Smart precopy live migration approach.OT: Fog, Cloud Cloudlet in MCC [10] Architecture, Stochastic performance modelling.OT: Cloudlet, Cloud Resource usage optimization [5] Architecture, Resource usage, and performance evaluation modelling.OT: Cloud Mobile code offloading [43] Architecture, offloading methods in Java and push notifications.OT: Cloud QoS Aware Computation Offloading [44] Problem formulation, optimal offloading decision process.OT: ad hoc cloud Adaptive MCC framework [45] Application partitioning, offloading decision algorithm.OT: Cloud Edge Computing Framework [22] Communication and Computation Models.OT: Edge nodes Distributed computational model [46] Resource utilization specification, management system. OT: CPS nodes, Cloud Scheduling internet of things applications [47] Scheduling method, performance metrics.OT: Cloud IoT and Cloud Computing Integration [48] Architecture and components.OT: Cloud Context-aware computation offloading [49] Design pattern and estimation model.OT: Cloud Edge-Fog cloud [50] Method for distributing the processing tasks. OT: Edge and fog nodes Framework for code offloading [25] Architecture and offloading decision-making engine.OT: Edge and Cloud Offloading target (OT).


### 2.3. Findings

After reviewing the frameworks for offloading mobile computation, three main findings can be drawn that justify our proposed model for designing multilayer MCC architectures:(1)
The general objective of existing MCC architectural approaches is to improve the overall application functioning. However, in most cases, they do not consider multiple options for offloading the work nor the intermediate network infrastructure.(2)
There are numerous frameworks on how to distribute the computation of the applications to perform partial remote execution. They are mainly focused on minimizing the energy consumption of devices, increasing the performance, and maximizing the overall QoS. However, they do not consider heterogeneous performance metrics such as money cost or global network traffic.(3)
The existing works in the literature do not provide a formal framework to formalize the overall offloading process; rather, they conduct numerical results of their reference architectures. Other works have insufficient depth, or they are focused on specific issues of each layer separately.The research presented in this article pursues the same objectives as mentioned above. To the best of our knowledge, this is the first study to provide, from a holistic perspective, a comprehensive model that provides an analysis of technical aspects and includes all known computing layers of the network. This model will allow us to handle complexity of the offloading process of MCC systems.Next section introduces the proposed multilayer architecture and presents a formal basis for modelling the offloading process in the MCC paradigm.


## 2.1. Outsourcing Options for MCC

The concept of MCC has been evolved in recent times with the goal of increasing the performance by offloading the workload outside the mobile devices. As a result, the outsourcing options have been increased along the communication network as shown in Figure1.Figure 1
Outsourcing options deployed along the communication network.These outsourcing options are deployed at different layers of the network forming a pool of computing platforms that are available depending on the application context.In the top, at the network core, thecloud layer remains the most powerful computing platform. This is the traditional way for outsourcing the workload and takes its origin from the Cloud Computing Paradigm. Much of the current research on MCC is using this option as a default for implementing the offloading process [5–7].Next, thecloudlet layer concept emerges. This infrastructure is a “little” data center with the objective of bringing the cloud closer to the users [8]. Usually, cloudlets are deployed inside the organizations that use it [9, 10]. The main advantages provided by cloudlets are twofold: in the first place, to reduce service disruptions and delays caused by the network to remote servers and secondly to ensure a high level of security of the data since it does not need to be sent through Internet [11]. Therefore, cloudlets are computing platforms similar to cloud servers but prepared and scaled according to the specific requirements of the organization.The same principle of cloudlets is applied to theMobile Edge Computing (MEC) Layer. A MEC server is a data center in a box, that, in this case, it is deployed by the mobile telecom operators in close proximity to mobile subscribers [12]. Thus, access to these computing services is made through telecommunication base stations using communication protocols such as 3G or LTE to offer a service environment with high bandwidth and low latency [13, 14]. In addition, the benefits of MEC paradigm will be enhanced by the advent of 5G communication protocol [15].These last two layers (cloudlet and MEC) can be enabled in a dynamic way to deliver computing support to the cloud servers. In this way, they are not replacing but complementing the Cloud Computing Paradigm by providing a flexible computing power when necessary [16]. This operation encourages the development of IoT solutions and enhances the mobile applications when devices are moving over time.Finally, a recent trend to develop the model of MCC is the Fog Computing Paradigm. Thefog layer is a set of devices such as routers, switches, or other networking devices closer or within the end users' local networks. Their aim is to provide processing efforts as close as possible where the data is generated [17]. Thus, these devices have some computing capabilities able to compute part of the workload of the “things” and other application devices [18, 19]. As a result, service delay is significantly reduced for end user applications.


## 2.2. MCC Frameworks

Realizing the vision of multilayer offloading architecture for implementing the MCC paradigm is a challenging task because of the complexity in handling the multiple aspects involved, especially those concerned with performance evaluation and tasks scheduling. In this issue, the motivation for offloading is diverse depending on the user requirements, device configuration or application constraints. Moreover, performance aspect can be of different nature such as power consumption, time delay, money cost of using external services, and network usage.There is a need to design suitable frameworks to formalize the performance components in a homogeneous way to make decisions on when and where to outsource the processing load. In this subsection, a review of recent frameworks of MCC paradigm is described in order to reach the knowledge border in this issue.There are several existing frameworks and architectures in the literature for outsourcing the tasks of the application workload to other computing platforms. The areas of application of these techniques cover many sectors and disciplines. In general, they develop the IoT and Cyber-Physical Systems (CPS) paradigms that are experiencing a great growth in recent years. The devices involved are a heterogeneous set which mainly consists of sensors, actuators, and embedded systems. In addition, the mobile phones and other mobile devices such as wearables are also in this kind of applications.As overview of the MCC proposals, they aim to carry out a collaborative work for distributing the processing load and meeting the application requirements [20, 21]. In most cases, these applications have special features and are high demanding of computing resources such as multimedia analysis [22], high volume of data [23], or real-time constraint [4]. By distributing the work, the cloud architecture bottleneck is overcome, and better delay and predictability can be achieved [24, 25].In general terms, offloading from the “things” or mobile devices to cloud, edge, or cloudlet server always produced significant increase in performance. It is clear that the slower the devices are, the sharpest these differences become. However, the offloading criteria can consider other aspects such as network usage or money cost of external computing services [26]. In these cases, an indirect relation exists between using outsourcing services and benefits. Thus, the benefit function needs to be redefined to consider a heterogeneous type of aspects in order to make decisions on where and when to outsource the application workload.The key parts of a framework for handling the offloading process are the architecture that defines the available outsourcing platforms, the decision method on where and when to offload, and the communication model that defines how to perform this process.Regarding the decision method for offloading, in general term, it is considered as a scheduler which decides when and where to offload taking into account the application constraints, the tasks’ features, and the performance aspects of the available computing platforms. Its main aim is how to reasonably allocate the tasks to available computing platforms to minimize the total cost and load. The optimal scheduling scheme belongs to NP-complete problem set and, therefore, traditional strategies cannot be applied in a suitable way for MCC applications.To address this issue, many approaches have been proposed based on different mathematical techniques and algorithms. In recent approaches, the scheduling method can be formulated as a constrained optimization problem where a suboptimal solution is usually the best choice. These methods can be used for multitask [27] and multiobjective [28] scheduling problems in several scenarios. In the same line, there are methods based on stochastic techniques and search algorithms to deal with the high complexity of this problem [29, 30]. Other proposals are based on dynamic programming methods in order to handle the runtime variations of tasks execution [31]. Also, a decision tree can be built to classify the conditions and decision parameters in order to get a fast offloading response [32]. Finally, other works propose innovative approaches to address this problem based on techniques used for processor scheduling area such as genetic algorithms [33], and imprecise computation strategies [4].In most cases, the offloading decisions are mainly focused on energy optimization [31] since it is a critical issue for mobile devices. However, other performance aspects such as response time and Quality of Service (QoS) are also being considered by these methods [4, 27, 29, 33].The frameworks can be implemented as a set of procedures, methods, and tools. These components can be part of the device itself or installed in a middleware layer in an external supervising device [34]. This layer is introduced as an intermediary between the devices and the computing platforms and it works as a smart gateway that monitors the underlying nodes and decides if offloading is needed or not [35, 36]. Table 1 summarizes the recent proposals on these aspects.Table 1
Recent frameworks for MCC.
Work Key aspects of framework Collaborative Working Architecture [20] General architecture and scheduler.OT: ad hoc cloud, fog, and Cloud Multilayered scheduling [37] Tasks specification, scheduling method.OT: ad hoc cloud of CPS nodes Cuckoo [38] Programming model and integration tools.OT: Cloud Data storage framework [39] Architecture, Database Management, and File Repository Model.OT: Cloud Flexible Framework [4] General architecture and Scheduling method.OT: Cloud Cyber-Manufacturing [40] Architecture, communication protocol and analysis.OT: Cloud, cloudlet, fog Federated IoT services [41] Management, problem formulation, and heuristic for tasks allocation over 5G.OT: MEC, ad hoc clouds VM migration framework [42] Smart precopy live migration approach.OT: Fog, Cloud Cloudlet in MCC [10] Architecture, Stochastic performance modelling.OT: Cloudlet, Cloud Resource usage optimization [5] Architecture, Resource usage, and performance evaluation modelling.OT: Cloud Mobile code offloading [43] Architecture, offloading methods in Java and push notifications.OT: Cloud QoS Aware Computation Offloading [44] Problem formulation, optimal offloading decision process.OT: ad hoc cloud Adaptive MCC framework [45] Application partitioning, offloading decision algorithm.OT: Cloud Edge Computing Framework [22] Communication and Computation Models.OT: Edge nodes Distributed computational model [46] Resource utilization specification, management system. OT: CPS nodes, Cloud Scheduling internet of things applications [47] Scheduling method, performance metrics.OT: Cloud IoT and Cloud Computing Integration [48] Architecture and components.OT: Cloud Context-aware computation offloading [49] Design pattern and estimation model.OT: Cloud Edge-Fog cloud [50] Method for distributing the processing tasks. OT: Edge and fog nodes Framework for code offloading [25] Architecture and offloading decision-making engine.OT: Edge and Cloud Offloading target (OT).


## 2.3. Findings

After reviewing the frameworks for offloading mobile computation, three main findings can be drawn that justify our proposed model for designing multilayer MCC architectures:(1)
The general objective of existing MCC architectural approaches is to improve the overall application functioning. However, in most cases, they do not consider multiple options for offloading the work nor the intermediate network infrastructure.(2)
There are numerous frameworks on how to distribute the computation of the applications to perform partial remote execution. They are mainly focused on minimizing the energy consumption of devices, increasing the performance, and maximizing the overall QoS. However, they do not consider heterogeneous performance metrics such as money cost or global network traffic.(3)
The existing works in the literature do not provide a formal framework to formalize the overall offloading process; rather, they conduct numerical results of their reference architectures. Other works have insufficient depth, or they are focused on specific issues of each layer separately.The research presented in this article pursues the same objectives as mentioned above. To the best of our knowledge, this is the first study to provide, from a holistic perspective, a comprehensive model that provides an analysis of technical aspects and includes all known computing layers of the network. This model will allow us to handle complexity of the offloading process of MCC systems.Next section introduces the proposed multilayer architecture and presents a formal basis for modelling the offloading process in the MCC paradigm.


## 3. Multilayer Architecture Model

### 3.1. Proposed Distributed Architecture

This architecture design promotes an adaptation to changing environments and enables a dynamic scaling of computational power able to assume a variable and/or intensive application workload in a more effective manner than existing proposals. The network layers considered for offloading computation can be those described in Section2.1. An overview of the network layers to be used by the proposed framework is depicted in Figure 1.This approach attempts to identify methods to obtain the best results and performance using the network infrastructure deployments and local processing capabilities. Multiple design configurations can be supported. The mobile devices and connected “things” can be heterogeneous and have different capabilities for processing data. The middle infrastructure layers can be deployed by stakeholders to improve the execution of their mobile or IoT applications and extend them to more potential customers, for example, advanced multimedia games and complex financial apps. These layers can be equipped with specialized hardware components for improving the complex calculations, including custom cloudlets with GPUs, DSP, and cryptographic coprocessors.Below, the main technical characteristics of the MCC paradigm are introduced together with the implications and advantages provided for them by the proposed model.Theapplication partitioning method is a critical aspect for enabling the outsourcing of the code at multiple heterogeneous layers of the network architecture, even at the nearest available layers. In this manner, although an elastic approach is desirable to provide flexibility, the time determining the offloadable code must be minimized. Thus, the proposed architecture must use a static partitioning to know, at any step, the candidate parts of the application to be offloaded. Other dynamic proposals can be applicable in a complementary fashion if resources exist.Theoffloading decision mechanism can be, at the first layer for either of the mentioned options: static or dynamic. However, to ensure that the maximum advantage of the availability of multiple layers with different profiles is selected, the dynamic option is preferable. In this case, the architecture must calculate, for each layer, the offloading decision according to an optimization problem that addresses all the issues specified. This algorithm must be able to be executed quickly to leverage the potential of the available resources. In the previous section the latest trend on this issue can be found. In any case, fast methods [32] and real-time strategies [4] must be introduced to provide a rapid response and address the time constraints.The desirablegranularity unit for outsourcing must be as small as possible to provide the highest flexibility; however, small size implies higher management cost. The granularity unit for the architecture can be variable, depending on the features of the target platform for offloading. That is, small parts of the application can be outsourced for fast execution on surrounding platforms and other intensive parts can be offloaded to specialized platforms. The optimal partitioning is an NP-complete problem [51]. To avoid time consumption in the automatic analysis of the code, the variable granularity approach can be made in the design stage of the application. This requires the collaboration of the application developers by annotating into the code the possibilities for outsourcing.Theoffloading mode for the proposed MCC architecture must meet the following minimum requirements: the method must be suitable for heterogeneous infrastructures and must use a lightweight method so as to not overload the limited resources of the edge layers (devices of ad hoc clouds and fog nodes). The mode based on mobile code would appear to be the best option; however, it requires further research on mobile agent management to become a robust option for implementing heterogeneous MCC architectures [3]. Until then, a combination of virtualization and client-server methods offers the best results for each layer.Thecontext-awareness feature is important to perform offloading to several layers of the network. The desirable configuration requires a self-adaptive architecture to know what layers and devices are available at any time for offloading the work. It is significant to perform this feature without intervention of an administrator to shape dynamic environments. It is precisely in these complex environments where many options exist and where the proposed architecture attains its full potential.Finally, thecost-benefit function must be computed on each level of the architecture to determine where to perform the computation. Thus, complex algorithms are not suitable for this function, although suboptimal decisions are found. A stability of the performance for network infrastructure in the same working conditions is supposed. Hence, prediction models based on look-up tables of history data are a fast approach to the offloading of application work. Further, this information can be complemented at minimum cost with data from the device itself and the main network parameters.Table2 presents the technical characteristics of offloading methods and frameworks for the MCC paradigm and highlights the key aspects for a multilayer architecture.Table 2
Technical characteristics of offloading methods and frameworks.
MCC technical characteristics Methods Key aspects for multilayer architecture Application partitioning method Static, dynamic Exploit the available nearest resources Offloading decision mechanism Static, dynamic Leverage the potential of the available resources Granularity unit Module, bundle, subroutine, process, thread, class, component, method Provide flexibility to the application needs Offloading mode Client-server communication, virtualization, mobile agents Support heterogeneous infrastructures Context-awareness offloading Adaptive schemes Awareness of available layers Cost-benefit model Resource monitoring and profiling, parametric analysis, stochastic methods Decide where to perform the computationsThe descriptions and recommendations regarding the operation of the proposed architecture reveal that there continue to be important challenges that must be addressed to leverage the available infrastructure at multiple layers of the network. In this regard, this work introduces basic ideas and notes on specific research issues for implementing a multilayer architecture.The next subsection describes the formal framework of the architecture and the elements involved in the distributed computing.


### 3.2. General Framework

This subsection describes the general aspects of the multilayer distributed architecture for outsourcing the processing load using the combination of computing layers of the available infrastructure.According to the stated working hypothesis, the main idea behind the multilayer architecture to address computation offloading is to offer a set of options for performing the computations at different network layers where available infrastructure exists. As a rule, it is advisable to perform the processing as close as possible to where the data are acquired to reduce the delay and global network traffic. However, the final decision on where to offload each task depends on many other aspects including application requirements, devices configuration, user preferences, size of input data, and pricing. The result is a flexible and scalable model where the computations can be performed on a variety of platforms and computing layers. The formulation introduced in this subsection is utilized to better describe the contributions of the proposed architecture for providing flexibility to the processing requirements of IoT applications. Important notations and expression used in this paper are provided in Tables3 and 4.Table 3
Summary of Notations.
Notation Definition (#Expression) t j Application Task Γ A p p l = { t 1 , t 2 , … , t n } Set of Application tasks (1) L = { L 0 , … , L c c } Set of available computing layers (2) p j Computing platform p k j Computing platform of layer k p 0 j Computing platform of layer base L j = { p j 1 , p j 2 , … , p j m j } Set of computing platf. of layer j (3) A p 0 i t i m e = { p 1 , i , p 2 , i , … , p c c , i } Set of the available upper platforms of devicep0i at a time (4) Ω = { d e l a y , p o w e r , … } Set of performance aspects (5) ∧ ( Γ A p p l ) = ( t 1 , p k 1 , j 1 ) , … Sequence of platforms on which the application is processed (13) ∧ m i n ( C m p α t i m e ) List of platforms that meets the minimum computing costs for theα performance aspect at a given time ∧ m i n ( N e t α t i m e ) List of platforms that meet the minimum communication costs for theα performance aspect at a given time Δ Saturation value in calculation cost D a t a ( t i , ) Volume of data generated by taskti (10) D a t a ( t i , p j , k ) Necessary data to be moved for computing the taskti in the platform pj of the layer k (11)Table 4
Summary of Formulations.
Formulation Definition (#Expression) C m p α t i m e ( t i , p k , j ) Processing cost of the taskti in the platform pk,j for the α performance aspect (7) C m p α t i m e Γ A p p l m i n Minimum processing cost of the application for theα performance aspect (17) N e t α t i m e ( t i , p k , j ) Communication cost to move the taskti to the platform pk,j for the α performance aspect (9) N e t α t i m e Γ A p p l m i n Minimum communication cost of the execution of the application for theα performance aspect ($\text {(18)}$) E α t i m e ( Γ A p p l ) Execution cost of the application for theα performance aspect (6), (14) E α t i m e Γ A p p l m i n Minimum execution cost of the application for theα performance aspect (16) N e t α t i m e Γ A p p l ∧ m i n Minimum communication cost through the distributed architecture for the minimum processing cost (19) D a t a ( Γ A p p l ) Volume of data to be moved for computing the whole application (15) D a t a Γ A p p l m i n Volume of data to be moved through the distributed architecture to run the application within the minimum cost. (20)First, we consider a granularity unit for offloading the application task. This unit can be one of those mentioned in Table2 under the condition that the applications considered by this framework consist of a list of tasks. The tasks can be executed sequentially or in parallel depending on the interrelationships between inputs and outputs. These tasks can be variable sized depending on the offloadable code. For example, a task can be a fragment of code, such as a set of instructions, a process, a method, or a subroutine.LetΓAppl be the set of tasks of an application:(1)ΓAppl=t1,t2,…,tn.These tasks can be processed at different computing layers and platforms. Letℒ be the set of available computing layers of the architecture:(2)L=L0,…,LccThe number of network processing layers depends on numerous aspects such as execution environment, available infrastructure, and configuration options. In all these cases,L0 is the layer of the single device where the task is generated and Lcc is the remote cloud-computing server. In between can be several available computing layers to perform the processing at different levels. For example, in an autonomous vehicle application context, L1 can be the network composed of several nearby vehicles, L2 can be the layer formed by the city traffic infrastructure or deployed cloudlets, and L3 can be the mobile edge-computing infrastructure behind the communication network. Other configurations of the infrastructure can be configured for this application.Each layer has a set of computing platforms, which can be heterogeneous with different processing capabilities and abilities according to their characteristics. Thus, layerLj has mj computing platforms, where Lj∈{0,…,cc} and mj>0:(3)Lj=pj1,pj2,…,pjmj,where pjk is platform k of layer j.The different layers of the network can be deployed in sequence or in a parallel configuration where each computing platform of a layer can execute the services of the upper layers and provides services to several elements of the lower layers.For a specific device (p0i), the list of available upper platforms at a giventime can be expressed by Ap0itime:(4)Ap0itime=p1,i,p2,i,…,pcc,iThe list of platforms described in Expression (4) represents the specific network architecture for the device and indicates the possibilities for offloading the work at the given time. This list can vary with time depending on the application context. For example, if the mobile device is moving, the available infrastructure can change.Several expressions can be introduced into the proposed general framework to define the behaviour of the multilayer architecture related to performance and resource consumption issues. In this section, the execution cost and data communication are analysed. The execution cost can be any of the different performance aspects that are involved in the execution of a task in a device. In this manner, letΩ be the set of performance aspects to consider:(5)Ω=timedelay,powerconsumption,money,…For each of these aspects, the overall execution cost (E) of an application in this distributed infrastructure consists of two main components as indicated in Expression (6): (a) the computing cost (Cmp) and (b) the communication cost along the network and layers (Net):(6)EαtimeΓAppl=CmpαtimeΓAppl+NetαtimeΓAppl,where α∈ Ω.The cost expressions are a function of time because the execution conditions can change at any time depending on the workload currently processing on each platform, the other processes that eventually are executing simultaneously, and the network traffic situations.Related to the first aspect (a), the processing capability of each computing platform can be in a range from zero to extremely high. Further, there may be platforms with specific capabilities that provide services to many applications and allow the acceleration of the processing of specific types of tasks. For example, GPUs can be installed on the cloudlet servers to accelerate multimedia algorithms. In this manner, the granularity of this calculation is the cost of computing each task. For each taskti of the application, the cost of executing ti on each element of the distributed architecture can be determined. Expression (7) is the cost of processing the ti task in platform pj of layer k for the α performance aspect:(7)Cmpαtimeti,pk,j∈R+⋃0The workload of a platform can vary during the day depending on the number of devices simultaneously connected and other features. This fact is common in the intermediate platforms and in the cloud infrastructure because they collect data from different elements of the lower levels. However, it is normal that they have redundant computing elements that perform massive parallel processing. If a platform cannot compute a task, it will be assigned a cost of the saturation valueδ. That is,(8)∀x∈R+⋃0,x+δ=δ.In addition to computational cost, the framework considers the communication cost along the architecture, that is, the cost of moving the tasks between the platforms and layers as well as the data they require. The following expression indicates the communication cost to move taskti to platform pj of layer k for the α performance aspect:(9)Netαtimeti,pk,j∈R+⋃0This expression is also a function of time and can have a variable result at any time according to different aspects such as network congestion, device connectivity, bandwidth availability, and pricing. The cost of moving the data in the same processing element is null; that is, executing the entire application on the same platform processor has no communication cost. Further, as in the case of the computational cost, if it is not possible to move the data to thep platform, then the communication cost is δ, for example, when the platform is not connected to the place where this data was generated or there is no connectivity.The communication cost can be any performance aspect of the setΩ defined in Expression (5). A complementary expression can be defined to determine the transferred data (in data units) between platforms when the processing is distributed. Expressions (10) and (11) indicate, respectively, the volume of data generated by each task and the necessary data to be moved for computing task ti in platform pj of layer k. If the task is processed in the same platform, the necessary data is zero. These functions are independent from time:(10)Datati,∈R+⋃0(11)Datati,pj,k∈R+⋃0Normally, the necessary input data of a task corresponds with the generated output data from the previous task:(12)Datati+1,pj,k=Datati,The data flow and connectivity of the computing platforms define a graph to share and distribute the application workload. In the base of the graph are the mobile/embedded devices; the upper side is formed by the cloud-computing servers. Between these two sides, several intermediate computing platforms can be installed. This infrastructure allows executing advanced applications and provides improved overall performance.In this architecture, movements of the tasks can occur based on the offloading configuration and execution costs. The many possible options allow flexible implementation of the applications to optimize any of the system parameters including minimizing response time, reducing the data flow through the communication network, minimizing the energy consumption of the devices, increasing the processing time of the cloud system, and minimizing the money cost of using external resources. Hence, the proposed multilayer architecture allows the design of numerous configurations for the execution of tasks depending on the type of application, execution restrictions, or operating conditions considering the above aspects.The sequence of platforms on which the application executes is defined by the vector∧(ΓAppl) as follows:(13)∧ΓAppl≡t1,pk1,j1,…,tn,pkn,jn,where ∀ti∈ΓAppl.∧ik,j=(ti,pk,j); that is, task ti is processed on platform j of layer k.Then, the execution cost of an application can be obtained expanding Expression (6) with the platforms of the vector ∧:(14)EαtimeΓAppl=∑iCmpαtime∧ik,j+Netαtime∧ik,j.Similarly, the amount of data to be moved for computing the entire application is obtained from the next expression:(15)DataΓAppl=∑iData∧ik,j.From the previous expressions, calculations can be made to optimize the processing according to the configuration criteria of the architecture. Consequently, a more appropriate sequence of platforms for outsourcing is obtained. This information can guide the scheduling methods and the offloading strategy to achieve the best performance.The following expression obtains the minimum cost of the execution of the application considering the cost of the platforms and cost of data movement along the communication network:(16)EαtimeΓApplmin=mink,j⁡∑iCmpαtime∧ik,j+Netαtime∧ik,jThe cost of one of the components of Expression (6) could be similarly obtained. That is, the next expressions produce the cost for the separate components:(17)CmpαtimeΓApplmin=mink,j⁡∑iCmpαtime∧ik,j(18)NetαtimeΓApplmin=mink,j⁡∑iifCmpαtime∧ik,j<δ:Nettime∧ik,jelse:δNote that Expressions (16), (17), and ($\text {(18)}$) only consider the calculations for the platforms which are able to compute the tasks.The list of platforms that meets the minimum cost is defined as follows. Let∧min(Cmpαtime) be the sequence of platforms executing each of the tasks that achieve the minimum computing cost of the application and let ∧min(Netαtime) be the sequence of platforms that achieve the minimum communication cost through the distributed architecture among the different options for offloading.Then, considering the minimum processing cost for one performance aspectα (for example time delay), the following expression obtains the minimum communication cost through the distributed architecture to execute the application:(19)NetαtimeΓAppl∧min=∑iNetαtime∧ik,j,where ∧i[k,j]∈∧min(Cmpαtime).Finally, the following expression obtains the amount of data to be moved through the distributed architecture to execute the application within the minimum cost:(20)DataΓApplmin=∑iData∧ik,j,where ∧i[k,j]∈∧min(Netαtime).It is notable that the information regarding computational and communication costs is not fixed and it is time dependent for some of the performance aspects. Thus, the computing platforms of the architecture can follow different strategies to derive the correct decisions regarding offloading tasks to another platform or layer.First, it can use prediction techniques based on historical data. In this manner, the devices can know the estimated performance of the available platforms quickly. A possible implementation can be a look-up table that stores the performance data for each interesting aspect. The tasks can be clustered to ensure manageable table sizes. For example, the similarity criteria can be any indicator of the type of specialized processing required, such as integer, floating point, multimedia, or cryptographic. This feature should be noted in the programming stage of the application to ease the offloading process. Moreover, this data must be updated after each operation.Secondly, there are methods for periodically testing the data time costs. Light threads can be launched at the beginning of processing to request performance conditions of the architecture.A combination of the above methods can be made to obtain more accurate information regarding the execution context of the architecture.In any case, an embedded middleware layer could be necessary to take charge of this job. This layer is already playing an increasingly important role in the edge computing paradigm to perform discovery and other broker services [52]. In this case, it is responsible for gathering and evaluating all the relevant data and identifying the available options to make the right offloading decision. The cost of this decentralized approach is lower than a centralized control hosted somewhere along the network. However, the rational operation of each device inad hoc scenario should be satisfactory, but presumably suboptimal [53].


## 3.1. Proposed Distributed Architecture

This architecture design promotes an adaptation to changing environments and enables a dynamic scaling of computational power able to assume a variable and/or intensive application workload in a more effective manner than existing proposals. The network layers considered for offloading computation can be those described in Section2.1. An overview of the network layers to be used by the proposed framework is depicted in Figure 1.This approach attempts to identify methods to obtain the best results and performance using the network infrastructure deployments and local processing capabilities. Multiple design configurations can be supported. The mobile devices and connected “things” can be heterogeneous and have different capabilities for processing data. The middle infrastructure layers can be deployed by stakeholders to improve the execution of their mobile or IoT applications and extend them to more potential customers, for example, advanced multimedia games and complex financial apps. These layers can be equipped with specialized hardware components for improving the complex calculations, including custom cloudlets with GPUs, DSP, and cryptographic coprocessors.Below, the main technical characteristics of the MCC paradigm are introduced together with the implications and advantages provided for them by the proposed model.Theapplication partitioning method is a critical aspect for enabling the outsourcing of the code at multiple heterogeneous layers of the network architecture, even at the nearest available layers. In this manner, although an elastic approach is desirable to provide flexibility, the time determining the offloadable code must be minimized. Thus, the proposed architecture must use a static partitioning to know, at any step, the candidate parts of the application to be offloaded. Other dynamic proposals can be applicable in a complementary fashion if resources exist.Theoffloading decision mechanism can be, at the first layer for either of the mentioned options: static or dynamic. However, to ensure that the maximum advantage of the availability of multiple layers with different profiles is selected, the dynamic option is preferable. In this case, the architecture must calculate, for each layer, the offloading decision according to an optimization problem that addresses all the issues specified. This algorithm must be able to be executed quickly to leverage the potential of the available resources. In the previous section the latest trend on this issue can be found. In any case, fast methods [32] and real-time strategies [4] must be introduced to provide a rapid response and address the time constraints.The desirablegranularity unit for outsourcing must be as small as possible to provide the highest flexibility; however, small size implies higher management cost. The granularity unit for the architecture can be variable, depending on the features of the target platform for offloading. That is, small parts of the application can be outsourced for fast execution on surrounding platforms and other intensive parts can be offloaded to specialized platforms. The optimal partitioning is an NP-complete problem [51]. To avoid time consumption in the automatic analysis of the code, the variable granularity approach can be made in the design stage of the application. This requires the collaboration of the application developers by annotating into the code the possibilities for outsourcing.Theoffloading mode for the proposed MCC architecture must meet the following minimum requirements: the method must be suitable for heterogeneous infrastructures and must use a lightweight method so as to not overload the limited resources of the edge layers (devices of ad hoc clouds and fog nodes). The mode based on mobile code would appear to be the best option; however, it requires further research on mobile agent management to become a robust option for implementing heterogeneous MCC architectures [3]. Until then, a combination of virtualization and client-server methods offers the best results for each layer.Thecontext-awareness feature is important to perform offloading to several layers of the network. The desirable configuration requires a self-adaptive architecture to know what layers and devices are available at any time for offloading the work. It is significant to perform this feature without intervention of an administrator to shape dynamic environments. It is precisely in these complex environments where many options exist and where the proposed architecture attains its full potential.Finally, thecost-benefit function must be computed on each level of the architecture to determine where to perform the computation. Thus, complex algorithms are not suitable for this function, although suboptimal decisions are found. A stability of the performance for network infrastructure in the same working conditions is supposed. Hence, prediction models based on look-up tables of history data are a fast approach to the offloading of application work. Further, this information can be complemented at minimum cost with data from the device itself and the main network parameters.Table2 presents the technical characteristics of offloading methods and frameworks for the MCC paradigm and highlights the key aspects for a multilayer architecture.Table 2
Technical characteristics of offloading methods and frameworks.
MCC technical characteristics Methods Key aspects for multilayer architecture Application partitioning method Static, dynamic Exploit the available nearest resources Offloading decision mechanism Static, dynamic Leverage the potential of the available resources Granularity unit Module, bundle, subroutine, process, thread, class, component, method Provide flexibility to the application needs Offloading mode Client-server communication, virtualization, mobile agents Support heterogeneous infrastructures Context-awareness offloading Adaptive schemes Awareness of available layers Cost-benefit model Resource monitoring and profiling, parametric analysis, stochastic methods Decide where to perform the computationsThe descriptions and recommendations regarding the operation of the proposed architecture reveal that there continue to be important challenges that must be addressed to leverage the available infrastructure at multiple layers of the network. In this regard, this work introduces basic ideas and notes on specific research issues for implementing a multilayer architecture.The next subsection describes the formal framework of the architecture and the elements involved in the distributed computing.


## 3.2. General Framework

This subsection describes the general aspects of the multilayer distributed architecture for outsourcing the processing load using the combination of computing layers of the available infrastructure.According to the stated working hypothesis, the main idea behind the multilayer architecture to address computation offloading is to offer a set of options for performing the computations at different network layers where available infrastructure exists. As a rule, it is advisable to perform the processing as close as possible to where the data are acquired to reduce the delay and global network traffic. However, the final decision on where to offload each task depends on many other aspects including application requirements, devices configuration, user preferences, size of input data, and pricing. The result is a flexible and scalable model where the computations can be performed on a variety of platforms and computing layers. The formulation introduced in this subsection is utilized to better describe the contributions of the proposed architecture for providing flexibility to the processing requirements of IoT applications. Important notations and expression used in this paper are provided in Tables3 and 4.Table 3
Summary of Notations.
Notation Definition (#Expression) t j Application Task Γ A p p l = { t 1 , t 2 , … , t n } Set of Application tasks (1) L = { L 0 , … , L c c } Set of available computing layers (2) p j Computing platform p k j Computing platform of layer k p 0 j Computing platform of layer base L j = { p j 1 , p j 2 , … , p j m j } Set of computing platf. of layer j (3) A p 0 i t i m e = { p 1 , i , p 2 , i , … , p c c , i } Set of the available upper platforms of devicep0i at a time (4) Ω = { d e l a y , p o w e r , … } Set of performance aspects (5) ∧ ( Γ A p p l ) = ( t 1 , p k 1 , j 1 ) , … Sequence of platforms on which the application is processed (13) ∧ m i n ( C m p α t i m e ) List of platforms that meets the minimum computing costs for theα performance aspect at a given time ∧ m i n ( N e t α t i m e ) List of platforms that meet the minimum communication costs for theα performance aspect at a given time Δ Saturation value in calculation cost D a t a ( t i , ) Volume of data generated by taskti (10) D a t a ( t i , p j , k ) Necessary data to be moved for computing the taskti in the platform pj of the layer k (11)Table 4
Summary of Formulations.
Formulation Definition (#Expression) C m p α t i m e ( t i , p k , j ) Processing cost of the taskti in the platform pk,j for the α performance aspect (7) C m p α t i m e Γ A p p l m i n Minimum processing cost of the application for theα performance aspect (17) N e t α t i m e ( t i , p k , j ) Communication cost to move the taskti to the platform pk,j for the α performance aspect (9) N e t α t i m e Γ A p p l m i n Minimum communication cost of the execution of the application for theα performance aspect ($\text {(18)}$) E α t i m e ( Γ A p p l ) Execution cost of the application for theα performance aspect (6), (14) E α t i m e Γ A p p l m i n Minimum execution cost of the application for theα performance aspect (16) N e t α t i m e Γ A p p l ∧ m i n Minimum communication cost through the distributed architecture for the minimum processing cost (19) D a t a ( Γ A p p l ) Volume of data to be moved for computing the whole application (15) D a t a Γ A p p l m i n Volume of data to be moved through the distributed architecture to run the application within the minimum cost. (20)First, we consider a granularity unit for offloading the application task. This unit can be one of those mentioned in Table2 under the condition that the applications considered by this framework consist of a list of tasks. The tasks can be executed sequentially or in parallel depending on the interrelationships between inputs and outputs. These tasks can be variable sized depending on the offloadable code. For example, a task can be a fragment of code, such as a set of instructions, a process, a method, or a subroutine.LetΓAppl be the set of tasks of an application:(1)ΓAppl=t1,t2,…,tn.These tasks can be processed at different computing layers and platforms. Letℒ be the set of available computing layers of the architecture:(2)L=L0,…,LccThe number of network processing layers depends on numerous aspects such as execution environment, available infrastructure, and configuration options. In all these cases,L0 is the layer of the single device where the task is generated and Lcc is the remote cloud-computing server. In between can be several available computing layers to perform the processing at different levels. For example, in an autonomous vehicle application context, L1 can be the network composed of several nearby vehicles, L2 can be the layer formed by the city traffic infrastructure or deployed cloudlets, and L3 can be the mobile edge-computing infrastructure behind the communication network. Other configurations of the infrastructure can be configured for this application.Each layer has a set of computing platforms, which can be heterogeneous with different processing capabilities and abilities according to their characteristics. Thus, layerLj has mj computing platforms, where Lj∈{0,…,cc} and mj>0:(3)Lj=pj1,pj2,…,pjmj,where pjk is platform k of layer j.The different layers of the network can be deployed in sequence or in a parallel configuration where each computing platform of a layer can execute the services of the upper layers and provides services to several elements of the lower layers.For a specific device (p0i), the list of available upper platforms at a giventime can be expressed by Ap0itime:(4)Ap0itime=p1,i,p2,i,…,pcc,iThe list of platforms described in Expression (4) represents the specific network architecture for the device and indicates the possibilities for offloading the work at the given time. This list can vary with time depending on the application context. For example, if the mobile device is moving, the available infrastructure can change.Several expressions can be introduced into the proposed general framework to define the behaviour of the multilayer architecture related to performance and resource consumption issues. In this section, the execution cost and data communication are analysed. The execution cost can be any of the different performance aspects that are involved in the execution of a task in a device. In this manner, letΩ be the set of performance aspects to consider:(5)Ω=timedelay,powerconsumption,money,…For each of these aspects, the overall execution cost (E) of an application in this distributed infrastructure consists of two main components as indicated in Expression (6): (a) the computing cost (Cmp) and (b) the communication cost along the network and layers (Net):(6)EαtimeΓAppl=CmpαtimeΓAppl+NetαtimeΓAppl,where α∈ Ω.The cost expressions are a function of time because the execution conditions can change at any time depending on the workload currently processing on each platform, the other processes that eventually are executing simultaneously, and the network traffic situations.Related to the first aspect (a), the processing capability of each computing platform can be in a range from zero to extremely high. Further, there may be platforms with specific capabilities that provide services to many applications and allow the acceleration of the processing of specific types of tasks. For example, GPUs can be installed on the cloudlet servers to accelerate multimedia algorithms. In this manner, the granularity of this calculation is the cost of computing each task. For each taskti of the application, the cost of executing ti on each element of the distributed architecture can be determined. Expression (7) is the cost of processing the ti task in platform pj of layer k for the α performance aspect:(7)Cmpαtimeti,pk,j∈R+⋃0The workload of a platform can vary during the day depending on the number of devices simultaneously connected and other features. This fact is common in the intermediate platforms and in the cloud infrastructure because they collect data from different elements of the lower levels. However, it is normal that they have redundant computing elements that perform massive parallel processing. If a platform cannot compute a task, it will be assigned a cost of the saturation valueδ. That is,(8)∀x∈R+⋃0,x+δ=δ.In addition to computational cost, the framework considers the communication cost along the architecture, that is, the cost of moving the tasks between the platforms and layers as well as the data they require. The following expression indicates the communication cost to move taskti to platform pj of layer k for the α performance aspect:(9)Netαtimeti,pk,j∈R+⋃0This expression is also a function of time and can have a variable result at any time according to different aspects such as network congestion, device connectivity, bandwidth availability, and pricing. The cost of moving the data in the same processing element is null; that is, executing the entire application on the same platform processor has no communication cost. Further, as in the case of the computational cost, if it is not possible to move the data to thep platform, then the communication cost is δ, for example, when the platform is not connected to the place where this data was generated or there is no connectivity.The communication cost can be any performance aspect of the setΩ defined in Expression (5). A complementary expression can be defined to determine the transferred data (in data units) between platforms when the processing is distributed. Expressions (10) and (11) indicate, respectively, the volume of data generated by each task and the necessary data to be moved for computing task ti in platform pj of layer k. If the task is processed in the same platform, the necessary data is zero. These functions are independent from time:(10)Datati,∈R+⋃0(11)Datati,pj,k∈R+⋃0Normally, the necessary input data of a task corresponds with the generated output data from the previous task:(12)Datati+1,pj,k=Datati,The data flow and connectivity of the computing platforms define a graph to share and distribute the application workload. In the base of the graph are the mobile/embedded devices; the upper side is formed by the cloud-computing servers. Between these two sides, several intermediate computing platforms can be installed. This infrastructure allows executing advanced applications and provides improved overall performance.In this architecture, movements of the tasks can occur based on the offloading configuration and execution costs. The many possible options allow flexible implementation of the applications to optimize any of the system parameters including minimizing response time, reducing the data flow through the communication network, minimizing the energy consumption of the devices, increasing the processing time of the cloud system, and minimizing the money cost of using external resources. Hence, the proposed multilayer architecture allows the design of numerous configurations for the execution of tasks depending on the type of application, execution restrictions, or operating conditions considering the above aspects.The sequence of platforms on which the application executes is defined by the vector∧(ΓAppl) as follows:(13)∧ΓAppl≡t1,pk1,j1,…,tn,pkn,jn,where ∀ti∈ΓAppl.∧ik,j=(ti,pk,j); that is, task ti is processed on platform j of layer k.Then, the execution cost of an application can be obtained expanding Expression (6) with the platforms of the vector ∧:(14)EαtimeΓAppl=∑iCmpαtime∧ik,j+Netαtime∧ik,j.Similarly, the amount of data to be moved for computing the entire application is obtained from the next expression:(15)DataΓAppl=∑iData∧ik,j.From the previous expressions, calculations can be made to optimize the processing according to the configuration criteria of the architecture. Consequently, a more appropriate sequence of platforms for outsourcing is obtained. This information can guide the scheduling methods and the offloading strategy to achieve the best performance.The following expression obtains the minimum cost of the execution of the application considering the cost of the platforms and cost of data movement along the communication network:(16)EαtimeΓApplmin=mink,j⁡∑iCmpαtime∧ik,j+Netαtime∧ik,jThe cost of one of the components of Expression (6) could be similarly obtained. That is, the next expressions produce the cost for the separate components:(17)CmpαtimeΓApplmin=mink,j⁡∑iCmpαtime∧ik,j(18)NetαtimeΓApplmin=mink,j⁡∑iifCmpαtime∧ik,j<δ:Nettime∧ik,jelse:δNote that Expressions (16), (17), and ($\text {(18)}$) only consider the calculations for the platforms which are able to compute the tasks.The list of platforms that meets the minimum cost is defined as follows. Let∧min(Cmpαtime) be the sequence of platforms executing each of the tasks that achieve the minimum computing cost of the application and let ∧min(Netαtime) be the sequence of platforms that achieve the minimum communication cost through the distributed architecture among the different options for offloading.Then, considering the minimum processing cost for one performance aspectα (for example time delay), the following expression obtains the minimum communication cost through the distributed architecture to execute the application:(19)NetαtimeΓAppl∧min=∑iNetαtime∧ik,j,where ∧i[k,j]∈∧min(Cmpαtime).Finally, the following expression obtains the amount of data to be moved through the distributed architecture to execute the application within the minimum cost:(20)DataΓApplmin=∑iData∧ik,j,where ∧i[k,j]∈∧min(Netαtime).It is notable that the information regarding computational and communication costs is not fixed and it is time dependent for some of the performance aspects. Thus, the computing platforms of the architecture can follow different strategies to derive the correct decisions regarding offloading tasks to another platform or layer.First, it can use prediction techniques based on historical data. In this manner, the devices can know the estimated performance of the available platforms quickly. A possible implementation can be a look-up table that stores the performance data for each interesting aspect. The tasks can be clustered to ensure manageable table sizes. For example, the similarity criteria can be any indicator of the type of specialized processing required, such as integer, floating point, multimedia, or cryptographic. This feature should be noted in the programming stage of the application to ease the offloading process. Moreover, this data must be updated after each operation.Secondly, there are methods for periodically testing the data time costs. Light threads can be launched at the beginning of processing to request performance conditions of the architecture.A combination of the above methods can be made to obtain more accurate information regarding the execution context of the architecture.In any case, an embedded middleware layer could be necessary to take charge of this job. This layer is already playing an increasingly important role in the edge computing paradigm to perform discovery and other broker services [52]. In this case, it is responsible for gathering and evaluating all the relevant data and identifying the available options to make the right offloading decision. The cost of this decentralized approach is lower than a centralized control hosted somewhere along the network. However, the rational operation of each device inad hoc scenario should be satisfactory, but presumably suboptimal [53].


## 4. Application Examples

This section describes how the proposed multilayer computational model can handle real-world scenarios in order to find the best execution cost (Eαtime(ΓAppl)) for computing an application taking into account the different available offloading layers. For this purpose, three application contexts have been defined to test the model under different conditions. These scenarios correspond to operation modes affected by user preferences or device configurations. This section just shows three examples. Of course, other application contexts could be defined. In all cases, the main contribution of the proposed model is the versatility in formalizing the offloading requirements of each environment and the better leveraging of the available resources in the communication network. The exact values in simulation and numerical examples provided in this section do not impact essentially to these outcomes.In these scenarios, the IoT layer is composed by mobile devices such as smartphones, tablet PCs, and smartwatches. These are heterogeneous devices and might have different computing and communication capabilities. In addition, the users might configure the devices according to their preferences and economic plans.The example application consists of an Augmented Reality (AR) system for Smart Cities which enables the user to move freely through the modelled environment of the city using their mobile devices. This technology recognizes what you are doing and then enhances it. The AR systems for Smart Cities have great potential for all involved [54], for example, the disabled citizens in order to know the resources for inclusive city and accessible paths along the city. Moreover, the latest AR solutions consider edge resources together with cloudlets and cloud server as high-end computing platforms to offload the AR applications [55]. Therefore, the available layers of the architecture are LA = {IoT devices, fog computing, fog nodes, cloudlet, MEC server, Cloud server.In this example, the scheduling algorithm of the middleware could be based on prediction techniques and, for example, the performance estimations could be like those shown in Table5 for each case. These data have been retrieved from other sources and on our own experiments of previous research on this matter [4, 46]. Scheduling algorithms based on historical data are common in dynamic environments [56]. Therefore, these performance estimations are recorded in a Look-Up Table that can be embedded in each device and frequently updated by broker services.Table 5
Performance estimations.
Offloading target Application context (A) BATTERY SAVING (B) MONETARY COST (C) REAL-TIME Ω = { b a t t e r y   c o n s u m p t i o n } Ω = { m o n e y } Ω = { t i m e   d e l a y } Netbatt.cons.time(ti,pk,j)mJ/Kbpsa C m p m o n e y . t i m e ( t i , p k , j ) ($/hour) N e t t i m e  d e l a y t i m e ( t i , p k , j ) (ms/KB) C m p t i m e  d e l a y t i m e ( t i , p k , j ) (ms) Fog nodes 1  0.2143 250 Cloudlet 1 0.256 0.4286 52 MEC server 1.56 0.128 0.7812 52 Cloud server 1.56 b / 1c 0.064 d 2.2177 50 aEstimations from [57]. bCloud server connection through mobile station. cCloud server connection through cloudlet. dOn-demand instance for t2.medium server (https://aws.amazon.com/ec2/spot/pricing/).


### 4.1. Battery Saving Application Context

The general definition of these contexts is that the user device is configured to save battery power. Battery consumption is a performance aspect for only battery-powered devices such as mobile devices or wireless sensors. Thus, it only applies to IoT devices. In addition, this is a static feature. That is(21)∀k>0,Cmpbatt.cons.timeti,pk,j=0It means that the processing cost of outsourcing platforms does not matter for this configuration since the only important thing is energy saving. Thus, under this configuration, the application workload will be outsourced whenever and wherever possible. However, the communication costs for moving the tasks to another computing platform are not void since data communication consumes battery. Therefore, in this scenario, the computation of a task (ti) will be offloaded when battery consumption of local execution is greater than communication cost to some outsourcing platform (pk,j). That is (22)∃k>0,Netbatt.cons.timeti,pk,j<Cmpbatt.cons.timeti,p0,jThis consumption depends on where the data is moved. Generally, communication costs through a wireless local network are lower than through a telecommunication network such as LTE [57]. Therefore, in this case a table with the communication cost of the mobile device is needed to implement the Net function. For example, Table 5(A) shows estimated data of battery consumption (Netbatt.cons.time(ti,pk,j)) for different communication options.As can be seen in Table5(A), the communication cost through mobile operators is higher than through wired Internet. For this reason, the cost of cloud server is different according to how the connection is made.


### 4.2. Monetary Cost Saving Application Context

The general definition of this context is that the user device does not have performance enough to run the VR application. It is only used for displaying the results and, therefore, it has to outsource the processing load. In such a context, it is configured to save processing expenses when using an external computing platform.This is a similar approach than the previous one but, in this case, the device is unable to compute any task; that is(23)Cmpmoneytimeti,p0,j=δThis scenario supposes that using outsourcing resources has money cost under theInfrastructure-as-a-service (IaaS) model. That is, the workload can be outsourced, but the infrastructure owner will charge you the cost. In addition, the cost can be variable depending on where the computing platform is placed, its performance, and the moment (hour/day/month) when it is required. The new cloud service brokerage efficiently trades infrastructure cloud services among multiple resource providers and consumers [58, 59]. In this way, billing is completely flexible and just based on resource usage.However, in order to know the processing cost, a dynamic monitoring is needed. For example, Table5(B) shows an estimated data (Cmpmoney.time(ti,pk,j)) for the different outsourcing options at an instanttime. In this case, fog layer is not considered since it is difficult to estimate the monetary cost of outsourcing to it. In addition, due to its limited computing capabilities, this layer is usually used for short periods of time which are not comparable with published hourly rates.Generally, cloudlet and MEC price should be higher than cloud since they cannot take advantage of economies of scale. In addition, they are limited to a more restricted area, and thus, they have less competitors for outsourcing. From other points of view, cloudlet layer can be deployed and owned by organizations and therefore they should have no costs for their users.Regarding Net function, the devices might have a communication cost. It could also be taken into account in order to offload through local wireless or communication network, since, in many cases, the former has no costs.


### 4.3. Real-Time Application Context

The general definition of these contexts is to minimize the computing delay of the application in order to meet quality constraints of demanding real-time AR algorithms.In this scenario, there exists a very dynamic nature of the cost matrix since it depends on the current workload of each computer. In addition, the data must be moved to the target computing platform, and therefore the communication costs are responsible for a relevant part of the total delay [45, 60]. For example, Table 5(C) shows an estimated data at an instanttime.As can be noted from the above data, the communication costs are increasing with distance from outsourcing platforms. In addition, fog and cloudlet layers can be into the same Local Area Network (LAN) of the device or at a very close one. Of course, these costs depend on communication technology used. For 5G technology the delay of MEC and cloud platforms will be significantly decreased.Regarding computing delay, in general, the computing costs of intermediate computing platforms and the cloud server are significantly lower than those of the device itself. In this way, a decreasing computing cost must occur for the most of application tasks:(24)∀pk,j∈Ap0itime,Cmpαtimeti,pk,j<Cmpαtimeti,pk-1,jThis consideration is quite frequent in environments designed for outsourcing of tasks from mobile devices or connected “things.”Table5(C) shows the fact that fog nodes are limited resourced platforms, but useful for reaching a fast response in data operation. Another aspect taken into account is that cloud servers are powerful platforms, but they are usually public infrastructures used by many applications at once. Thus, their perceived performance is similar to MEC and cloudlet platforms.There may be certain applications more suited than others to make the most of this model. Certainly, applications containing intensive computing tasks will be more candidates to outsource than others; moreover, those tasks that require or produce a large volume of data should be processed as close as possible to the data sources to minimize communication costs.


## 4.1. Battery Saving Application Context

The general definition of these contexts is that the user device is configured to save battery power. Battery consumption is a performance aspect for only battery-powered devices such as mobile devices or wireless sensors. Thus, it only applies to IoT devices. In addition, this is a static feature. That is(21)∀k>0,Cmpbatt.cons.timeti,pk,j=0It means that the processing cost of outsourcing platforms does not matter for this configuration since the only important thing is energy saving. Thus, under this configuration, the application workload will be outsourced whenever and wherever possible. However, the communication costs for moving the tasks to another computing platform are not void since data communication consumes battery. Therefore, in this scenario, the computation of a task (ti) will be offloaded when battery consumption of local execution is greater than communication cost to some outsourcing platform (pk,j). That is (22)∃k>0,Netbatt.cons.timeti,pk,j<Cmpbatt.cons.timeti,p0,jThis consumption depends on where the data is moved. Generally, communication costs through a wireless local network are lower than through a telecommunication network such as LTE [57]. Therefore, in this case a table with the communication cost of the mobile device is needed to implement the Net function. For example, Table 5(A) shows estimated data of battery consumption (Netbatt.cons.time(ti,pk,j)) for different communication options.As can be seen in Table5(A), the communication cost through mobile operators is higher than through wired Internet. For this reason, the cost of cloud server is different according to how the connection is made.


## 4.2. Monetary Cost Saving Application Context

The general definition of this context is that the user device does not have performance enough to run the VR application. It is only used for displaying the results and, therefore, it has to outsource the processing load. In such a context, it is configured to save processing expenses when using an external computing platform.This is a similar approach than the previous one but, in this case, the device is unable to compute any task; that is(23)Cmpmoneytimeti,p0,j=δThis scenario supposes that using outsourcing resources has money cost under theInfrastructure-as-a-service (IaaS) model. That is, the workload can be outsourced, but the infrastructure owner will charge you the cost. In addition, the cost can be variable depending on where the computing platform is placed, its performance, and the moment (hour/day/month) when it is required. The new cloud service brokerage efficiently trades infrastructure cloud services among multiple resource providers and consumers [58, 59]. In this way, billing is completely flexible and just based on resource usage.However, in order to know the processing cost, a dynamic monitoring is needed. For example, Table5(B) shows an estimated data (Cmpmoney.time(ti,pk,j)) for the different outsourcing options at an instanttime. In this case, fog layer is not considered since it is difficult to estimate the monetary cost of outsourcing to it. In addition, due to its limited computing capabilities, this layer is usually used for short periods of time which are not comparable with published hourly rates.Generally, cloudlet and MEC price should be higher than cloud since they cannot take advantage of economies of scale. In addition, they are limited to a more restricted area, and thus, they have less competitors for outsourcing. From other points of view, cloudlet layer can be deployed and owned by organizations and therefore they should have no costs for their users.Regarding Net function, the devices might have a communication cost. It could also be taken into account in order to offload through local wireless or communication network, since, in many cases, the former has no costs.


## 4.3. Real-Time Application Context

The general definition of these contexts is to minimize the computing delay of the application in order to meet quality constraints of demanding real-time AR algorithms.In this scenario, there exists a very dynamic nature of the cost matrix since it depends on the current workload of each computer. In addition, the data must be moved to the target computing platform, and therefore the communication costs are responsible for a relevant part of the total delay [45, 60]. For example, Table 5(C) shows an estimated data at an instanttime.As can be noted from the above data, the communication costs are increasing with distance from outsourcing platforms. In addition, fog and cloudlet layers can be into the same Local Area Network (LAN) of the device or at a very close one. Of course, these costs depend on communication technology used. For 5G technology the delay of MEC and cloud platforms will be significantly decreased.Regarding computing delay, in general, the computing costs of intermediate computing platforms and the cloud server are significantly lower than those of the device itself. In this way, a decreasing computing cost must occur for the most of application tasks:(24)∀pk,j∈Ap0itime,Cmpαtimeti,pk,j<Cmpαtimeti,pk-1,jThis consideration is quite frequent in environments designed for outsourcing of tasks from mobile devices or connected “things.”Table5(C) shows the fact that fog nodes are limited resourced platforms, but useful for reaching a fast response in data operation. Another aspect taken into account is that cloud servers are powerful platforms, but they are usually public infrastructures used by many applications at once. Thus, their perceived performance is similar to MEC and cloudlet platforms.There may be certain applications more suited than others to make the most of this model. Certainly, applications containing intensive computing tasks will be more candidates to outsource than others; moreover, those tasks that require or produce a large volume of data should be processed as close as possible to the data sources to minimize communication costs.


## 5. Conclusions and Future Work

MCC is a recent paradigm with disruptive implications for mobile computing and for the development of IoT and CPS advanced applications. This paradigm promotes the QoS of devices and ‘things’ by outsourcing their computation tasks to external computing platforms. The majority of proposals in MCC take into account only the cloud layer for outsourcing the application workload. Another computing infrastructure hosted at different network layer is also being introduced recently (cloudlets, fog nodes, etc.).This work generalizes the idea of MCC paradigm to multiple computing platforms at different network layers. The proposed model considers not only the cloud layer, but also other network computing layers such as fog computing, Mobile Edge Computing, and cloudlet computing platforms.To the best of our knowledge, this paper is the first work that extends the MCC paradigm to the available computing layers by considering them in a comprehensive and integrated fashion with the distributed architecture. To this end, a general framework of a multilayer network architecture is described. The proposed model formalizes the processing of the application workload and the implications of the distributed configurations in the performance and in the communications needs. This model offers a versatile approach where different performance aspects can be considered within the same framework (time delay, power consumption, money, etc.). In this way, the proposal analyzes from a holistic perspective the technical aspects involved in Mobile Cloud Computing paradigm taking into account the contributions and results of the more recent works on these topics.This framework enables decision making regarding scheduling the outsourcing of the applications tasks based on the current and historical information of the systems. Several examples of these features have been described in three scenarios where different performance aspects and computing platforms are involved. Each of them is conditioned by its user preferences or device configurations. The results show the versatility of the model to represent all the elements involved.In the future, this research can be extended to cover remaining challenges around this paradigm, for example, (a) the specification of the network topology and link capacity of the computing platforms along the network; (b) the definition of a middleware layer to drive the outsourcing process; (c) the introduction of discovery and broker services, and a suitable decision method for outsourcing; and (d) the design of mechanisms for collecting and disseminating the performance information, and the evaluation about the associate cost for doing that. In any case, the extension of the model can be made from the proposal of this work by adding new modules and features.



---

*Source: 3951495-2019-02-11.xml*