# BPFC-SDNs: A Blockchain-Based and Policy-Oriented Forwarding Control for the SDN Interdomain

Authors: Yingying Ma; Zhaocheng Wang; Chaowen Chang; Ping Wu
Journal: Security and Communication Networks
 (2023)

---

## Abstract

Policy-oriented forwarding control has been widely recognized in a software-defined network (SDN). However, in the multidomain SDN scenario, policy-oriented forwarding control confronts the challenges that the lack of a higher-level abstraction policy paradigm, cross-domain policy unknowability and policy conflict, a distributed and untrusted communication environment, and latency sensitivity. To tackle these challenges, this article proposes BPFC-SDNs, a blockchain-based and policy-oriented forwarding control for the SDN interdomain. As the basis of BPFC-SDNs, we define an attribute-based interdomain forwarding control policy paradigm and implement secure and trusted cross-domain information and policy sharing based on blockchain. The main idea of BPFC-SDNs is to achieve cross-domain dataflow forwarding control based on the global collaborative policy. Specifically, we propose a physically centralized and logically isolated architecture to ensure efficient and secure information exchange between the SDN and blockchain. Moreover, we design a combined on-chain and off-chain functional model to separate forwarding control from the blockchain, which enables forwarding control based on trusted data while avoiding the introduction of high latency and computational overhead of the blockchain. Finally, we implement a prototype for BPFC-SDNs, and the experimental results indicate that BPFC-SDNs can provide effective forwarding control for the SDN interdomain with acceptable latency and good scalability.


---

## Body

## 1. Introduction

As a new network architecture, the software-defined network (SDN) has the characteristics of forwarding and control separation, logic concentration, and open programmability, so it provides a good foundation for flexible and efficient network management and control [1]. With the widespread deployment of SDN, in the SDN multidomain environment, different organizations using SDN as network infrastructure have the need to control the forwarding path of cross-domain dataflows in a fine-grained manner for security, the network carrying capacity, benefits, and other considerations [2]. For example, a company might want fine-grained control of its different service traffic to be carried by different service providers, or an enterprise might want the high-security level dataflows that it sends to be transmitted by high-security level forwarding paths, to ensure that high-security level dataflows are isolated from low-security level paths, or a transmission domain might refuse to carry dataflows of a certain service type due to service capacity limitations, or a transmission domain only receives dataflows sent by devices with a trust level higher than a certain level, and so on. Policy-based is a widely agreed method for SDN dataflow forwarding control. Therefore, how to achieve fine-grained dataflow forwarding path control among SDN domains based on policies is a problem worth studying.There have been various studies on SDN policy forwarding control and policy implementation. They include controlling dataflows forwarding based on custom policies [3–5], implementing traffic scheduling based on policies [6], filtering or redirecting abnormal data flows based on policies [7], network programming language to facilitate network policy creation, development, and deployment [8, 9], implementing policy expression as well as policy conflict detection and resolution based on specific data structures [10, 11], optimizing policy deployment [12–15], resolving inconsistencies during policy implementation [16, 17], detecting and resolving the overlaps and conflicts among the flow rules [18, 19], designing policy implementation schemes to adapt to dynamic changes in the link state [20, 21], and forth.The above studies provide useful solutions to their problems; however, there are some differences between them and the research questions in this article.First, the scenarios are different. Most existing schemes are studied in the context of SDN intradomain [3, 4, 7, 10, 11, 14–18, 20, 21], where policy control or policy enforcement can be performed uniformly by the intradomain controller; however, SDN interdomain policy forwarding control is in a distributed network environment where each domain independently formulates policies and thus faces cross-domain policy agnostic and policy conflict problems. As a result, in the process of data transmission, the transmission domain does not know the policy constraints of the source domain, so it cannot forward dataflow according to the policy constraints of the source domain; similarly, as the source domain does not know the filtering policies of the transmission domain for dataflow, data transmission may be interrupted owing to the policy conflicts between the transmission domain and the source domain.Second, the information basis for policy formulation is different. In SDN intradomain policy schemes, the intradomain global network state can be read from the controller to formulate address-oriented (IP, MAC, and switch port number) or identifier-oriented (the identifier of the switch, user, or middleware) policies [3, 4, 10, 11, 17]; however, in the SDN distributed multidomain environment, there is no central controller that holds interdomain global network state, so SDN cross-domain network information is opaque. This would make it impossible to directly develop address-oriented or identifier-oriented interdomain forwarding control policies for SDN interdomains.Third, the need for policy representation capabilities is different. Compared to the interdomain requirements for forwarding control of dataflows with certain attribute characteristics, the intradomain policies are described using addresses or identifiers and a fixed number of tuples [3, 4, 10, 12, 13, 17], which lack a higher level of abstraction and flexibility in policy expression. At the same time, most of the existing intradomain implementation schemes are oriented to ACL policy or service function chain policy [10–13, 17, 21]. Since ACL policy can only be used to allow or deny dataflows and cannot constrain the dataflow forwarding paths on the whole, the service function chain can only control the middleware sequences through which the dataflows pass, which is not specifically path-oriented forwarding control. Therefore, both policies cannot meet the requirements of path-oriented forwarding control.Finally, there are a few policy-oriented SDN interdomain forwarding control schemes. [5] is a security policy architecture for SDN intradomain and interdomain. Interdomain policy forwarding control enables cross-domain policy delivery on the data plane by passing policy tokens through packets; however, it does not discuss the lack of security and trustworthiness of cross-domain policy delivery and interdomain policy conflicts.In response to the previous requirements and challenges, blockchain and attribute-based access control inspire us. As a distributed ledger technology that deeply integrates technologies such as P2P networks, cryptography, smart contracts, and consensus mechanisms, blockchain can provide automatic and transparent transaction execution as well as secure and trusted data-sharing services in a distributed and untrusted environment [22]. Therefore, a secure and trusted cross-domain policy sharing can be provided for SDN interdomain forwarding control based on blockchain. In the access control field, attribute-based access control is suitable for open, massive, and dynamic scenarios, and attribute-oriented policies can flexibly express rich semantics in a unified manner [23]. For example, the identifier, address, location, role, and security level can be described uniformly as attributes. Therefore, higher-level policy expression can be provided for fine-grained forwarding control between SDN domains by defining dataflow and path-oriented policies.Currently, blockchain technology has important applications in the field of networking and information security. For example, [24] provides a systematic discussion of blockchain applications and challenges in IoT, [25] gives a detailed analysis of the performance evaluation of a blockchain-based security and privacy protection system, and [26] is a secure energy exchange scheme based on deep learning and blockchain. Specifically, there are some security schemes that apply blockchain to SDN, such as blockchain-based secure routing for multidomain SDN [27], blockchain-based multicontroller deployment [28], and blockchain-based SDN flow rule verification [29]. Moreover, there are some schemes that apply blockchain to attribute-based access control [30, 31]. These schemes are mainly designed to achieve access control of data resources rather than control of the forwarding path of dataflows. Compared to access control, forwarding control has a higher frequency of policy requests and greater latency sensitivity. Therefore, in blockchain-based forwarding control, the architecture and functional model need to be reasonably designed to balance the requirements of security and latency sensitivity.In summary, this article proposes a blockchain-based and policy-oriented forwarding control scheme for multidomain SDN. The scheme defines a forwarding control policy paradigm based on the attributes of dataflows and paths, enables secure and trusted cross-domain network information and policy sharing in the distributed SDN interdomain environment, and finally realizes a fine-grained control for the dataflow forwarding paths through global collaborative policies. The specific contributions of this article are as follows:(1)
We define a higher-level abstraction policy paradigm based on the attributes of dataflow and forwarding path, which can control the dataflow forwarding path in a fine-grained manner and provide a unified, flexible, and enriched expression capability.(2)
We realize domain information and policy sharing based on blockchain to solve the problem of cross-domain policy unknowability and ensure the security and trustworthiness of policy delivery in the distributed SDN interdomain environment. In addition, we innovatively implement cross-domain forwarding control based on globally collaborating policies to eliminate the problem of cross-domain policy conflicts.(3)
We propose a physically centralized and logically isolated architecture that deploys blockchain node functions on the local controller, which enables the SDN to exchange data with blockchain safely and efficiently.(4)
We design an on-chain and off-chain combination function model that separates forwarding control from blockchain and executes policy collaboration on the local controller, which achieves policy collaboration based on trusted data and keeps blockchain from incurring excessive latency and complex calculation to dataflow forwarding.The rest of this article is organized as follows: Section2 gives a comprehensive review of related work. Then, Section 3 presents the theoretical framework of BPFC-SDNs. Section 4 defines the policy paradigm for BPFC-SDNs. Section 5 designs the architecture, functional model, and work process of BPFC-SDNs. Section 6 introduces the transaction structure, contracts, and policy collaboration algorithms. Finally, the effectiveness and performance evaluation are reported in Sections 7 and 8 gives the summary.


## 2. Related Work

This section overviews the related efforts in policy-oriented SDN forwarding control, policy implementation in SDN, blockchain-based data-sharing and access control, and the combined application of blockchain and the SDN.


### 2.1. Policy-Oriented SDN Forwarding Control

In recent years, policy-oriented SDN forwarding control methods have been widely recognized. As the prototype of SDN, ethane [3] controls the network by a policy declared over high-level names to determine the packet forwarding path and eliminate attacks caused by IP address spoofing. OpenSec [4] employs OpenFlow matching fields to describe dataflows, thus enabling policy-oriented forwarding control. B4 [6] achieves traffic scheduling by dynamically adjusting policies based on intelligent algorithms. Bohatei [7] proposed a policy-oriented defense system in multiple data centers controlled by a single ISP. The previously mentioned policy-oriented management or control schemes are within an SDN domain or multiple SDN domains managed by a single organization. However, the issue of policy-oriented forwarding control among SDN domains belonging to different organizations has not been involved in the previous research study. In 2019, Varadharajan et al. [5] proposed a policy-oriented SDN security architecture called PbSA, which extends the policy-oriented forwarding control to the cross-organizational SDN interdomain scenario. To solve the cross-domain policy agnostic problem, PbSA implements the cross-domain policy delivery via policy transfer tokens. Nevertheless, there are still three main problems in the current SDN interdomain policy-oriented forwarding control mechanism. First, the lack of security and credibility of cross-domain policy transmission has not been taken into consideration. Second, policy conflicts will lead to flow transmission being interrupted. Third, delivering the policy transfer tokens with packets and performing policy routing computation domain by domain reduce the data plane forwarding rate and transmission efficiency. Therefore, in the SDN interdomain scenario, how to implement secure, trusted, and efficient policy forwarding control requires further research.


### 2.2. Policy Implementation in SDN

Currently, for the SDN policy implementation, researchers have proposed helpful solutions corresponding to specific problems. (1) Network programming languages have been developed to provide convenience for network policy creation, development, and deployment. For example, Frenetic [8] and Pyretic [9]. However, Pyretic represents only actions such as drop, passthrough, forward by a specified port, flood, and modify the value of a header field and cannot describe that a dataflow should be forwarded along the path with certain attributes. (2) Policy expression and policy conflict detection and resolution. [10] represents ACL policies based on 3D shape, which is not suitable for representing attribute-based interdomain policies because it can only represent three policy constraint tuples. [11] expresses ACL policies and service function chain policies based on a policy graph abstraction (PGA). Since the vertices of the graph are described by membership predicates, each vertex corresponds to a set of network entities that do not overlap with each other. While SDN interdomain policies are based on attributes, if attribute predicates are used as vertices of the graph, the network entities corresponding to different vertices will be overlapped and still cannot achieve policy composition. (3) Policy optimization deployment. [12–15] investigate which switches the ACL policy should be deployed to and how to deploy the ACL policy to reduce the number of flow rules on the switch and avoid the conflicts of flow rules on the switch. (4) Inconsistency in policy implementation. [16] uses a state-transition diagram and fuzz-testing to detect whether there are inconsistencies in the process of transforming ACL policies into flow rules by different components. [17] automatically installs flow rules when the network policy is changed to ensure the consistency of flow rules with the ACL policy. (5) Policy implementation schemes designed to adapt to the dynamic change of the link state. [20] generates paths for data flows that satisfy the policy by automatically retagging switches on the path. [21] avoids ACL policy violations caused by link failures by using machine learning algorithms to predict link failures and then proactively calculating the ACL policy installation location. Based on the previous analysis, to achieve fine-grained SDN interdomain data flow forwarding path control, it is required that a specially designed policy expression paradigm and a policy synthesis method for the policy paradigm.


### 2.3. Blockchain-Based Access Control and Data Sharing

At present, blockchain technology is widely used in data sharing and access control. Liu et al. [32] proposed a blockchain-based big data access control mechanism that ensures the immutability, auditability, and verifiability of access control policies via transactions and realizes automatic and transparent access control using smart contracts. Shi and Li [33] reviewed the access control schemes in the IoT environment and pointed out that owing to the lack of trust in cross-organization or cross-domain access control, the blockchain can act as an open and trusted platform. The works [34, 35] both proposed a blockchain-based data-sharing mechanism, which applies blockchain to record access policy so as to provide a secure and verifiable data-sharing scheme. Inspired by the research, blockchain can provide services such as secure and trusted data sharing or automatic and transparent execution for policy-oriented forwarding control among SDN domains.However, it is worth noting that the research purposes and objectives of forwarding control and access control are different because forwarding control is more sensitive to latency and has higher request frequency than access control. Therefore, how to reasonably apply blockchain to SDN interdomain policy-oriented forwarding control requires further research so as to minimize the impact on data forwarding.


### 2.4. Combination Applications of Blockchain and SDN

Recently, the combined application of blockchain and SDN has attracted much attention in academia and the industry. In the research on the combination of blockchain and SDN, the architectures have varied owing to different research goals. Here, the architectures are divided into three categories as follows.First, blockchain and the SDN are deployed independently and communicate remotely. For example, Boussard et al. [36] proposed a home IoT security architecture based on the SDN and blockchain, in which the blockchain acts as a distributed, trusted, and verifiable database. Hu et al. [37] proposed a blockchain-based SDN flow rule verification scheme, which equips each forwarding node in the SDN with a blockchain node, and the controller communicates with the blockchain node through TLS. The advantage of this kind of architecture is that the functions are clearly divided, and it is easy to manage and maintain. However, owing to the independent deployment, the remote communication overhead is relatively large. Moreover, in order to ensure the security of the remote communication between the SDN and blockchain nodes, a security mechanism needs to be introduced, which further increases computation overhead and latency, so it is not suitable for latency-sensitive applications.Second, blockchain nodes communicate via the SDN network infrastructure; that is, data forwarding between blockchain nodes is centrally managed and controlled by the SDN controller. For example, in the representative works of energy transactions based on blockchain and the SDN on the Internet of Vehicles [38, 39], energy transaction forwarding depends on the SDN data plane; if the controller is attacked, the overall security and credibility of the blockchain network are directly threatened. In sum, if blockchain uses the SDN as the network infrastructure, the centralized management and control of SDN pose security risks to the data forwarding between blockchain nodes and affect its distributed and decentralized trust mechanism.Third, the blockchain node function is deployed on the SDN control plane. In 2019, Gao et al. [40] proposed an Internet of Vehicles’ framework based on blockchain and the SDN, deploying blockchain on the SDN control layer to provide vehicle authentication and trusted management services. In 2021, Ren et al. [41] applied blockchain and attribute-based encryption to solve the cross-domain access control of SDN controller resources by IoT applications. This kind of architecture is suitable for resolving the problems in the SDN interdomain scenario. However, these schemes are focused on authentication and access control; concerning forwarding control, specific analysis is required for specific issues.Therefore, in order to achieve policy-oriented forwarding control between SDN domains based on blockchain, further research is needed.


## 2.1. Policy-Oriented SDN Forwarding Control

In recent years, policy-oriented SDN forwarding control methods have been widely recognized. As the prototype of SDN, ethane [3] controls the network by a policy declared over high-level names to determine the packet forwarding path and eliminate attacks caused by IP address spoofing. OpenSec [4] employs OpenFlow matching fields to describe dataflows, thus enabling policy-oriented forwarding control. B4 [6] achieves traffic scheduling by dynamically adjusting policies based on intelligent algorithms. Bohatei [7] proposed a policy-oriented defense system in multiple data centers controlled by a single ISP. The previously mentioned policy-oriented management or control schemes are within an SDN domain or multiple SDN domains managed by a single organization. However, the issue of policy-oriented forwarding control among SDN domains belonging to different organizations has not been involved in the previous research study. In 2019, Varadharajan et al. [5] proposed a policy-oriented SDN security architecture called PbSA, which extends the policy-oriented forwarding control to the cross-organizational SDN interdomain scenario. To solve the cross-domain policy agnostic problem, PbSA implements the cross-domain policy delivery via policy transfer tokens. Nevertheless, there are still three main problems in the current SDN interdomain policy-oriented forwarding control mechanism. First, the lack of security and credibility of cross-domain policy transmission has not been taken into consideration. Second, policy conflicts will lead to flow transmission being interrupted. Third, delivering the policy transfer tokens with packets and performing policy routing computation domain by domain reduce the data plane forwarding rate and transmission efficiency. Therefore, in the SDN interdomain scenario, how to implement secure, trusted, and efficient policy forwarding control requires further research.


## 2.2. Policy Implementation in SDN

Currently, for the SDN policy implementation, researchers have proposed helpful solutions corresponding to specific problems. (1) Network programming languages have been developed to provide convenience for network policy creation, development, and deployment. For example, Frenetic [8] and Pyretic [9]. However, Pyretic represents only actions such as drop, passthrough, forward by a specified port, flood, and modify the value of a header field and cannot describe that a dataflow should be forwarded along the path with certain attributes. (2) Policy expression and policy conflict detection and resolution. [10] represents ACL policies based on 3D shape, which is not suitable for representing attribute-based interdomain policies because it can only represent three policy constraint tuples. [11] expresses ACL policies and service function chain policies based on a policy graph abstraction (PGA). Since the vertices of the graph are described by membership predicates, each vertex corresponds to a set of network entities that do not overlap with each other. While SDN interdomain policies are based on attributes, if attribute predicates are used as vertices of the graph, the network entities corresponding to different vertices will be overlapped and still cannot achieve policy composition. (3) Policy optimization deployment. [12–15] investigate which switches the ACL policy should be deployed to and how to deploy the ACL policy to reduce the number of flow rules on the switch and avoid the conflicts of flow rules on the switch. (4) Inconsistency in policy implementation. [16] uses a state-transition diagram and fuzz-testing to detect whether there are inconsistencies in the process of transforming ACL policies into flow rules by different components. [17] automatically installs flow rules when the network policy is changed to ensure the consistency of flow rules with the ACL policy. (5) Policy implementation schemes designed to adapt to the dynamic change of the link state. [20] generates paths for data flows that satisfy the policy by automatically retagging switches on the path. [21] avoids ACL policy violations caused by link failures by using machine learning algorithms to predict link failures and then proactively calculating the ACL policy installation location. Based on the previous analysis, to achieve fine-grained SDN interdomain data flow forwarding path control, it is required that a specially designed policy expression paradigm and a policy synthesis method for the policy paradigm.


## 2.3. Blockchain-Based Access Control and Data Sharing

At present, blockchain technology is widely used in data sharing and access control. Liu et al. [32] proposed a blockchain-based big data access control mechanism that ensures the immutability, auditability, and verifiability of access control policies via transactions and realizes automatic and transparent access control using smart contracts. Shi and Li [33] reviewed the access control schemes in the IoT environment and pointed out that owing to the lack of trust in cross-organization or cross-domain access control, the blockchain can act as an open and trusted platform. The works [34, 35] both proposed a blockchain-based data-sharing mechanism, which applies blockchain to record access policy so as to provide a secure and verifiable data-sharing scheme. Inspired by the research, blockchain can provide services such as secure and trusted data sharing or automatic and transparent execution for policy-oriented forwarding control among SDN domains.However, it is worth noting that the research purposes and objectives of forwarding control and access control are different because forwarding control is more sensitive to latency and has higher request frequency than access control. Therefore, how to reasonably apply blockchain to SDN interdomain policy-oriented forwarding control requires further research so as to minimize the impact on data forwarding.


## 2.4. Combination Applications of Blockchain and SDN

Recently, the combined application of blockchain and SDN has attracted much attention in academia and the industry. In the research on the combination of blockchain and SDN, the architectures have varied owing to different research goals. Here, the architectures are divided into three categories as follows.First, blockchain and the SDN are deployed independently and communicate remotely. For example, Boussard et al. [36] proposed a home IoT security architecture based on the SDN and blockchain, in which the blockchain acts as a distributed, trusted, and verifiable database. Hu et al. [37] proposed a blockchain-based SDN flow rule verification scheme, which equips each forwarding node in the SDN with a blockchain node, and the controller communicates with the blockchain node through TLS. The advantage of this kind of architecture is that the functions are clearly divided, and it is easy to manage and maintain. However, owing to the independent deployment, the remote communication overhead is relatively large. Moreover, in order to ensure the security of the remote communication between the SDN and blockchain nodes, a security mechanism needs to be introduced, which further increases computation overhead and latency, so it is not suitable for latency-sensitive applications.Second, blockchain nodes communicate via the SDN network infrastructure; that is, data forwarding between blockchain nodes is centrally managed and controlled by the SDN controller. For example, in the representative works of energy transactions based on blockchain and the SDN on the Internet of Vehicles [38, 39], energy transaction forwarding depends on the SDN data plane; if the controller is attacked, the overall security and credibility of the blockchain network are directly threatened. In sum, if blockchain uses the SDN as the network infrastructure, the centralized management and control of SDN pose security risks to the data forwarding between blockchain nodes and affect its distributed and decentralized trust mechanism.Third, the blockchain node function is deployed on the SDN control plane. In 2019, Gao et al. [40] proposed an Internet of Vehicles’ framework based on blockchain and the SDN, deploying blockchain on the SDN control layer to provide vehicle authentication and trusted management services. In 2021, Ren et al. [41] applied blockchain and attribute-based encryption to solve the cross-domain access control of SDN controller resources by IoT applications. This kind of architecture is suitable for resolving the problems in the SDN interdomain scenario. However, these schemes are focused on authentication and access control; concerning forwarding control, specific analysis is required for specific issues.Therefore, in order to achieve policy-oriented forwarding control between SDN domains based on blockchain, further research is needed.


## 3. Theoretical Framework of BPFC-SDNs

In order to achieve fine-grained forwarding control for dataflow forwarding path in multidomain SDN, BPFC-SDNs are proposed, which is inspired by blockchain-based data sharing and attribute-based access control. The theoretical framework of BPFC-SDNs is shown in Figure1, including four core parts: domain information management, policy management, blockchain-based data sharing, and policy collaboration, which are extended SDN controller application functions.(1)
Domain Information Management. In order to provide global domain information for policy formulation and collaboration, domain information is registered with the domain information management function in each SDN domain and uploaded to the blockchain by transactions for sharing.(2)
Policy Management. To achieve fine-grained SDN interdomain dataflow forwarding control and to provide global policies for policy collaboration, the policy management function of each SDN domain creates SDN interdomain forwarding control policies based on the attributes of dataflow and path according to its requirements and uploads them to the blockchain by transactions for sharing.(3)
Blockchain-Based Data Sharing. Secure and trusted domain information and policy sharing are achieved through smart contracts and consensus mechanisms. In order to balance the requirements of system security and latency sensitivity, BPFC-SDNs do not put forwarding control on the chain but make the blockchain distribute global domain information and policies to each SDN domain; thus, each SDN domain performs forwarding control based on global data locally.(4)
Policy Collaboration. In order to avoid interruptions in data transmission due to cross-domain policy conflicts, the policy collaboration function synthesizes policies before dataflows are forwarded. The main process is that when an SDN domain generates a new dataflow, its controller uploads the key header fields of the first packet of the new dataflow to the policy collaboration function module; the policy collaboration function queries the global domain information based on the key fields and maps to the attributes of the dataflow; then, the policy collaboration function queries the global policies based on the attributes of the dataflow and obtains the matched policy set of the dataflow; finally, the path synthesis is performed on the matched policy set to get the global collaborative policy for this dataflow.Figure 1
The theoretical framework of BPFC-SDNs.For a complete description of the SDN interdomain policy-based forwarding control process, the framework also includes the following: initially, each domain registers information about the network entity attributes of its own domain with the extended application; and finally, the controller computes interdomain forwarding paths for the dataflow based on the global collaborative policy and the interdomain global virtual topology [42] and performs interdomain dataflow forwarding. But these are not the innovative tasks accomplished by BPFC-SDNs. In the next Sections 4–6, the attribute-based forwarding control policy paradigm, the system design of BPFC-SDNs, and the transactions, contracts, and algorithms in BPFC-SDNs are described in detail, respectively.


## 4. Policy Paradigm Based on the Attributes of Dataflow and Path

To realize dataflow and path-oriented forwarding control among SDN multidomains, a forwarding control policy paradigm based on the attributes of dataflow and path is put forward. In the following section, it will be elaborated on three aspects: related definitions, the policy paradigm, and policy examples.


### 4.1. Related Definitions

Definition 1 (Attribute).
Attribute refers to a variable with a certain data type and value range, which can be abstracted as attr,Value. Here, attr represents the attribute name, and Value represents the value range of the attribute, namely, Value=value1,value2,…,valuen.Definition 2 (Attribute Name-Value Pair (avp)).
avp represents the specific value of an attribute, which can be abstracted into two tuples of attr,value, indicating attr=value. A collection of attribute name-value pairs can be represented by AVP.
In BPFC-SDNs, domain information includes the attributes of the domain, users, devices, and network services. The attributes of a domain, a user, a device, or a kind of network service are described by a set of attribute name-value pairs.
For example,domAVP = {<domID, 001>, <dom_net, 202.20.2.0/24>, <dom_type, EDU>, <dom_sl, 3>, and <dom_tl = 2>} indicates that the subnet address of domain “001” is “202.20.2.0/24,” which belongs to the “education” type of organization. The security level of this domain is “3,” and the trust level is “2.”userAVP = {<userID, 00001>, <user_org, orgnationA>, <user_depart, depatmentB>, <user_type, manager>, and <user_tl, 2>} indicates that the user whose user ID is “00001” is from “departmentB” of “organizationA,” whose role is “manager,” and whose trust level is “2.”servAVP = {<servID, 0001>, <serv_type, videoconferencing> <ser_protocol, SIP>, and <ser_dport, 5060>} indicates that the type of network service with service ID “0001” is “videoconferencing,” the protocol used is “SIP,” and the corresponding port number is “5060.”Definition 3 (Dataflow Attributes (flowAVP)).
In BPFC-SDNs, the attributes of the dataflow are mapped based on the key header fields of the first packet of the dataflow and the global domain information. Thus, a dataflow can be described by the AVP set of the source domain, source user, source device, destination domain, destination users, destination devices, and services of the dataflow, which is as follows:(1)flowAVP=srcdomAVP,srcuserAVP,srcdevAVP,dstdomAVP,dstuserAVP,dstdevAVP,servAVP.
The source attribute of a dataflow can be composed of theAVP of the source domain, source user, and source device, which can be formally expressed as follows:(2)srcAVP=srcdomAVP,srcuserAVP,srcdevAVP.
Similarly, the destination attribute of a dataflow can be formally expressed as follows:(3)dstAVP=dstdomAVP,dstuserAVP,dstdevAVP.
Therefore, a dataflow can be represented as follows:(4)flowAVP=srcAVP,dstAVP,servAVP.Definition 4 (Attribute Predicate (ap)).
ap is used to limit the value range of an attribute, and ap can be abstracted into a triple attr,∝,value, ∝∈=,≠,<,≤,>,≥,in,not in,between.Definition 5 (Attribute Predicate Evaluation).
The attribute predicate evaluation means that for a given attribute name-value pairavp and attribute predicate ap, the evaluation result of ap to avp is true if and only if both of the attribute names are the same and the attribute value of avp belongs to the value range of ap, which can be formally expressed as(5)avp⊲ap=true⇔avp.attr=ap.attr∧avp.value∝ap.value.


### 4.2. Policy Paradigm

The need for the SDN interdomain dataflow forwarding path control can be specified as three objectives: (1) the source domain constrains the dataflow transmission path; (2) the transmission domain or destination domain filters the incoming traffic; and (3) path constraints and traffic filtering realize the forwarding control of the dataflow. To express the previous forwarding control objectives, the following policy paradigm is defined.Definition 6.
Policy paradigm based on the attributes of dataflow and path. The forwarding control policy paradigm oriented to dataflow and path attributes is defined as follows:(6)policy=domID,policyID,policyType,SrcAP,DstAP,ServAP,Path,action.
The meaning of the notations ofpolicy is as follows:domID: indicating the identifier of the domain configuring this policy.policyID: indicating the identifier of this policy in this domain. The combination of domID and policyID can uniquely specify a policy in a multidomain environment.policyType: indicating the policy type. Policies in BPFC-SDNs are divided into two categories. policyType = SOURCE represents that the policy is used to constrain the dataflow from this domain, while policyType = TRANSFER represents that the policy is used to filter the traffic coming from other domains.SrcAP: representing the source attribute constraints on the dataflow, which is composed of the source domain attribute predicates, the source user attribute predicates, and the source device attribute predicates through logical conjunction, namely,(7)SrcAP=srcdomAP∧srcuserAP∧srcdevAP.DstAP: representing the destination attribute constraints on the dataflow, which is composed of the destination domain attribute predicates, the destination user attribute predicates, and the destination device attribute predicates through logical conjunction, namely,(8)DstAP=dstdomAP∧dstuserAP∧dstdevAP.ServAP: representing the service attribute constraints on the dataflow, which is formed by the logical conjunction of the attribute predicates such as the protocol, port number, or self-defined service type. It is expressed as follows:(9)ServAP=servap1∧servap2,…,∧servaps.Path: indicating the path constraint on the dataflow, which is formed by the conjunction of attribute predicates such as the security level, trust level, or domain identifier of the transmission domain. Path is used by the source domain to limit the security level or trust level of the transmission domains that may carry the dataflow or to limit some specific domains that the dataflow must or cannot pass through. It is expressed as follows:(10)Path=pathap1∧pathap2∧…∧pathapp.action: representing the forwarding action on the policy-constrained path for the dataflow that satisfies this policy constraint, including drop and forward.
The combination of path constraints and actions can determine the forwarding path decision of a dataflow, as shown in Table1. This article makes an appointment that the source domain can only limit the forwarding path of the sourcing dataflow, as shown in lines 1–4, and the transmission domains only have the right to determine whether to forward or discard the dataflow but cannot restrict the interdomain forwarding path of the dataflow coming from other domains, as shown in lines 5–6.Table 1
Forwarding control implications of the policy.
No.Policy typePathActionForwarding control implicationDeveloped by1SOURCESL ≥ 3forwardDataflows matching this policy can only be forwarded by domains with a security level greater than or equal to 3Source domain2SOURCETL ≤ 2dropDataflows matching this policy are not allowed to be forwarded by domains with a trust level less than or equal to 23SOURCEAS3forwardDataflows matching this policy should be forwarded by the AS3 domain4SOURCEAS2dropDataflows matching this policy cannot be forwarded through the AS2 domain5TRANSFERNullforwardThis domain receives and forwards dataflows that match this policyTransmission domain or destination domain6TRANSFERNulldropThis domain refuses to receive dataflows that match this policyDefinition 7 (Policy Matching).
Policy matching means that given a dataflow and a policy, the dataflow matches the policy if and only if for every attribute predicate in thepolicy, the value of the corresponding attribute name-value pair of the dataflow belongs to the value range of the attribute predicate, which can be formally expressed as follows:(11)flowAVP⊲policy=true⇔for∀apj∈policy.AP,∃avpi∈flowAVP.AVP,avpi.attr=apj.attr∧avpi.value∝apj.value.


### 4.3. Policy Examples

Taking the SDN multidomain environment shown in Figure2 as an example, the forwarding control policy based on the attributes of dataflow and path is explained as follows: the tables in Figure 2 provide part of the attribute information of the corresponding domain or device.(12)policyAS11=AS1,1,SOURCE,AS1,∗,device_i,AS5,∗,∗,20,21,SL≥3,forward.Figure 2
A multidomain network with policy-oriented forwarding control.This is the first policy formulated by the AS1 domain: the FTP traffic from the device_i in the AS1 domain to the AS5 domain must be transmitted by the domains with a security level greater than or equal to 3.Function analysis: the source domain realizes the path selection of the dataflow by specifying the security level of the transmission domain.(13)policyAS12=AS1,2,SOURCE,AS1,∗,∗,AS5,∗,∗,80,433,AS3,forward.This is the second policy formulated by the AS1 domain: HTTP and HTTPS traffic from the AS1 domain to the AS5 domain should be forwarded through the AS3 domain.Function analysis: the source domain realizes the path selection of the dataflow by specifying the domain identifier of the transmission domain that must be passed through.(14)policyAS33=AS3,3,TRANSFER,AS2,∗,∗,∗,∗,∗,SMTP,∗,drop.This is the third policy established by the AS3 domain: the AS3 domain refuses to carry SMTP traffic originating from the AS2 domain.Functional analysis: the transmission domain has the right to choose the dataflows it carries.(15)policyAS44=AS4,4,TRANSFER,∗,∗,dev_sl≤3,∗,∗,∗,∗,∗,drop.This is the fourth policy formulated by the AS4 domain: the AS4 domain refuses to carry dataflows originating from devices with a security level less than or equal to 3.Function analysis: the transmission domain implements traffic filtering by limiting the source attributes of a dataflow.(16)policyAS55=AS5,5,SOURCE,∗,user_tl≥2,∗,AS5,∗,∗,∗,∗,forward.This is the fifth policy formulated by the AS5 domain: the AS5 domain allows the trust level of the user greater than or equal to 2 to access it.Function analysis: the destination domain can realize forwarding or filtering of the dataflow through the attributes of the source user.It can be seen from the policy examples that the management domain formulates forwarding control policies according to its service requirements and implements path selection for dataflows through the attribute constraints of dataflows and paths. In conclusion, the BPFC-SDNs’ forwarding control policy based on the attributes of dataflow and path meets the evolution of the network forwarding control requirements, from simply reachability-oriented forwarding control requirements to service-oriented fine-grained canalization forwarding control requirements. Furthermore, according to the path constraints on different security levels or trust levels, the BPFC-SDNs’ forwarding control policy can realize the physical isolation of the dataflow forwarding path and improve the transmission security of the important cross-domain dataflow.


## 4.1. Related Definitions

Definition 1 (Attribute).
Attribute refers to a variable with a certain data type and value range, which can be abstracted as attr,Value. Here, attr represents the attribute name, and Value represents the value range of the attribute, namely, Value=value1,value2,…,valuen.Definition 2 (Attribute Name-Value Pair (avp)).
avp represents the specific value of an attribute, which can be abstracted into two tuples of attr,value, indicating attr=value. A collection of attribute name-value pairs can be represented by AVP.
In BPFC-SDNs, domain information includes the attributes of the domain, users, devices, and network services. The attributes of a domain, a user, a device, or a kind of network service are described by a set of attribute name-value pairs.
For example,domAVP = {<domID, 001>, <dom_net, 202.20.2.0/24>, <dom_type, EDU>, <dom_sl, 3>, and <dom_tl = 2>} indicates that the subnet address of domain “001” is “202.20.2.0/24,” which belongs to the “education” type of organization. The security level of this domain is “3,” and the trust level is “2.”userAVP = {<userID, 00001>, <user_org, orgnationA>, <user_depart, depatmentB>, <user_type, manager>, and <user_tl, 2>} indicates that the user whose user ID is “00001” is from “departmentB” of “organizationA,” whose role is “manager,” and whose trust level is “2.”servAVP = {<servID, 0001>, <serv_type, videoconferencing> <ser_protocol, SIP>, and <ser_dport, 5060>} indicates that the type of network service with service ID “0001” is “videoconferencing,” the protocol used is “SIP,” and the corresponding port number is “5060.”Definition 3 (Dataflow Attributes (flowAVP)).
In BPFC-SDNs, the attributes of the dataflow are mapped based on the key header fields of the first packet of the dataflow and the global domain information. Thus, a dataflow can be described by the AVP set of the source domain, source user, source device, destination domain, destination users, destination devices, and services of the dataflow, which is as follows:(1)flowAVP=srcdomAVP,srcuserAVP,srcdevAVP,dstdomAVP,dstuserAVP,dstdevAVP,servAVP.
The source attribute of a dataflow can be composed of theAVP of the source domain, source user, and source device, which can be formally expressed as follows:(2)srcAVP=srcdomAVP,srcuserAVP,srcdevAVP.
Similarly, the destination attribute of a dataflow can be formally expressed as follows:(3)dstAVP=dstdomAVP,dstuserAVP,dstdevAVP.
Therefore, a dataflow can be represented as follows:(4)flowAVP=srcAVP,dstAVP,servAVP.Definition 4 (Attribute Predicate (ap)).
ap is used to limit the value range of an attribute, and ap can be abstracted into a triple attr,∝,value, ∝∈=,≠,<,≤,>,≥,in,not in,between.Definition 5 (Attribute Predicate Evaluation).
The attribute predicate evaluation means that for a given attribute name-value pairavp and attribute predicate ap, the evaluation result of ap to avp is true if and only if both of the attribute names are the same and the attribute value of avp belongs to the value range of ap, which can be formally expressed as(5)avp⊲ap=true⇔avp.attr=ap.attr∧avp.value∝ap.value.


## 4.2. Policy Paradigm

The need for the SDN interdomain dataflow forwarding path control can be specified as three objectives: (1) the source domain constrains the dataflow transmission path; (2) the transmission domain or destination domain filters the incoming traffic; and (3) path constraints and traffic filtering realize the forwarding control of the dataflow. To express the previous forwarding control objectives, the following policy paradigm is defined.Definition 6.
Policy paradigm based on the attributes of dataflow and path. The forwarding control policy paradigm oriented to dataflow and path attributes is defined as follows:(6)policy=domID,policyID,policyType,SrcAP,DstAP,ServAP,Path,action.
The meaning of the notations ofpolicy is as follows:domID: indicating the identifier of the domain configuring this policy.policyID: indicating the identifier of this policy in this domain. The combination of domID and policyID can uniquely specify a policy in a multidomain environment.policyType: indicating the policy type. Policies in BPFC-SDNs are divided into two categories. policyType = SOURCE represents that the policy is used to constrain the dataflow from this domain, while policyType = TRANSFER represents that the policy is used to filter the traffic coming from other domains.SrcAP: representing the source attribute constraints on the dataflow, which is composed of the source domain attribute predicates, the source user attribute predicates, and the source device attribute predicates through logical conjunction, namely,(7)SrcAP=srcdomAP∧srcuserAP∧srcdevAP.DstAP: representing the destination attribute constraints on the dataflow, which is composed of the destination domain attribute predicates, the destination user attribute predicates, and the destination device attribute predicates through logical conjunction, namely,(8)DstAP=dstdomAP∧dstuserAP∧dstdevAP.ServAP: representing the service attribute constraints on the dataflow, which is formed by the logical conjunction of the attribute predicates such as the protocol, port number, or self-defined service type. It is expressed as follows:(9)ServAP=servap1∧servap2,…,∧servaps.Path: indicating the path constraint on the dataflow, which is formed by the conjunction of attribute predicates such as the security level, trust level, or domain identifier of the transmission domain. Path is used by the source domain to limit the security level or trust level of the transmission domains that may carry the dataflow or to limit some specific domains that the dataflow must or cannot pass through. It is expressed as follows:(10)Path=pathap1∧pathap2∧…∧pathapp.action: representing the forwarding action on the policy-constrained path for the dataflow that satisfies this policy constraint, including drop and forward.
The combination of path constraints and actions can determine the forwarding path decision of a dataflow, as shown in Table1. This article makes an appointment that the source domain can only limit the forwarding path of the sourcing dataflow, as shown in lines 1–4, and the transmission domains only have the right to determine whether to forward or discard the dataflow but cannot restrict the interdomain forwarding path of the dataflow coming from other domains, as shown in lines 5–6.Table 1
Forwarding control implications of the policy.
No.Policy typePathActionForwarding control implicationDeveloped by1SOURCESL ≥ 3forwardDataflows matching this policy can only be forwarded by domains with a security level greater than or equal to 3Source domain2SOURCETL ≤ 2dropDataflows matching this policy are not allowed to be forwarded by domains with a trust level less than or equal to 23SOURCEAS3forwardDataflows matching this policy should be forwarded by the AS3 domain4SOURCEAS2dropDataflows matching this policy cannot be forwarded through the AS2 domain5TRANSFERNullforwardThis domain receives and forwards dataflows that match this policyTransmission domain or destination domain6TRANSFERNulldropThis domain refuses to receive dataflows that match this policyDefinition 7 (Policy Matching).
Policy matching means that given a dataflow and a policy, the dataflow matches the policy if and only if for every attribute predicate in thepolicy, the value of the corresponding attribute name-value pair of the dataflow belongs to the value range of the attribute predicate, which can be formally expressed as follows:(11)flowAVP⊲policy=true⇔for∀apj∈policy.AP,∃avpi∈flowAVP.AVP,avpi.attr=apj.attr∧avpi.value∝apj.value.


## 4.3. Policy Examples

Taking the SDN multidomain environment shown in Figure2 as an example, the forwarding control policy based on the attributes of dataflow and path is explained as follows: the tables in Figure 2 provide part of the attribute information of the corresponding domain or device.(12)policyAS11=AS1,1,SOURCE,AS1,∗,device_i,AS5,∗,∗,20,21,SL≥3,forward.Figure 2
A multidomain network with policy-oriented forwarding control.This is the first policy formulated by the AS1 domain: the FTP traffic from the device_i in the AS1 domain to the AS5 domain must be transmitted by the domains with a security level greater than or equal to 3.Function analysis: the source domain realizes the path selection of the dataflow by specifying the security level of the transmission domain.(13)policyAS12=AS1,2,SOURCE,AS1,∗,∗,AS5,∗,∗,80,433,AS3,forward.This is the second policy formulated by the AS1 domain: HTTP and HTTPS traffic from the AS1 domain to the AS5 domain should be forwarded through the AS3 domain.Function analysis: the source domain realizes the path selection of the dataflow by specifying the domain identifier of the transmission domain that must be passed through.(14)policyAS33=AS3,3,TRANSFER,AS2,∗,∗,∗,∗,∗,SMTP,∗,drop.This is the third policy established by the AS3 domain: the AS3 domain refuses to carry SMTP traffic originating from the AS2 domain.Functional analysis: the transmission domain has the right to choose the dataflows it carries.(15)policyAS44=AS4,4,TRANSFER,∗,∗,dev_sl≤3,∗,∗,∗,∗,∗,drop.This is the fourth policy formulated by the AS4 domain: the AS4 domain refuses to carry dataflows originating from devices with a security level less than or equal to 3.Function analysis: the transmission domain implements traffic filtering by limiting the source attributes of a dataflow.(16)policyAS55=AS5,5,SOURCE,∗,user_tl≥2,∗,AS5,∗,∗,∗,∗,forward.This is the fifth policy formulated by the AS5 domain: the AS5 domain allows the trust level of the user greater than or equal to 2 to access it.Function analysis: the destination domain can realize forwarding or filtering of the dataflow through the attributes of the source user.It can be seen from the policy examples that the management domain formulates forwarding control policies according to its service requirements and implements path selection for dataflows through the attribute constraints of dataflows and paths. In conclusion, the BPFC-SDNs’ forwarding control policy based on the attributes of dataflow and path meets the evolution of the network forwarding control requirements, from simply reachability-oriented forwarding control requirements to service-oriented fine-grained canalization forwarding control requirements. Furthermore, according to the path constraints on different security levels or trust levels, the BPFC-SDNs’ forwarding control policy can realize the physical isolation of the dataflow forwarding path and improve the transmission security of the important cross-domain dataflow.


## 5. System Design of BPFC-SDNs

In order to tackle the challenges of cross-domain policy agnostic and policy conflict, the distributed and untrusted interdomain communication environment, and forwarding control latency-sensitivity faced by SDN interdomain policy forwarding control, the architecture and functional model of BPFC-SDNs are proposed. In addition, the workflow of BPFC-SDNs is explained in this section.


### 5.1. The Architecture of BPFC-SDNs

The architecture of the BPFC-SDNs is shown in Figure3. In the multidomain SDN scenario, the control plane of each domain consists of four models, including the original SDN controller, BPFC application, blockchain application, and WE-bridge.Figure 3
The architecture of BPFC-SDNs.The architecture of BPFC-SDNs mainly includes the following six key points:(1)
In each SDN domain, we deploy the blockchain application and the original SDN controller on the same local server, and the blockchain-based policy forwarding control application (BPFC APP) interacts with the original SDN controller and the blockchain through the local application program interface.(2)
The blockchain application in each domain serves as the accounting node of the blockchain and forms a blockchain network through P2P. Therefore, each SDN domain stores a blockchain global ledger.(3)
By synchronizing the blockchain contract data to the local cache of the SDN control plane, the BPFC APP can quickly read the global trusted data from the local cache.(4)
Because the realization of forwarding control requires network topology, according to the literature [42], the WE-bridge located in the SDN network system layer of each domain is connected in a full mesh manner, based on the SSL protocol, and through the JSON file to securely and quickly exchange the virtual network views among domains. As a result, each domain can acquire a global network view over multiple domains based on the WE-bridge. Since this is not the focus of this article, it is assumed that each SDN controller in BPFC-SDNs can dynamically obtain the global network view over multidomains in real time through the WE-bridge component.(5)
To ensure the normal transmission of data within the SDN domain, it is assumed that the interactions within the control plane are secure and trusted and that the communication between the controller plane and the data plane is secure and trusted.(6)
To improve the robustness of the system and solve the single point of failure problem, multiple control nodes can be deployed in the same domain based on distributed caches, such as Redis [43], to implement active-standby or even multiactive.This deployment method of the physical concentration and logical isolation of blockchain and the SDN controller has three advantages:(1)
Compared with the physical location separate deployment, it can provide secure and high-speed information interaction and contract data synchronization, avoiding high transmission overhead and insecure network environment of remote communication.(2)
Compared with the high coupling of the controller function and the blockchain function, it can reduce the complexity of the system, and it is easy to implement and maintain.(3)
At the same time, the method deploying blockchain function nodes on the local controller, hence making all domains complete function execution together, better conforms to the essential property of distributed multiparty comanagement of the blockchain than the physical separate deployment.


### 5.2. The Function Model of BPFC-SDNs

Based on the architecture, a blockchain-based policy-oriented SDN interdomain forwarding control function model is proposed, as shown in Figure4. BPCF-SDNs ensure the authentication and integrity of domain information and policy information uploaded by each SDN domain based on the transaction signature mechanism. It realizes automatic and transparent domain information and policy release and updates through smart contract technology. It guarantees that the current block stores the latest state of global domain information and policies based on the account model, avoiding the UTXO (unspent transaction output) [44] model traversing the entire blockchain to obtain the latest state and improving the data retrieval rate. It achieves the credibility of cross-domain data sharing effectively based on the PBFT (practical Byzantine fault tolerance) [45] consensus mechanism. Finally, it ensures that the domain information and policies shared are tamper-proof and traceable based on the blockchain storage structure. In sum, the blockchain in BPFC-SDNs can provide a secure and trusted global data basis for policy-oriented SDN interdomain forwarding control.Figure 4
The function model of BPFC-SDNs.The functional model of BPFC-SDNs mainly includes the following parts:(1)
BPFC Application. It includes the domain information manager, policy manager, and policy collaboration engine. The domain information manager is responsible for registering the attribute information of domains, users, and devices, updating the mapping relationship of IP addresses to users and devices, and generating domain information transactions (DomInfTx) to upload to the blockchain. The policy manager is responsible for formulating SDN interdomain forwarding control policies oriented to the attributes of dataflow and path, including SOURCE policies and TRANSFER policies, and generating policy transactions (PolicyTx) to upload to the blockchain. The policy collaboration engine obtains global collaborative policies by querying the local cache and performing attribute mapping, policy matching, and path synthesizing.The attribute information of a domain is written into the domain information manager by its domain administrator. In addition, the identifier and attributes of network services are defined uniformly by offline negotiation and are shared by the blockchain. This article assumes that the domain information manager can obtain the correct registration information. The work on how to obtain the correct registration information can be addressed by specialized access authentication schemes, such as [3, 46], which are not included in this article.(2)
Blockchain Application. It includes the domain information management contract (DIM contract) and policy management contract (PM contract). The DIM contract is responsible for publishing and updating domain information, while the PM contract is responsible for policy publishing and updating. Through the consensus mechanism, the published or updated domain information and policy information are confirmed on the chain and written into the ledger database of the blockchain nodes in each SDN domain. Then, the domain information and policies in the DIM contract and PM contract accounts are dynamically synchronized to the local cache in real time.(3)
SDN Baseline Controller. The modules related to BPFC-SDNs include flow policy requester, WE-Bridge, interdomain path computation, and intradomain path computation. The flow policy requester is responsible for generating flow policy requests for dataflows whose destination addresses belong to other domains and submitting the flow policy requests to the policy collaboration engine. WE-Bridge provides a global interdomain network view to the interdomain path computation component. Interdomain path computation combines the global collaborative policy and interdomain network view to calculate interdomain path segments and transmit them to corresponding transmission domain controllers through east-west interfaces. The intradomain path computation of the source domain and the related transmission domains calculates the intradomain forwarding path according to the interdomain path segment and issue flow rules through flow_mod messages.


#### 5.2.1. Discussion on the Design of the Functional Model

It is worth noting that we do not hand over the policy collaboration function to the blockchain but let it be carried out on the local controller. This is for the following three reasons:(1)
The fundamental reason is that the attribute-based forwarding control cannot perform global policy collaboration before the arrival of dataflow but can only perform policy collaboration according to the arrival of dataflow, so the policy collaboration delay affects the dataflow forwarding latency. Moreover, because the arrival of dataflows is frequent and the blockchain is not suitable for frequent or real-time operations, the policy collaboration should be performed locally.(2)
If policy collaboration is implemented based on blockchain smart contracts, it will bring exponential computational overhead and latency. Before each dataflow was forwarded across domains, the results of its forwarding policy collaboration will be broadcasted between domains through blockchain communication, which will introduce huge communication overhead.(3)
Because we assume that the inner interactions within each control plane are secure and trusted, based on the trusted data foundation provided by the blockchain, each domain can achieve secure and trusted policy collaboration. As a result, it is not necessary to implement the supervision of the policy collaboration process based on the blockchain. If there is a need for supervision, the results of the local policy collaboration can be uploaded to the blockchain for verification, which is convenient for subsequent auditing.


### 5.3. The Overall Workflow of BPFC-SDNs

The overall workflow of BPFC-SDNs is divided into three stages, as shown in Figure5.(1)
Information Sharing. On the one hand, the domain information manager in the BPFC app of each domain encloses the attribute information of its domain, users, devices, and the mapping relationships of IP to the user identifier and device identifier in DomInfTx and then signs and uploads DomInfTx to the Blockchain. Next, Blockchain verifies the signature of DomInfTx and realizes the publishing or update of the domain information by calling the DIM contract. On the other hand, the policy manager in the BPFC app of each domain formulates the cross-domain forwarding control policies, encapsulates them into PolicyTx, and then signs and uploads PolicyTx to the Blockchain. Next, Blockchain verifies the signature of PolicyTx and realizes the publishing or update of the policies by calling the PM contract. Meanwhile, the blockchain function node in each domain dynamically synchronizes the domain information in the DIM contract account and the policies in the PM contract account to the local cache of the controller. Thus, the information-sharing stage realizes secure and trusted SDN domain information and policy sharing.When a new flow arrives at the ingress switch of a certain domain, the ingress switch queries its flow table; if there is a flow entry matching, it is forwarded according to the matched flow rules; if it does not exist, the first packet of the dataflow is uploaded to the controller through the packet_in message. Then, it triggers the forwarding control process based on policy collaboration.(2)
Policy Collaborating. First, the flow policy requester in the SDN baseline controller uploads the flow policy request message containing the first packet of the cross-domain dataflow to the policy collaboration engine in the BPFC application. Then, the policy collaboration engine obtains the global collaboration policy by querying the global domain information and global policies in the local cache and performing “attribute mapping,” “policy matching,” and “path synthesizing.” Finally, the policy collaboration engine issues the global collaboration policy of the dataflow to the SDN baseline controller. The three functions of policy collaboration are detailed in Section 6.4.(3)
Path Generating. Following the steps, the SDN baseline controller receives the global collaborative policy and hands it to the interdomain path computation component. The interdomain path computation component combines the global collaborative policy and the global domain-level network view provided by WE-bridge to calculate the interdomain path segments that satisfy global policy constraints and then distribute the interdomain path segments to controllers in corresponding transmission domains through east-west interfaces. The interdomain path segment structure is shown in Figure 6, where FlowSN is the identifier of the dataflow and is calculated by hashing the key fields in the pack header of the first package of the dataflow. The controller in each corresponding SDN domain calculates the intradomain path based on the interdomain path segment and delivers the flow entries by flow_mod messages.Figure 5
The overall workflow of BPFC-SDNs.Finally, these paths connect in turn and the policy-oriented cross-domain forwarding control is implemented.Figure 6
The interdomain path segment structure.


## 5.1. The Architecture of BPFC-SDNs

The architecture of the BPFC-SDNs is shown in Figure3. In the multidomain SDN scenario, the control plane of each domain consists of four models, including the original SDN controller, BPFC application, blockchain application, and WE-bridge.Figure 3
The architecture of BPFC-SDNs.The architecture of BPFC-SDNs mainly includes the following six key points:(1)
In each SDN domain, we deploy the blockchain application and the original SDN controller on the same local server, and the blockchain-based policy forwarding control application (BPFC APP) interacts with the original SDN controller and the blockchain through the local application program interface.(2)
The blockchain application in each domain serves as the accounting node of the blockchain and forms a blockchain network through P2P. Therefore, each SDN domain stores a blockchain global ledger.(3)
By synchronizing the blockchain contract data to the local cache of the SDN control plane, the BPFC APP can quickly read the global trusted data from the local cache.(4)
Because the realization of forwarding control requires network topology, according to the literature [42], the WE-bridge located in the SDN network system layer of each domain is connected in a full mesh manner, based on the SSL protocol, and through the JSON file to securely and quickly exchange the virtual network views among domains. As a result, each domain can acquire a global network view over multiple domains based on the WE-bridge. Since this is not the focus of this article, it is assumed that each SDN controller in BPFC-SDNs can dynamically obtain the global network view over multidomains in real time through the WE-bridge component.(5)
To ensure the normal transmission of data within the SDN domain, it is assumed that the interactions within the control plane are secure and trusted and that the communication between the controller plane and the data plane is secure and trusted.(6)
To improve the robustness of the system and solve the single point of failure problem, multiple control nodes can be deployed in the same domain based on distributed caches, such as Redis [43], to implement active-standby or even multiactive.This deployment method of the physical concentration and logical isolation of blockchain and the SDN controller has three advantages:(1)
Compared with the physical location separate deployment, it can provide secure and high-speed information interaction and contract data synchronization, avoiding high transmission overhead and insecure network environment of remote communication.(2)
Compared with the high coupling of the controller function and the blockchain function, it can reduce the complexity of the system, and it is easy to implement and maintain.(3)
At the same time, the method deploying blockchain function nodes on the local controller, hence making all domains complete function execution together, better conforms to the essential property of distributed multiparty comanagement of the blockchain than the physical separate deployment.


## 5.2. The Function Model of BPFC-SDNs

Based on the architecture, a blockchain-based policy-oriented SDN interdomain forwarding control function model is proposed, as shown in Figure4. BPCF-SDNs ensure the authentication and integrity of domain information and policy information uploaded by each SDN domain based on the transaction signature mechanism. It realizes automatic and transparent domain information and policy release and updates through smart contract technology. It guarantees that the current block stores the latest state of global domain information and policies based on the account model, avoiding the UTXO (unspent transaction output) [44] model traversing the entire blockchain to obtain the latest state and improving the data retrieval rate. It achieves the credibility of cross-domain data sharing effectively based on the PBFT (practical Byzantine fault tolerance) [45] consensus mechanism. Finally, it ensures that the domain information and policies shared are tamper-proof and traceable based on the blockchain storage structure. In sum, the blockchain in BPFC-SDNs can provide a secure and trusted global data basis for policy-oriented SDN interdomain forwarding control.Figure 4
The function model of BPFC-SDNs.The functional model of BPFC-SDNs mainly includes the following parts:(1)
BPFC Application. It includes the domain information manager, policy manager, and policy collaboration engine. The domain information manager is responsible for registering the attribute information of domains, users, and devices, updating the mapping relationship of IP addresses to users and devices, and generating domain information transactions (DomInfTx) to upload to the blockchain. The policy manager is responsible for formulating SDN interdomain forwarding control policies oriented to the attributes of dataflow and path, including SOURCE policies and TRANSFER policies, and generating policy transactions (PolicyTx) to upload to the blockchain. The policy collaboration engine obtains global collaborative policies by querying the local cache and performing attribute mapping, policy matching, and path synthesizing.The attribute information of a domain is written into the domain information manager by its domain administrator. In addition, the identifier and attributes of network services are defined uniformly by offline negotiation and are shared by the blockchain. This article assumes that the domain information manager can obtain the correct registration information. The work on how to obtain the correct registration information can be addressed by specialized access authentication schemes, such as [3, 46], which are not included in this article.(2)
Blockchain Application. It includes the domain information management contract (DIM contract) and policy management contract (PM contract). The DIM contract is responsible for publishing and updating domain information, while the PM contract is responsible for policy publishing and updating. Through the consensus mechanism, the published or updated domain information and policy information are confirmed on the chain and written into the ledger database of the blockchain nodes in each SDN domain. Then, the domain information and policies in the DIM contract and PM contract accounts are dynamically synchronized to the local cache in real time.(3)
SDN Baseline Controller. The modules related to BPFC-SDNs include flow policy requester, WE-Bridge, interdomain path computation, and intradomain path computation. The flow policy requester is responsible for generating flow policy requests for dataflows whose destination addresses belong to other domains and submitting the flow policy requests to the policy collaboration engine. WE-Bridge provides a global interdomain network view to the interdomain path computation component. Interdomain path computation combines the global collaborative policy and interdomain network view to calculate interdomain path segments and transmit them to corresponding transmission domain controllers through east-west interfaces. The intradomain path computation of the source domain and the related transmission domains calculates the intradomain forwarding path according to the interdomain path segment and issue flow rules through flow_mod messages.


### 5.2.1. Discussion on the Design of the Functional Model

It is worth noting that we do not hand over the policy collaboration function to the blockchain but let it be carried out on the local controller. This is for the following three reasons:(1)
The fundamental reason is that the attribute-based forwarding control cannot perform global policy collaboration before the arrival of dataflow but can only perform policy collaboration according to the arrival of dataflow, so the policy collaboration delay affects the dataflow forwarding latency. Moreover, because the arrival of dataflows is frequent and the blockchain is not suitable for frequent or real-time operations, the policy collaboration should be performed locally.(2)
If policy collaboration is implemented based on blockchain smart contracts, it will bring exponential computational overhead and latency. Before each dataflow was forwarded across domains, the results of its forwarding policy collaboration will be broadcasted between domains through blockchain communication, which will introduce huge communication overhead.(3)
Because we assume that the inner interactions within each control plane are secure and trusted, based on the trusted data foundation provided by the blockchain, each domain can achieve secure and trusted policy collaboration. As a result, it is not necessary to implement the supervision of the policy collaboration process based on the blockchain. If there is a need for supervision, the results of the local policy collaboration can be uploaded to the blockchain for verification, which is convenient for subsequent auditing.


## 5.2.1. Discussion on the Design of the Functional Model

It is worth noting that we do not hand over the policy collaboration function to the blockchain but let it be carried out on the local controller. This is for the following three reasons:(1)
The fundamental reason is that the attribute-based forwarding control cannot perform global policy collaboration before the arrival of dataflow but can only perform policy collaboration according to the arrival of dataflow, so the policy collaboration delay affects the dataflow forwarding latency. Moreover, because the arrival of dataflows is frequent and the blockchain is not suitable for frequent or real-time operations, the policy collaboration should be performed locally.(2)
If policy collaboration is implemented based on blockchain smart contracts, it will bring exponential computational overhead and latency. Before each dataflow was forwarded across domains, the results of its forwarding policy collaboration will be broadcasted between domains through blockchain communication, which will introduce huge communication overhead.(3)
Because we assume that the inner interactions within each control plane are secure and trusted, based on the trusted data foundation provided by the blockchain, each domain can achieve secure and trusted policy collaboration. As a result, it is not necessary to implement the supervision of the policy collaboration process based on the blockchain. If there is a need for supervision, the results of the local policy collaboration can be uploaded to the blockchain for verification, which is convenient for subsequent auditing.


## 5.3. The Overall Workflow of BPFC-SDNs

The overall workflow of BPFC-SDNs is divided into three stages, as shown in Figure5.(1)
Information Sharing. On the one hand, the domain information manager in the BPFC app of each domain encloses the attribute information of its domain, users, devices, and the mapping relationships of IP to the user identifier and device identifier in DomInfTx and then signs and uploads DomInfTx to the Blockchain. Next, Blockchain verifies the signature of DomInfTx and realizes the publishing or update of the domain information by calling the DIM contract. On the other hand, the policy manager in the BPFC app of each domain formulates the cross-domain forwarding control policies, encapsulates them into PolicyTx, and then signs and uploads PolicyTx to the Blockchain. Next, Blockchain verifies the signature of PolicyTx and realizes the publishing or update of the policies by calling the PM contract. Meanwhile, the blockchain function node in each domain dynamically synchronizes the domain information in the DIM contract account and the policies in the PM contract account to the local cache of the controller. Thus, the information-sharing stage realizes secure and trusted SDN domain information and policy sharing.When a new flow arrives at the ingress switch of a certain domain, the ingress switch queries its flow table; if there is a flow entry matching, it is forwarded according to the matched flow rules; if it does not exist, the first packet of the dataflow is uploaded to the controller through the packet_in message. Then, it triggers the forwarding control process based on policy collaboration.(2)
Policy Collaborating. First, the flow policy requester in the SDN baseline controller uploads the flow policy request message containing the first packet of the cross-domain dataflow to the policy collaboration engine in the BPFC application. Then, the policy collaboration engine obtains the global collaboration policy by querying the global domain information and global policies in the local cache and performing “attribute mapping,” “policy matching,” and “path synthesizing.” Finally, the policy collaboration engine issues the global collaboration policy of the dataflow to the SDN baseline controller. The three functions of policy collaboration are detailed in Section 6.4.(3)
Path Generating. Following the steps, the SDN baseline controller receives the global collaborative policy and hands it to the interdomain path computation component. The interdomain path computation component combines the global collaborative policy and the global domain-level network view provided by WE-bridge to calculate the interdomain path segments that satisfy global policy constraints and then distribute the interdomain path segments to controllers in corresponding transmission domains through east-west interfaces. The interdomain path segment structure is shown in Figure 6, where FlowSN is the identifier of the dataflow and is calculated by hashing the key fields in the pack header of the first package of the dataflow. The controller in each corresponding SDN domain calculates the intradomain path based on the interdomain path segment and delivers the flow entries by flow_mod messages.Figure 5
The overall workflow of BPFC-SDNs.Finally, these paths connect in turn and the policy-oriented cross-domain forwarding control is implemented.Figure 6
The interdomain path segment structure.


## 6. Transactions, Contracts, and Algorithms in BPFC-SDNs

### 6.1. Transactions

In order to achieve secure and trusted domain information and policy sharing, the transactions DomInfTx and PolicyTx are designed, and the abstract structure of transactions is shown in Figure7.Figure 7
Transaction structure.Figure7(a) is the general transaction structure.ID: transaction identifier.PK: the public key of the blockchain account of a certain domain. The BPFC application in each SDN domain has its own public-private key pair. The public key is used as the unique identifier of the external account on the blockchain, and the private key is used to sign transactions.TxType: transaction type.TxData: transaction data.tp: the timestamp of the transaction release.sign: the signature of the transaction, which is computed by the account private key.Domain information transaction (DomInfTx): the structure of DomInfTx is shown in Figure7(b), and its transaction data are DomInf, which includes the domain ID, the set of mapping relationships of IP to the user and device, and the set of the attribute name-value pairs of its domain, users, and devices. DomInf is stored in the form of key-value pairs, with the domID as the key and the rest as the value. Its formal representation is shown in Figure 7(d).Policy transaction (PolicyTx): the structure of PolicyTx is shown in Figure7(c), and its transaction data are PolicySet, including domID, the source policy set, and the transmission policy set, both of which are created by the SDN domain releasing this transaction. Likewise, PolicySet is stored as key-value pairs as well, with domID as the key and the rest as the value. The formal representation of PolicySet is shown in Figure 7(e).


### 6.2. DIM Contract

DIM contract implements domain information publishing or updating by utilizing the domain information publishing function DomInfPublish(), as shown in Algorithm1. By calling the DomInfPublish(), the mapping relationship between the IP and the entity as well as the attribute information of the entity are written into the DIM contract account, with the domID as the index. The blockchain function node of each domain synchronizes the data in the DIM contract account to the local cache, which is noted as DomInfDB.Algorithm 1: DIM contract.
(1)
DomInfPublish(DomInf)(2)
domID← DomInf.getDomID()(3)
if domID does not exist(4)
put(DomInf[domID])(5)
else(6)
update(DomInf[domID])(7)
end if(8)
end DomInfPublish()


### 6.3. PM Contract

PM contract implements policy publishing and updating by taking advantage of the PolicyPublish() function, as shown in Algorithm2. By calling the PolicyPublish(), the policies are written into the PM contract account, which is indexed by the domID. The blockchain function nodes of each domain synchronize the data in the PM contract account to the local cache, which is noted as PolicyDB.Algorithm 2: PM contract.
(1)
PolicyPublish(PolicySet)(2)
domID← PolicySet.getDomID()(3)
if domID not exists(4)
put(PolicySet[domID])(5)
else(6)
update(PolicySet[domID])(7)
end if(8)
end PolicyPublish()


### 6.4. Policy Collaboration Algorithm

When a new dataflow arrives, the policy collaboration engine is triggered to execute the policy collaboration algorithm PolicyCollaborate(), as shown in Algorithm3. The policy collaboration algorithm is realized in three steps. First, it parses the flow policy request to obtain the header of the first packet of the dataflow; by querying DomInfDB, the network identifiers are mapped to the attributes of the dataflow (lines 1-2). Then, according to the attributes of the dataflow, policy matching is performed to obtain all the policies that constrain this dataflow (line 3). Finally, it performs path synthesis on the matching policy set to obtain the global path constraint of the dataflow and return global collaborative policy (lines 4–5).Algorithm 3: PolicyCollaborate().
Input: FlowPolicyRequestOutput: collaborativePolicy(1) packetHeader = Parser(FlowPolicyRequest)(2) mapAttrSet = AttributeMap(packetHeader)(3) matchPolicySet = PolicyMatch(mapAttrSet.flowAttrSet)(4) collaborativePolicy = PathSynthesize(mapAttrSet.flowSN, matchPolicySet)(5) returncollaborativePolicy


#### 6.4.1. Attribute Mapping

The attribute mapping function AttributeMap() maps the attributes of the dataflow according to the key fields of the packet header. As shown in Algorithm4, the workflow of AttributeMap() is as follows:(1)
It reads the source IP field of the packet header, then gets the corresponding entity identifiers according to the source IP, and further obtains the attribute name-value pair set of the source domain, source user, and source device from DomInfDB. Finally, it composites them as the source attribute of the dataflow (lines 2–4).(2)
It reads the destination IP, and the same as above, it obtains the destination attribute of the dataflow (lines 5–7).(3)
It reads the destination port number, protocol type, or self-defined service field as the service identifier and then obtains the attribute name-value pair set of the service from DomInfDB according to the service identifier (lines 8–9).(4)
It combines the source attribute, destination attribute, and service attribute to form the attribute set of this dataflow, namely, flowAVP (line 10).(5)
It generates the corresponding flow identifier FlowSN by hashing the key fields of the packet header (line 11).(6)
The flow identifier FlowSN and the flow attribute set flowAVP are combined as the mapped attribute set of the requesting flow, namely, mapAttrSet, and it is returned as the result of the attribute mapping function AttributeMap() (lines 12–13).Algorithm 4: AttributeMap().
Input: packetHeader;Output: mapAttrSet;(1) srcAVP =∅; dstAVP = ∅; flowAVP = ∅; flowSN = null(2) {domID, userID, devID} = DomInfDB.getID (packetHeader.srcIP)(3) {domAVP, userAVP, devAVP} = DomInfDB.getAVP({domID, userID, devID})(4) srcAVP← {domAVP, userAVP, devAVP}(5) {domID, userID, devID} = DomInfDB.getID(packetHeader.dstIP)(6) {domAVP, userAVP, devAVP} = DomInfDB.getAVP({domID, userID, devID})(7) destAVP← {domAVP, userAVP, devAVP}(8) servID = DomInfDB.getID (packetHeader.service)(9) servAVP = DomInfDB.getAVP (servID)(10) flowAVP← {srcAVP, destAVP, servAVP}(11) flowSN← Hash (packerHeader.keyFields)(12) mapAttrSet← <flowSN, flowAVP>(13) returnmapAttrSet


#### 6.4.2. Policy Matching

According to the flow attributes flowAVP, the policy matching function PolicyMatch() obtains a policy set that restricts the forwarding path of the dataflow. As shown in Algorithm5, the workflow of PolicyMatch() is as follows:(1)
It initializes the flow attribute variable flowAVP as the attribute set of the flow requesting forwarding, the attribute matching mark as 0, and the source policy set, transfer policy set, relevant policy set, and matching policy set are initialized as empty (line 1).(2)
According to the source domain identifier of the dataflow, the SOURCE policies of the source domain are read from PolicyDB (line 2).(3)
Similarly, the TRANSFER policies of all domains are read from PolicyDB (line 3).(4)
The SOURCE policies and the TRANSFER policies are combined as the relevant policy set (line 4).(5)
For every policy in the relevant policy set, if the policy matches the dataflow as Definition 7, it will be added into the matchPolicySet; otherwise, the next policy will be traversed until all relevant policies are traversed (lines 5–24).(6)
It returns the matched policy set (line 25).Algorithm 5: PolicyMatch().
Input: mapAttrSet;Output: matchPolicySet;(1) flowAVP = mapAttrSet.flowAVP; mark = 0; srcPolicySet =∅; transferPolicySet = ∅; relevantPolicySet = ∅; matchPolicySet = ∅(2) srcPolicySet← PolicyDB.getPolicySet (flowAVP.srcAVP.domID, SOURCE)(3) transferPolicySet← PolicyDB.getPolicySet (any, TRANSFER)(4) relevantPolicySet← srcPolicySet ∪ transferPolicySet(5)for (i = 0; i < getNum(relevantPolicySet.policy); i++) do(6)for (j = 0; j < getNum(relevantPolicySet.policy[i].ap); j++) do(7)for (k = 0; k < getNum(flowAVP.avp); k++) do(8)if (relevantPolicySet.policy[i].ap[j].attr ! = flowAVP.avp[k].attr) then(9)     continue(10)else if (relevantPolicySet.policy[i].ap[j].attr = = flowAVP.avp[k].attr&&relevantPolicySet.policy[i].ap[j].value∩flowAVP.avp[k].value = = ∅) then(11)     break(12)else(13)     mark = mark++(14)     break(15)end if(16)end for(17)if (mark = 0) then(18)    break(19)end if(20)end for(21)if (mark = = getNum(relevantPolicySet.policy[i].ap) then(22)   matchPolicySet.increase(relevantPolicySet.policy[i])(23)end if(24)end for(25) returnmatchPolicySet


#### 6.4.3. Path Synthesizing

The path synthesis function PathSynthesize() performs path constraint synthesis on the matchPolicySet of the flow to obtain its global path constraints. As shown in Algorithm6, the workflow of PathSynthesize() is as follows:(1)
It initializes the global path constraint of the flow flowPath as the universe, the variablespath and action as null, and the global collaborative policy of the dataflow collaborativePolicy as empty (line 1).(2)
For each SOURCE policy in the matchPolicySet, if its action is forward, its path constraint will be written into the flowPath; if its action isdrop, its path constraint will be inverted and written into the flowPath (lines 2–4, lines 5–13).(3)
For each TRANSFER policy in the matchPolicySet, if its path constraint is not empty, it violates the appointment in this article (see Section 4.2), and then an error is reported; if its path constraint is empty, and the action isdrop, then the reverse of the field identifier will be written into the flowPath; if the path constraint is empty and the action is forward, it does not affect dataflow forwarding and traverse the next policy (lines 2–4, lines 14–25).(4)
It combines the flow identifier flowSN, the flow path constraint flowPath, and the action forward as a global collaborative policy and returns the global collaborative policy (lines 31–32).Algorithm 6: PathSynthesize().
Input: mapAttrSet.flowSN, matchPolicySetOutput: collaborativePolicy(1) flowPath =U; path = null; action = null; collaborativePolicy = ∅(2)for (i = 0; i < getNum(matchPolicySet); i++) do(3)  path← getPath(matchPolicySet.policy[i])(4)  action← getAction(matchPolicySet.policy[i])(5)if (matchPolicySet.policy[i].policyType = = SOURCE) then(6)if (action = = forward) then(7)    flowPath← flowPath∧path(8)else if (action = = drop) then(9)    flowPath← flowPath∧¬path(10)else(11)    report error(12)    continue(13)end if(14)else if (matchPolicySet.policy[i].policyType = = TRANSFER) then(15)if (path! = null) then(16)    report error(17)    continue(18)else(19)if (action = = drop) then(20)     domID← getDomID(matchPolicySet.policy[i])(21)     flowPath← flowPath∧¬domID(22)else(23)     continue(24)end if(25)end if(26)else(27)   report error(28)   continue(29)end if(30)end for(31) collaborativePolicy← <flowSN, flowPath, forward>(32) returncollaborativePolicy


## 6.1. Transactions

In order to achieve secure and trusted domain information and policy sharing, the transactions DomInfTx and PolicyTx are designed, and the abstract structure of transactions is shown in Figure7.Figure 7
Transaction structure.Figure7(a) is the general transaction structure.ID: transaction identifier.PK: the public key of the blockchain account of a certain domain. The BPFC application in each SDN domain has its own public-private key pair. The public key is used as the unique identifier of the external account on the blockchain, and the private key is used to sign transactions.TxType: transaction type.TxData: transaction data.tp: the timestamp of the transaction release.sign: the signature of the transaction, which is computed by the account private key.Domain information transaction (DomInfTx): the structure of DomInfTx is shown in Figure7(b), and its transaction data are DomInf, which includes the domain ID, the set of mapping relationships of IP to the user and device, and the set of the attribute name-value pairs of its domain, users, and devices. DomInf is stored in the form of key-value pairs, with the domID as the key and the rest as the value. Its formal representation is shown in Figure 7(d).Policy transaction (PolicyTx): the structure of PolicyTx is shown in Figure7(c), and its transaction data are PolicySet, including domID, the source policy set, and the transmission policy set, both of which are created by the SDN domain releasing this transaction. Likewise, PolicySet is stored as key-value pairs as well, with domID as the key and the rest as the value. The formal representation of PolicySet is shown in Figure 7(e).


## 6.2. DIM Contract

DIM contract implements domain information publishing or updating by utilizing the domain information publishing function DomInfPublish(), as shown in Algorithm1. By calling the DomInfPublish(), the mapping relationship between the IP and the entity as well as the attribute information of the entity are written into the DIM contract account, with the domID as the index. The blockchain function node of each domain synchronizes the data in the DIM contract account to the local cache, which is noted as DomInfDB.Algorithm 1: DIM contract.
(1)
DomInfPublish(DomInf)(2)
domID← DomInf.getDomID()(3)
if domID does not exist(4)
put(DomInf[domID])(5)
else(6)
update(DomInf[domID])(7)
end if(8)
end DomInfPublish()


## 6.3. PM Contract

PM contract implements policy publishing and updating by taking advantage of the PolicyPublish() function, as shown in Algorithm2. By calling the PolicyPublish(), the policies are written into the PM contract account, which is indexed by the domID. The blockchain function nodes of each domain synchronize the data in the PM contract account to the local cache, which is noted as PolicyDB.Algorithm 2: PM contract.
(1)
PolicyPublish(PolicySet)(2)
domID← PolicySet.getDomID()(3)
if domID not exists(4)
put(PolicySet[domID])(5)
else(6)
update(PolicySet[domID])(7)
end if(8)
end PolicyPublish()


## 6.4. Policy Collaboration Algorithm

When a new dataflow arrives, the policy collaboration engine is triggered to execute the policy collaboration algorithm PolicyCollaborate(), as shown in Algorithm3. The policy collaboration algorithm is realized in three steps. First, it parses the flow policy request to obtain the header of the first packet of the dataflow; by querying DomInfDB, the network identifiers are mapped to the attributes of the dataflow (lines 1-2). Then, according to the attributes of the dataflow, policy matching is performed to obtain all the policies that constrain this dataflow (line 3). Finally, it performs path synthesis on the matching policy set to obtain the global path constraint of the dataflow and return global collaborative policy (lines 4–5).Algorithm 3: PolicyCollaborate().
Input: FlowPolicyRequestOutput: collaborativePolicy(1) packetHeader = Parser(FlowPolicyRequest)(2) mapAttrSet = AttributeMap(packetHeader)(3) matchPolicySet = PolicyMatch(mapAttrSet.flowAttrSet)(4) collaborativePolicy = PathSynthesize(mapAttrSet.flowSN, matchPolicySet)(5) returncollaborativePolicy


### 6.4.1. Attribute Mapping

The attribute mapping function AttributeMap() maps the attributes of the dataflow according to the key fields of the packet header. As shown in Algorithm4, the workflow of AttributeMap() is as follows:(1)
It reads the source IP field of the packet header, then gets the corresponding entity identifiers according to the source IP, and further obtains the attribute name-value pair set of the source domain, source user, and source device from DomInfDB. Finally, it composites them as the source attribute of the dataflow (lines 2–4).(2)
It reads the destination IP, and the same as above, it obtains the destination attribute of the dataflow (lines 5–7).(3)
It reads the destination port number, protocol type, or self-defined service field as the service identifier and then obtains the attribute name-value pair set of the service from DomInfDB according to the service identifier (lines 8–9).(4)
It combines the source attribute, destination attribute, and service attribute to form the attribute set of this dataflow, namely, flowAVP (line 10).(5)
It generates the corresponding flow identifier FlowSN by hashing the key fields of the packet header (line 11).(6)
The flow identifier FlowSN and the flow attribute set flowAVP are combined as the mapped attribute set of the requesting flow, namely, mapAttrSet, and it is returned as the result of the attribute mapping function AttributeMap() (lines 12–13).Algorithm 4: AttributeMap().
Input: packetHeader;Output: mapAttrSet;(1) srcAVP =∅; dstAVP = ∅; flowAVP = ∅; flowSN = null(2) {domID, userID, devID} = DomInfDB.getID (packetHeader.srcIP)(3) {domAVP, userAVP, devAVP} = DomInfDB.getAVP({domID, userID, devID})(4) srcAVP← {domAVP, userAVP, devAVP}(5) {domID, userID, devID} = DomInfDB.getID(packetHeader.dstIP)(6) {domAVP, userAVP, devAVP} = DomInfDB.getAVP({domID, userID, devID})(7) destAVP← {domAVP, userAVP, devAVP}(8) servID = DomInfDB.getID (packetHeader.service)(9) servAVP = DomInfDB.getAVP (servID)(10) flowAVP← {srcAVP, destAVP, servAVP}(11) flowSN← Hash (packerHeader.keyFields)(12) mapAttrSet← <flowSN, flowAVP>(13) returnmapAttrSet


### 6.4.2. Policy Matching

According to the flow attributes flowAVP, the policy matching function PolicyMatch() obtains a policy set that restricts the forwarding path of the dataflow. As shown in Algorithm5, the workflow of PolicyMatch() is as follows:(1)
It initializes the flow attribute variable flowAVP as the attribute set of the flow requesting forwarding, the attribute matching mark as 0, and the source policy set, transfer policy set, relevant policy set, and matching policy set are initialized as empty (line 1).(2)
According to the source domain identifier of the dataflow, the SOURCE policies of the source domain are read from PolicyDB (line 2).(3)
Similarly, the TRANSFER policies of all domains are read from PolicyDB (line 3).(4)
The SOURCE policies and the TRANSFER policies are combined as the relevant policy set (line 4).(5)
For every policy in the relevant policy set, if the policy matches the dataflow as Definition 7, it will be added into the matchPolicySet; otherwise, the next policy will be traversed until all relevant policies are traversed (lines 5–24).(6)
It returns the matched policy set (line 25).Algorithm 5: PolicyMatch().
Input: mapAttrSet;Output: matchPolicySet;(1) flowAVP = mapAttrSet.flowAVP; mark = 0; srcPolicySet =∅; transferPolicySet = ∅; relevantPolicySet = ∅; matchPolicySet = ∅(2) srcPolicySet← PolicyDB.getPolicySet (flowAVP.srcAVP.domID, SOURCE)(3) transferPolicySet← PolicyDB.getPolicySet (any, TRANSFER)(4) relevantPolicySet← srcPolicySet ∪ transferPolicySet(5)for (i = 0; i < getNum(relevantPolicySet.policy); i++) do(6)for (j = 0; j < getNum(relevantPolicySet.policy[i].ap); j++) do(7)for (k = 0; k < getNum(flowAVP.avp); k++) do(8)if (relevantPolicySet.policy[i].ap[j].attr ! = flowAVP.avp[k].attr) then(9)     continue(10)else if (relevantPolicySet.policy[i].ap[j].attr = = flowAVP.avp[k].attr&&relevantPolicySet.policy[i].ap[j].value∩flowAVP.avp[k].value = = ∅) then(11)     break(12)else(13)     mark = mark++(14)     break(15)end if(16)end for(17)if (mark = 0) then(18)    break(19)end if(20)end for(21)if (mark = = getNum(relevantPolicySet.policy[i].ap) then(22)   matchPolicySet.increase(relevantPolicySet.policy[i])(23)end if(24)end for(25) returnmatchPolicySet


### 6.4.3. Path Synthesizing

The path synthesis function PathSynthesize() performs path constraint synthesis on the matchPolicySet of the flow to obtain its global path constraints. As shown in Algorithm6, the workflow of PathSynthesize() is as follows:(1)
It initializes the global path constraint of the flow flowPath as the universe, the variablespath and action as null, and the global collaborative policy of the dataflow collaborativePolicy as empty (line 1).(2)
For each SOURCE policy in the matchPolicySet, if its action is forward, its path constraint will be written into the flowPath; if its action isdrop, its path constraint will be inverted and written into the flowPath (lines 2–4, lines 5–13).(3)
For each TRANSFER policy in the matchPolicySet, if its path constraint is not empty, it violates the appointment in this article (see Section 4.2), and then an error is reported; if its path constraint is empty, and the action isdrop, then the reverse of the field identifier will be written into the flowPath; if the path constraint is empty and the action is forward, it does not affect dataflow forwarding and traverse the next policy (lines 2–4, lines 14–25).(4)
It combines the flow identifier flowSN, the flow path constraint flowPath, and the action forward as a global collaborative policy and returns the global collaborative policy (lines 31–32).Algorithm 6: PathSynthesize().
Input: mapAttrSet.flowSN, matchPolicySetOutput: collaborativePolicy(1) flowPath =U; path = null; action = null; collaborativePolicy = ∅(2)for (i = 0; i < getNum(matchPolicySet); i++) do(3)  path← getPath(matchPolicySet.policy[i])(4)  action← getAction(matchPolicySet.policy[i])(5)if (matchPolicySet.policy[i].policyType = = SOURCE) then(6)if (action = = forward) then(7)    flowPath← flowPath∧path(8)else if (action = = drop) then(9)    flowPath← flowPath∧¬path(10)else(11)    report error(12)    continue(13)end if(14)else if (matchPolicySet.policy[i].policyType = = TRANSFER) then(15)if (path! = null) then(16)    report error(17)    continue(18)else(19)if (action = = drop) then(20)     domID← getDomID(matchPolicySet.policy[i])(21)     flowPath← flowPath∧¬domID(22)else(23)     continue(24)end if(25)end if(26)else(27)   report error(28)   continue(29)end if(30)end for(31) collaborativePolicy← <flowSN, flowPath, forward>(32) returncollaborativePolicy


## 6.4.1. Attribute Mapping

The attribute mapping function AttributeMap() maps the attributes of the dataflow according to the key fields of the packet header. As shown in Algorithm4, the workflow of AttributeMap() is as follows:(1)
It reads the source IP field of the packet header, then gets the corresponding entity identifiers according to the source IP, and further obtains the attribute name-value pair set of the source domain, source user, and source device from DomInfDB. Finally, it composites them as the source attribute of the dataflow (lines 2–4).(2)
It reads the destination IP, and the same as above, it obtains the destination attribute of the dataflow (lines 5–7).(3)
It reads the destination port number, protocol type, or self-defined service field as the service identifier and then obtains the attribute name-value pair set of the service from DomInfDB according to the service identifier (lines 8–9).(4)
It combines the source attribute, destination attribute, and service attribute to form the attribute set of this dataflow, namely, flowAVP (line 10).(5)
It generates the corresponding flow identifier FlowSN by hashing the key fields of the packet header (line 11).(6)
The flow identifier FlowSN and the flow attribute set flowAVP are combined as the mapped attribute set of the requesting flow, namely, mapAttrSet, and it is returned as the result of the attribute mapping function AttributeMap() (lines 12–13).Algorithm 4: AttributeMap().
Input: packetHeader;Output: mapAttrSet;(1) srcAVP =∅; dstAVP = ∅; flowAVP = ∅; flowSN = null(2) {domID, userID, devID} = DomInfDB.getID (packetHeader.srcIP)(3) {domAVP, userAVP, devAVP} = DomInfDB.getAVP({domID, userID, devID})(4) srcAVP← {domAVP, userAVP, devAVP}(5) {domID, userID, devID} = DomInfDB.getID(packetHeader.dstIP)(6) {domAVP, userAVP, devAVP} = DomInfDB.getAVP({domID, userID, devID})(7) destAVP← {domAVP, userAVP, devAVP}(8) servID = DomInfDB.getID (packetHeader.service)(9) servAVP = DomInfDB.getAVP (servID)(10) flowAVP← {srcAVP, destAVP, servAVP}(11) flowSN← Hash (packerHeader.keyFields)(12) mapAttrSet← <flowSN, flowAVP>(13) returnmapAttrSet


## 6.4.2. Policy Matching

According to the flow attributes flowAVP, the policy matching function PolicyMatch() obtains a policy set that restricts the forwarding path of the dataflow. As shown in Algorithm5, the workflow of PolicyMatch() is as follows:(1)
It initializes the flow attribute variable flowAVP as the attribute set of the flow requesting forwarding, the attribute matching mark as 0, and the source policy set, transfer policy set, relevant policy set, and matching policy set are initialized as empty (line 1).(2)
According to the source domain identifier of the dataflow, the SOURCE policies of the source domain are read from PolicyDB (line 2).(3)
Similarly, the TRANSFER policies of all domains are read from PolicyDB (line 3).(4)
The SOURCE policies and the TRANSFER policies are combined as the relevant policy set (line 4).(5)
For every policy in the relevant policy set, if the policy matches the dataflow as Definition 7, it will be added into the matchPolicySet; otherwise, the next policy will be traversed until all relevant policies are traversed (lines 5–24).(6)
It returns the matched policy set (line 25).Algorithm 5: PolicyMatch().
Input: mapAttrSet;Output: matchPolicySet;(1) flowAVP = mapAttrSet.flowAVP; mark = 0; srcPolicySet =∅; transferPolicySet = ∅; relevantPolicySet = ∅; matchPolicySet = ∅(2) srcPolicySet← PolicyDB.getPolicySet (flowAVP.srcAVP.domID, SOURCE)(3) transferPolicySet← PolicyDB.getPolicySet (any, TRANSFER)(4) relevantPolicySet← srcPolicySet ∪ transferPolicySet(5)for (i = 0; i < getNum(relevantPolicySet.policy); i++) do(6)for (j = 0; j < getNum(relevantPolicySet.policy[i].ap); j++) do(7)for (k = 0; k < getNum(flowAVP.avp); k++) do(8)if (relevantPolicySet.policy[i].ap[j].attr ! = flowAVP.avp[k].attr) then(9)     continue(10)else if (relevantPolicySet.policy[i].ap[j].attr = = flowAVP.avp[k].attr&&relevantPolicySet.policy[i].ap[j].value∩flowAVP.avp[k].value = = ∅) then(11)     break(12)else(13)     mark = mark++(14)     break(15)end if(16)end for(17)if (mark = 0) then(18)    break(19)end if(20)end for(21)if (mark = = getNum(relevantPolicySet.policy[i].ap) then(22)   matchPolicySet.increase(relevantPolicySet.policy[i])(23)end if(24)end for(25) returnmatchPolicySet


## 6.4.3. Path Synthesizing

The path synthesis function PathSynthesize() performs path constraint synthesis on the matchPolicySet of the flow to obtain its global path constraints. As shown in Algorithm6, the workflow of PathSynthesize() is as follows:(1)
It initializes the global path constraint of the flow flowPath as the universe, the variablespath and action as null, and the global collaborative policy of the dataflow collaborativePolicy as empty (line 1).(2)
For each SOURCE policy in the matchPolicySet, if its action is forward, its path constraint will be written into the flowPath; if its action isdrop, its path constraint will be inverted and written into the flowPath (lines 2–4, lines 5–13).(3)
For each TRANSFER policy in the matchPolicySet, if its path constraint is not empty, it violates the appointment in this article (see Section 4.2), and then an error is reported; if its path constraint is empty, and the action isdrop, then the reverse of the field identifier will be written into the flowPath; if the path constraint is empty and the action is forward, it does not affect dataflow forwarding and traverse the next policy (lines 2–4, lines 14–25).(4)
It combines the flow identifier flowSN, the flow path constraint flowPath, and the action forward as a global collaborative policy and returns the global collaborative policy (lines 31–32).Algorithm 6: PathSynthesize().
Input: mapAttrSet.flowSN, matchPolicySetOutput: collaborativePolicy(1) flowPath =U; path = null; action = null; collaborativePolicy = ∅(2)for (i = 0; i < getNum(matchPolicySet); i++) do(3)  path← getPath(matchPolicySet.policy[i])(4)  action← getAction(matchPolicySet.policy[i])(5)if (matchPolicySet.policy[i].policyType = = SOURCE) then(6)if (action = = forward) then(7)    flowPath← flowPath∧path(8)else if (action = = drop) then(9)    flowPath← flowPath∧¬path(10)else(11)    report error(12)    continue(13)end if(14)else if (matchPolicySet.policy[i].policyType = = TRANSFER) then(15)if (path! = null) then(16)    report error(17)    continue(18)else(19)if (action = = drop) then(20)     domID← getDomID(matchPolicySet.policy[i])(21)     flowPath← flowPath∧¬domID(22)else(23)     continue(24)end if(25)end if(26)else(27)   report error(28)   continue(29)end if(30)end for(31) collaborativePolicy← <flowSN, flowPath, forward>(32) returncollaborativePolicy


## 7. Experiments and Evaluation

In order to verify the effectiveness of BPFC-SDNs and evaluate their performance, we implement its proof-of-concept prototype. We utilize Mininet to build the simulated SDN multidomain environment and adopt Floodlight as the controller. We add the BPFC application in each controller and deploy the FISCO BCOS [47] blockchain node function on each domain control node. FISCO BCOS is an open-source consortium blockchain platform, which fits well with the BPFC-SDNs. The domain information and policies are transmitted and stored in JSON file format. The experiments are conducted on an Ubuntu-18.04 system in a VMware Workstation 15 pro virtual machine. The host configuration is Intel Core i7-10750H 2.6 GHz CPU and 32 G of RAM.In the experiment, the baseline system is the same as BPFC-SDNs that each domain controller has a global network view of multidomain; however, the baseline system only has the function of routing, without the function of policy-oriented forwarding control. As a comparison, PbSA [5] provides a policy-oriented forwarding control function among SDN domains and implements cross-domain policy transfer by carrying policy transfer tokens in data packets. BPFC-SDNs provide policy-oriented forwarding control function among SDN domains and realize secure and trusted domain information and policies sharing through blockchain.In this section, we first test the functional effectiveness of BPFC-SDNs, then analyse the security and trustworthiness of BPFC-SDNs, and finally evaluate the performance of BPFC-SDNs.


### 7.1. Functional Effectiveness Testing of BPFC-SDNs

To verify the effectiveness of BPFC-SDNs, two experiments are designed. The validity verification experiment of the policy-oriented forwarding control function, by configuring the forwarding control policy in the source domain, verifies whether the BPFC-SDNs can forward the dataflow based on the path constraints in the policy. To verify the validity of the information-sharing and policy collaboration function, the experiment configures a policy in a transmission domain conflicting with the source domain and tests whether BPFC-SDNs can forward dataflow based on the global policy constraints. In order to realize the experiments, the topology shown in Figure2 (Section 4.3) is built. The attribute information of each domain is shown in the tables in Figure 2, and the whole link bandwidth is set to 1 Gbit/s.


#### 7.1.1. Validation of Policy-Oriented Forwarding Control Function

The experiment assumes that to improve the security of file transmission, AS1 formulates a policypolicyAS11 (see Section 4.3), which enforces that the FTP traffic sent from device_i to AS5 should be forwarded by a domain with a security level greater than or equal to 3. The experiment utilized the packet manipulation program Scapy to construct FTP packets sent from device_i to device_j at a rate of 30 package/s. The experiment lasted 10 seconds each time and was repeated 100 times. The data packets passing through the source domain egress switches S12, S13, and S14 were captured by TCPDUMP. The packet sending rate of each switch was counted under the baseline system and BPFC-SDNs, respectively, and the results are shown in Figure 8.Figure 8
The flow forwarding paths under baseline and BPFC-SDNs when installpolicyAS11.The experimental results show that the baseline system forwards dataflows to S12, S13, and S14, corresponding to the interdomain forwarding path AS1-AS2-AS5, AS1-AS3-AS5, and AS1-AS4-AS5, respectively. However, BPFC-SDNs forward the dataflow only to S13 and S14; namely, the dataflow forwarding paths in BPFC-SDNs are AS1-AS3-AS5 and AS1-AS4-AS5, which complies with the path constraint that the security level of the transmission domain should be greater than or equal to 3 in the policypolicyAS11. This indicates that in the same experimental environment, the baseline system cannot realize the policy-oriented forwarding control, while the BPFC-SDNs can realize the policy-oriented forwarding control function.


#### 7.1.2. Validation of Information-Sharing and Policy Collaboration Function

On the basis of the experiment in the previous step, AS4 as a domain with a high-security level formulates a policypolicyAS44 (Section 4.3) to refuse to carry dataflows originating from devices with a security level less than or equal to 3 so as to ensure the path isolation of high-security level data transmission. The experiment still constructed FTP packets sent from device_i to device_j at a rate of 30 packet/s. Moreover, each experiment lasted for 10 seconds and was repeated 100 times. Figure 9 presents the packet sending rate of egress switches S12, S13, and S14 in AS1, respectively, under baseline and BPFC-SDNs.Figure 9
The flow forwarding paths under baseline and BPFC-SDNs when installpolicyAS11 and policyAS44.The experimental results show that the baseline system still forwards data packets to AS2, AS3, and AS4, and the data forwarding is not affected by the policypolicyAS44. However, all traffic of BPFC-SDNs is forwarded to AS3, which satisfies both policyAS11 and policyAS44. Compared with the previous experiment, the source domain in BPFC-SDNs no longer forwards dataflows to AS4, which shows that the source domain has taken the policy constraints of the transmission domain AS4 into consideration when calculating the interdomain routing, thereby avoiding data transfer interruption. Additionally, in PbSA, policies are transmitted domain by domain through policy transfer tokens, and the source domain cannot know the policy of the transport domain. Once there is a policy conflict between the transmission domain and the source domain, data transmission will be interrupted. Therefore, the experimental result verifies that the information-sharing and policy collaboration function of BPFC-SDNs can effectively solve the cross-domain policy agnostic and policy conflict problems.


### 7.2. Security and Trustworthiness Analysis of BPFC-SDNs

According to the overall workflow of BPFC-SDNs, it is analysed that BPFC-SDNs can realize secure and trusted policy-oriented forwarding control.In the information-sharing phase, the transaction signature verification mechanism ensures the authenticity and integrity of domain information and policies, as shown in Figure10(a); the consensus mechanism ensures the nonrepudiation and traceability of domain information and policies as shown in Figure 10(b); the blockchain contract data are synchronized to the controller local cache in a secure and trusted environment of the controller as shown in Figure 10(c). Thus, BPFC-SDNs achieve secure and trusted domain information and policy sharing in the information-sharing phase. In the policy collaboration phase, secure and trusted controllers can implement policy collaboration based on globally trusted domain information and policies as shown in Figure 10(d). Therefore, BPFC-SDNs can achieve secure and trusted cross-domain policy collaboration locally at the controller. In the path generation phase, the secure and trusted controller computes interdomain paths based on the globally trusted policy and the global network view securely transmitted by the SSL protocol and distributes interdomain path segments via SSL as shown in Figure 10(e). Therefore, trusted interdomain path segments can be obtained securely among SDN domains. In conclusion, BPFC-SDNs can realize secure and trusted policy forwarding control.Figure 10
The security and trustworthiness analysis of BPFC-SDNs.However, PbSA transmits policy constraints across domains through the data plane without security measures and thus faces common security threats in the data plane such as injection, tampering, and discarding. When the policy transfer token is hijacked, tampered, or injected, it will invalidate the SOURCE forwarding policy, which will bring serious security risks to the policy-oriented forwarding control.


### 7.3. Performance Evaluation of BPFC-SDNs

To evaluate the performance of BPFC-SDNs, an experimental topology is built as shown in Figure11. Six SDN domains are connected sequentially, with four switches in the data plane of each domain. The experiments evaluate the performance of BPFC-SDNs in five aspects: blockchain-based policy sharing performance evaluation, controller latency, end-to-end flow establishment latency, flow establishment latency at different policy scales, and throughput, roughly in the order of the workflow of BPFC-SDNs.Figure 11
The experimental topology for BPFC-SDN performance evaluation.


#### 7.3.1. Blockchain-Based Policy Sharing Performance Evaluation

In the information-sharing phase, BPFC-SDNs implement domain information and policy sharing based on blockchain. In order to evaluate the performance of blockchain-based data sharing in BPFC-SDNs, taking policy sharing as an example, the experiment tested the total time consumed by blockchain to execute policy publishing when uploading different numbers of policies in a batch. Each transaction in the experiment contains 10 policies, and the block timeout is set to 1 s. The experimental results are shown in Figure12.Figure 12
The execution time of blockchain-based policy sharing.The experimental results in Figure12 show that as the batch upload policies increase from 500 to 3000, the total time consumed for the blockchain to execute policy publishing rises slowly from about 1.13 s to about 1.78 s, and when the number of batch upload policies reaches 3500, the latency rises steeply to 2.91 s. This is because when the number of transactions is small, the waiting time for a block timeout takes up more time for the transaction confirmation, such as PE = 500, and the waiting time for creating a block account for about 88% of the total time spent on transaction confirming. With the number of bath upload policies increasing, the time for transaction calculation gradually increases, which makes the latency on the chain grow slowly. However, when the batch upload policies exceed the capacity of one block, the extra policies have to wait for a new block to be generated, thus leading to a steep increase in the latency, and the increased latency is mainly for the new block waiting latency, which is about 1 s.Figure13 shows the throughput rate of the blockchain and the average latency of publishing a policy when uploading different numbers of policies in a batch during the experiment. It can be found that when the batch uploading policies reach 3000, the average policy publishing latency of the blockchain is the smallest, about 0.6 ms, and the throughput rate is the largest, at about 1685 policies per second.Figure 13
The throughput and average latency of blockchain-based policy sharing.The time consumption of the domain information-sharing phase is similar to that of the policies and is not shown here again. It is worth noting that although the latency of blockchain domain information and policy sharing is in the order of seconds, this is acceptable because BPFC-SDNs isolate data sharing from forwarding control, and blockchain-based data-sharing does not affect dataflow forwarding control. At the same time, the result also confirms that the forwarding control function should not be executed directly by the blockchain owing to the second-level latency.


#### 7.3.2. Control Plane Latency

In the policy collaboration phase, when a new cross-domain dataflow arrives, it triggers the controller of BPFC-SDNs to perform policy collaboration and calculate the forwarding path. To test the average processing latency of every controller in BPFC-SDNs for implementing cross-domain dataflow forwarding, a dataflow from h1 in AS1 to h6 in AS6 was constructed by Scapy in the experimental topology, and the experiment was repeated 30 times, and the results were averaged. The average latency of each controller is shown in Figure14, where the controller latency is the time elapsed from the time the controller receives the packet_in request for the dataflow to the time it issues the flow_mod messages.Figure 14
The average latency of each controller.The experimental results show three characteristics. First, both baseline and BPFC-SDNs have higher latency in the source domain and lower latency in the transmission domain. The source domain latency of baseline and BPFC-SDNs is higher than the transmission domain latency by about 1.58 ms and 3.77 ms, respectively. This is because both baseline and BPFC-SDNs need to calculate the global interdomain path segments in the source domain based on the global network view, while the transmission domain only needs to calculate the intradomain forwarding path based on the path segment distributed to it. Therefore, when a new dataflow is received, the source domain controller latency of baseline and BPFC-SDNs is always higher than that of the transmission domains. Second, the source domain controller latency of BPFC-SDNs is higher than that of baseline, while the transmission domain latency is basically the same as that of baseline, about 2.64 ms. This is because baseline calculates routes directly based on interdomain topology, while the source domain controller of BPFC-SDNs needs to perform policy collaboration and calculate interdomain path segments based on global collaboration policy. Therefore, the extended policy forwarding control function of BPFC-SDNs introduces a latency of about 2.2 ms in the source domain compared to baseline. Third, the controller latency in the source domain of baseline and BPFC-SDNs is higher than that of PbSA, but the controller latency in the transmission domain is lower than that of PbSA. This is because PbSA needs to calculate policy routes in each domain. As described in [5], PbSA computes only the next hop (AS level) in each domain according to policies.


#### 7.3.3. End-to-End Flow Establishment Latency

To evaluate the impact of BPFC-SDNs on the data forwarding latency and the scalability of BPFC-SDNs, experiments were conducted to construct dataflows from h1 in AS1 to h2 in AS2, h3 in AS3, h4 in AS4, h5 in AS5, and h6 in AS6 by Scapy to test the end-to-end flow establishment latency of baseline and BPFC-SDNs across different numbers of domains. We then compared the flow establishment latency of baseline and BPFC-SDNs with that of PbSA. The end-to-end flow establishment latency refers to the time from the source host to the destination host for the first packet of the dataflow. Each cross-domain dataflow was sent 30 times repeatedly, and the dataflow establishment latencies were averaged as shown in Figure15. At the same time, the latency growth rates are shown in Figure 16. The number of SDN domains 2, 3, 4, 5, and 6 in Figures 14 and 15 represents that the flow is transferred through (AS1 and AS2), (AS1, AS2, and AS3), (AS1, AS2, AS3, and AS4), (AS1, AS2, AS3, AS4, and AS5), and (AS1, AS2, AS3, AS4, AS5, and AS6), respectively.Figure 15
The flow establishment latency.Figure 16
The growth rate of flow establishment latency. (a) Latency growth rate of BPFC-SDNs compared to baseline. (b) Latency growth rate with the varying number of domains.
(a)(b)The experimental results show three features. First, as shown in Figure15, the end-to-end flow establishment latency of BPFC-SDNs increases linearly with the number of domains but within an acceptable range. This indicates that BPFC-SDNs can meet the latency sensitivity requirements of forwarding control. Secondly, Figure 16(a) shows that compared with baseline, under the same network scale, the latency increase rate of BPFC-SDNs gradually decreases with the increase of the number of domains, from 25% for two domains to 9% for six domains. This indicates that BPFC-SDNs have good scalability because with the number of domains increasing, the latency increase rate tends to decrease. Third, with the growth of the number of domains, the latency difference between BPFC-SDNs and PbSA gradually decreases, from 1.53 ms slower than PbSA for two domains to 1.4 ms faster than PbSA for six domains, as shown in Figure 15. This is due to the fact that PbSA performs policy token upload, policy retrieval, interdomain route calculation, and other operations in each domain, so the transmission latency of each domain in PbSA is always larger than that of BPFC-SDNs. It is also calculated that with the growth of the number of domains, the latency growth rate of BPFC-SDNs is always lower than that of PbSA, as shown in Figure 16(b), indicating that BPFC-SDNs have better scalability compared with PbSA.


#### 7.3.4. The Impact of the Policy Scale on the Flow Establishment Latency

The policy scale affects policy collaboration, which in turn affects forwarding control latency. Therefore, in the experiment environment of six domains, the effect of the policy scale on end-to-end flow establishment latency is tested by setting different numbers of policies in each domain.The experimental results in Figure17 show that the flow establishment latency of BPFC-SDNs is gradually increasing with the increase of the policy scale. For each increase of 50 policies per domain, the flow establishment latency increases by about 0.5 ms on average, and the average latency growth rate is about 2%.Figure 17
The flow establishment latency with the varying number of policies.


#### 7.3.5. The Throughput of BPFC-SDNs

Compared to baseline, BPFC-SDNs have an extended policy forwarding control function. To evaluate the impact of the policy forwarding control function on system throughput, cross-domain dataflows were injected into the network at different rates via custom packet sending scripts of Scapy in baseline and BPFC-SDNs, respectively. The experimental environment was six domains, each containing 100 policies, and each experiment lasted 10 s and was repeated 10 times. The experiments measured the flow establishment rate at different flow request rates, and the results are shown in Figure18.Figure 18
The flow establishment rate with the varying flow request rate.The experimental results show that the throughput of the baseline is about 4589 flow/s, and the throughput of BPFC-SDNs is about 3932 flow/s, which is about 14.3% lower than that of the baseline. Therefore, BPFC-SDNs sacrifice some network performance while providing secure and trusted cross-domain forwarding control functions. This is mainly because of the performance bottleneck brought by the extended policy collaboration function in the source domain. At present, there are special methods to improve the efficiency of policy retrieval [48]. In addition, due to the irrelevance of domain information and policies in each domain, attribute mapping and policy matching can be performed in parallel. These will be helpful in improving the throughput of BPFC-SDNs.


## 7.1. Functional Effectiveness Testing of BPFC-SDNs

To verify the effectiveness of BPFC-SDNs, two experiments are designed. The validity verification experiment of the policy-oriented forwarding control function, by configuring the forwarding control policy in the source domain, verifies whether the BPFC-SDNs can forward the dataflow based on the path constraints in the policy. To verify the validity of the information-sharing and policy collaboration function, the experiment configures a policy in a transmission domain conflicting with the source domain and tests whether BPFC-SDNs can forward dataflow based on the global policy constraints. In order to realize the experiments, the topology shown in Figure2 (Section 4.3) is built. The attribute information of each domain is shown in the tables in Figure 2, and the whole link bandwidth is set to 1 Gbit/s.


### 7.1.1. Validation of Policy-Oriented Forwarding Control Function

The experiment assumes that to improve the security of file transmission, AS1 formulates a policypolicyAS11 (see Section 4.3), which enforces that the FTP traffic sent from device_i to AS5 should be forwarded by a domain with a security level greater than or equal to 3. The experiment utilized the packet manipulation program Scapy to construct FTP packets sent from device_i to device_j at a rate of 30 package/s. The experiment lasted 10 seconds each time and was repeated 100 times. The data packets passing through the source domain egress switches S12, S13, and S14 were captured by TCPDUMP. The packet sending rate of each switch was counted under the baseline system and BPFC-SDNs, respectively, and the results are shown in Figure 8.Figure 8
The flow forwarding paths under baseline and BPFC-SDNs when installpolicyAS11.The experimental results show that the baseline system forwards dataflows to S12, S13, and S14, corresponding to the interdomain forwarding path AS1-AS2-AS5, AS1-AS3-AS5, and AS1-AS4-AS5, respectively. However, BPFC-SDNs forward the dataflow only to S13 and S14; namely, the dataflow forwarding paths in BPFC-SDNs are AS1-AS3-AS5 and AS1-AS4-AS5, which complies with the path constraint that the security level of the transmission domain should be greater than or equal to 3 in the policypolicyAS11. This indicates that in the same experimental environment, the baseline system cannot realize the policy-oriented forwarding control, while the BPFC-SDNs can realize the policy-oriented forwarding control function.


### 7.1.2. Validation of Information-Sharing and Policy Collaboration Function

On the basis of the experiment in the previous step, AS4 as a domain with a high-security level formulates a policypolicyAS44 (Section 4.3) to refuse to carry dataflows originating from devices with a security level less than or equal to 3 so as to ensure the path isolation of high-security level data transmission. The experiment still constructed FTP packets sent from device_i to device_j at a rate of 30 packet/s. Moreover, each experiment lasted for 10 seconds and was repeated 100 times. Figure 9 presents the packet sending rate of egress switches S12, S13, and S14 in AS1, respectively, under baseline and BPFC-SDNs.Figure 9
The flow forwarding paths under baseline and BPFC-SDNs when installpolicyAS11 and policyAS44.The experimental results show that the baseline system still forwards data packets to AS2, AS3, and AS4, and the data forwarding is not affected by the policypolicyAS44. However, all traffic of BPFC-SDNs is forwarded to AS3, which satisfies both policyAS11 and policyAS44. Compared with the previous experiment, the source domain in BPFC-SDNs no longer forwards dataflows to AS4, which shows that the source domain has taken the policy constraints of the transmission domain AS4 into consideration when calculating the interdomain routing, thereby avoiding data transfer interruption. Additionally, in PbSA, policies are transmitted domain by domain through policy transfer tokens, and the source domain cannot know the policy of the transport domain. Once there is a policy conflict between the transmission domain and the source domain, data transmission will be interrupted. Therefore, the experimental result verifies that the information-sharing and policy collaboration function of BPFC-SDNs can effectively solve the cross-domain policy agnostic and policy conflict problems.


## 7.1.1. Validation of Policy-Oriented Forwarding Control Function

The experiment assumes that to improve the security of file transmission, AS1 formulates a policypolicyAS11 (see Section 4.3), which enforces that the FTP traffic sent from device_i to AS5 should be forwarded by a domain with a security level greater than or equal to 3. The experiment utilized the packet manipulation program Scapy to construct FTP packets sent from device_i to device_j at a rate of 30 package/s. The experiment lasted 10 seconds each time and was repeated 100 times. The data packets passing through the source domain egress switches S12, S13, and S14 were captured by TCPDUMP. The packet sending rate of each switch was counted under the baseline system and BPFC-SDNs, respectively, and the results are shown in Figure 8.Figure 8
The flow forwarding paths under baseline and BPFC-SDNs when installpolicyAS11.The experimental results show that the baseline system forwards dataflows to S12, S13, and S14, corresponding to the interdomain forwarding path AS1-AS2-AS5, AS1-AS3-AS5, and AS1-AS4-AS5, respectively. However, BPFC-SDNs forward the dataflow only to S13 and S14; namely, the dataflow forwarding paths in BPFC-SDNs are AS1-AS3-AS5 and AS1-AS4-AS5, which complies with the path constraint that the security level of the transmission domain should be greater than or equal to 3 in the policypolicyAS11. This indicates that in the same experimental environment, the baseline system cannot realize the policy-oriented forwarding control, while the BPFC-SDNs can realize the policy-oriented forwarding control function.


## 7.1.2. Validation of Information-Sharing and Policy Collaboration Function

On the basis of the experiment in the previous step, AS4 as a domain with a high-security level formulates a policypolicyAS44 (Section 4.3) to refuse to carry dataflows originating from devices with a security level less than or equal to 3 so as to ensure the path isolation of high-security level data transmission. The experiment still constructed FTP packets sent from device_i to device_j at a rate of 30 packet/s. Moreover, each experiment lasted for 10 seconds and was repeated 100 times. Figure 9 presents the packet sending rate of egress switches S12, S13, and S14 in AS1, respectively, under baseline and BPFC-SDNs.Figure 9
The flow forwarding paths under baseline and BPFC-SDNs when installpolicyAS11 and policyAS44.The experimental results show that the baseline system still forwards data packets to AS2, AS3, and AS4, and the data forwarding is not affected by the policypolicyAS44. However, all traffic of BPFC-SDNs is forwarded to AS3, which satisfies both policyAS11 and policyAS44. Compared with the previous experiment, the source domain in BPFC-SDNs no longer forwards dataflows to AS4, which shows that the source domain has taken the policy constraints of the transmission domain AS4 into consideration when calculating the interdomain routing, thereby avoiding data transfer interruption. Additionally, in PbSA, policies are transmitted domain by domain through policy transfer tokens, and the source domain cannot know the policy of the transport domain. Once there is a policy conflict between the transmission domain and the source domain, data transmission will be interrupted. Therefore, the experimental result verifies that the information-sharing and policy collaboration function of BPFC-SDNs can effectively solve the cross-domain policy agnostic and policy conflict problems.


## 7.2. Security and Trustworthiness Analysis of BPFC-SDNs

According to the overall workflow of BPFC-SDNs, it is analysed that BPFC-SDNs can realize secure and trusted policy-oriented forwarding control.In the information-sharing phase, the transaction signature verification mechanism ensures the authenticity and integrity of domain information and policies, as shown in Figure10(a); the consensus mechanism ensures the nonrepudiation and traceability of domain information and policies as shown in Figure 10(b); the blockchain contract data are synchronized to the controller local cache in a secure and trusted environment of the controller as shown in Figure 10(c). Thus, BPFC-SDNs achieve secure and trusted domain information and policy sharing in the information-sharing phase. In the policy collaboration phase, secure and trusted controllers can implement policy collaboration based on globally trusted domain information and policies as shown in Figure 10(d). Therefore, BPFC-SDNs can achieve secure and trusted cross-domain policy collaboration locally at the controller. In the path generation phase, the secure and trusted controller computes interdomain paths based on the globally trusted policy and the global network view securely transmitted by the SSL protocol and distributes interdomain path segments via SSL as shown in Figure 10(e). Therefore, trusted interdomain path segments can be obtained securely among SDN domains. In conclusion, BPFC-SDNs can realize secure and trusted policy forwarding control.Figure 10
The security and trustworthiness analysis of BPFC-SDNs.However, PbSA transmits policy constraints across domains through the data plane without security measures and thus faces common security threats in the data plane such as injection, tampering, and discarding. When the policy transfer token is hijacked, tampered, or injected, it will invalidate the SOURCE forwarding policy, which will bring serious security risks to the policy-oriented forwarding control.


## 7.3. Performance Evaluation of BPFC-SDNs

To evaluate the performance of BPFC-SDNs, an experimental topology is built as shown in Figure11. Six SDN domains are connected sequentially, with four switches in the data plane of each domain. The experiments evaluate the performance of BPFC-SDNs in five aspects: blockchain-based policy sharing performance evaluation, controller latency, end-to-end flow establishment latency, flow establishment latency at different policy scales, and throughput, roughly in the order of the workflow of BPFC-SDNs.Figure 11
The experimental topology for BPFC-SDN performance evaluation.


### 7.3.1. Blockchain-Based Policy Sharing Performance Evaluation

In the information-sharing phase, BPFC-SDNs implement domain information and policy sharing based on blockchain. In order to evaluate the performance of blockchain-based data sharing in BPFC-SDNs, taking policy sharing as an example, the experiment tested the total time consumed by blockchain to execute policy publishing when uploading different numbers of policies in a batch. Each transaction in the experiment contains 10 policies, and the block timeout is set to 1 s. The experimental results are shown in Figure12.Figure 12
The execution time of blockchain-based policy sharing.The experimental results in Figure12 show that as the batch upload policies increase from 500 to 3000, the total time consumed for the blockchain to execute policy publishing rises slowly from about 1.13 s to about 1.78 s, and when the number of batch upload policies reaches 3500, the latency rises steeply to 2.91 s. This is because when the number of transactions is small, the waiting time for a block timeout takes up more time for the transaction confirmation, such as PE = 500, and the waiting time for creating a block account for about 88% of the total time spent on transaction confirming. With the number of bath upload policies increasing, the time for transaction calculation gradually increases, which makes the latency on the chain grow slowly. However, when the batch upload policies exceed the capacity of one block, the extra policies have to wait for a new block to be generated, thus leading to a steep increase in the latency, and the increased latency is mainly for the new block waiting latency, which is about 1 s.Figure13 shows the throughput rate of the blockchain and the average latency of publishing a policy when uploading different numbers of policies in a batch during the experiment. It can be found that when the batch uploading policies reach 3000, the average policy publishing latency of the blockchain is the smallest, about 0.6 ms, and the throughput rate is the largest, at about 1685 policies per second.Figure 13
The throughput and average latency of blockchain-based policy sharing.The time consumption of the domain information-sharing phase is similar to that of the policies and is not shown here again. It is worth noting that although the latency of blockchain domain information and policy sharing is in the order of seconds, this is acceptable because BPFC-SDNs isolate data sharing from forwarding control, and blockchain-based data-sharing does not affect dataflow forwarding control. At the same time, the result also confirms that the forwarding control function should not be executed directly by the blockchain owing to the second-level latency.


### 7.3.2. Control Plane Latency

In the policy collaboration phase, when a new cross-domain dataflow arrives, it triggers the controller of BPFC-SDNs to perform policy collaboration and calculate the forwarding path. To test the average processing latency of every controller in BPFC-SDNs for implementing cross-domain dataflow forwarding, a dataflow from h1 in AS1 to h6 in AS6 was constructed by Scapy in the experimental topology, and the experiment was repeated 30 times, and the results were averaged. The average latency of each controller is shown in Figure14, where the controller latency is the time elapsed from the time the controller receives the packet_in request for the dataflow to the time it issues the flow_mod messages.Figure 14
The average latency of each controller.The experimental results show three characteristics. First, both baseline and BPFC-SDNs have higher latency in the source domain and lower latency in the transmission domain. The source domain latency of baseline and BPFC-SDNs is higher than the transmission domain latency by about 1.58 ms and 3.77 ms, respectively. This is because both baseline and BPFC-SDNs need to calculate the global interdomain path segments in the source domain based on the global network view, while the transmission domain only needs to calculate the intradomain forwarding path based on the path segment distributed to it. Therefore, when a new dataflow is received, the source domain controller latency of baseline and BPFC-SDNs is always higher than that of the transmission domains. Second, the source domain controller latency of BPFC-SDNs is higher than that of baseline, while the transmission domain latency is basically the same as that of baseline, about 2.64 ms. This is because baseline calculates routes directly based on interdomain topology, while the source domain controller of BPFC-SDNs needs to perform policy collaboration and calculate interdomain path segments based on global collaboration policy. Therefore, the extended policy forwarding control function of BPFC-SDNs introduces a latency of about 2.2 ms in the source domain compared to baseline. Third, the controller latency in the source domain of baseline and BPFC-SDNs is higher than that of PbSA, but the controller latency in the transmission domain is lower than that of PbSA. This is because PbSA needs to calculate policy routes in each domain. As described in [5], PbSA computes only the next hop (AS level) in each domain according to policies.


### 7.3.3. End-to-End Flow Establishment Latency

To evaluate the impact of BPFC-SDNs on the data forwarding latency and the scalability of BPFC-SDNs, experiments were conducted to construct dataflows from h1 in AS1 to h2 in AS2, h3 in AS3, h4 in AS4, h5 in AS5, and h6 in AS6 by Scapy to test the end-to-end flow establishment latency of baseline and BPFC-SDNs across different numbers of domains. We then compared the flow establishment latency of baseline and BPFC-SDNs with that of PbSA. The end-to-end flow establishment latency refers to the time from the source host to the destination host for the first packet of the dataflow. Each cross-domain dataflow was sent 30 times repeatedly, and the dataflow establishment latencies were averaged as shown in Figure15. At the same time, the latency growth rates are shown in Figure 16. The number of SDN domains 2, 3, 4, 5, and 6 in Figures 14 and 15 represents that the flow is transferred through (AS1 and AS2), (AS1, AS2, and AS3), (AS1, AS2, AS3, and AS4), (AS1, AS2, AS3, AS4, and AS5), and (AS1, AS2, AS3, AS4, AS5, and AS6), respectively.Figure 15
The flow establishment latency.Figure 16
The growth rate of flow establishment latency. (a) Latency growth rate of BPFC-SDNs compared to baseline. (b) Latency growth rate with the varying number of domains.
(a)(b)The experimental results show three features. First, as shown in Figure15, the end-to-end flow establishment latency of BPFC-SDNs increases linearly with the number of domains but within an acceptable range. This indicates that BPFC-SDNs can meet the latency sensitivity requirements of forwarding control. Secondly, Figure 16(a) shows that compared with baseline, under the same network scale, the latency increase rate of BPFC-SDNs gradually decreases with the increase of the number of domains, from 25% for two domains to 9% for six domains. This indicates that BPFC-SDNs have good scalability because with the number of domains increasing, the latency increase rate tends to decrease. Third, with the growth of the number of domains, the latency difference between BPFC-SDNs and PbSA gradually decreases, from 1.53 ms slower than PbSA for two domains to 1.4 ms faster than PbSA for six domains, as shown in Figure 15. This is due to the fact that PbSA performs policy token upload, policy retrieval, interdomain route calculation, and other operations in each domain, so the transmission latency of each domain in PbSA is always larger than that of BPFC-SDNs. It is also calculated that with the growth of the number of domains, the latency growth rate of BPFC-SDNs is always lower than that of PbSA, as shown in Figure 16(b), indicating that BPFC-SDNs have better scalability compared with PbSA.


### 7.3.4. The Impact of the Policy Scale on the Flow Establishment Latency

The policy scale affects policy collaboration, which in turn affects forwarding control latency. Therefore, in the experiment environment of six domains, the effect of the policy scale on end-to-end flow establishment latency is tested by setting different numbers of policies in each domain.The experimental results in Figure17 show that the flow establishment latency of BPFC-SDNs is gradually increasing with the increase of the policy scale. For each increase of 50 policies per domain, the flow establishment latency increases by about 0.5 ms on average, and the average latency growth rate is about 2%.Figure 17
The flow establishment latency with the varying number of policies.


### 7.3.5. The Throughput of BPFC-SDNs

Compared to baseline, BPFC-SDNs have an extended policy forwarding control function. To evaluate the impact of the policy forwarding control function on system throughput, cross-domain dataflows were injected into the network at different rates via custom packet sending scripts of Scapy in baseline and BPFC-SDNs, respectively. The experimental environment was six domains, each containing 100 policies, and each experiment lasted 10 s and was repeated 10 times. The experiments measured the flow establishment rate at different flow request rates, and the results are shown in Figure18.Figure 18
The flow establishment rate with the varying flow request rate.The experimental results show that the throughput of the baseline is about 4589 flow/s, and the throughput of BPFC-SDNs is about 3932 flow/s, which is about 14.3% lower than that of the baseline. Therefore, BPFC-SDNs sacrifice some network performance while providing secure and trusted cross-domain forwarding control functions. This is mainly because of the performance bottleneck brought by the extended policy collaboration function in the source domain. At present, there are special methods to improve the efficiency of policy retrieval [48]. In addition, due to the irrelevance of domain information and policies in each domain, attribute mapping and policy matching can be performed in parallel. These will be helpful in improving the throughput of BPFC-SDNs.


## 7.3.1. Blockchain-Based Policy Sharing Performance Evaluation

In the information-sharing phase, BPFC-SDNs implement domain information and policy sharing based on blockchain. In order to evaluate the performance of blockchain-based data sharing in BPFC-SDNs, taking policy sharing as an example, the experiment tested the total time consumed by blockchain to execute policy publishing when uploading different numbers of policies in a batch. Each transaction in the experiment contains 10 policies, and the block timeout is set to 1 s. The experimental results are shown in Figure12.Figure 12
The execution time of blockchain-based policy sharing.The experimental results in Figure12 show that as the batch upload policies increase from 500 to 3000, the total time consumed for the blockchain to execute policy publishing rises slowly from about 1.13 s to about 1.78 s, and when the number of batch upload policies reaches 3500, the latency rises steeply to 2.91 s. This is because when the number of transactions is small, the waiting time for a block timeout takes up more time for the transaction confirmation, such as PE = 500, and the waiting time for creating a block account for about 88% of the total time spent on transaction confirming. With the number of bath upload policies increasing, the time for transaction calculation gradually increases, which makes the latency on the chain grow slowly. However, when the batch upload policies exceed the capacity of one block, the extra policies have to wait for a new block to be generated, thus leading to a steep increase in the latency, and the increased latency is mainly for the new block waiting latency, which is about 1 s.Figure13 shows the throughput rate of the blockchain and the average latency of publishing a policy when uploading different numbers of policies in a batch during the experiment. It can be found that when the batch uploading policies reach 3000, the average policy publishing latency of the blockchain is the smallest, about 0.6 ms, and the throughput rate is the largest, at about 1685 policies per second.Figure 13
The throughput and average latency of blockchain-based policy sharing.The time consumption of the domain information-sharing phase is similar to that of the policies and is not shown here again. It is worth noting that although the latency of blockchain domain information and policy sharing is in the order of seconds, this is acceptable because BPFC-SDNs isolate data sharing from forwarding control, and blockchain-based data-sharing does not affect dataflow forwarding control. At the same time, the result also confirms that the forwarding control function should not be executed directly by the blockchain owing to the second-level latency.


## 7.3.2. Control Plane Latency

In the policy collaboration phase, when a new cross-domain dataflow arrives, it triggers the controller of BPFC-SDNs to perform policy collaboration and calculate the forwarding path. To test the average processing latency of every controller in BPFC-SDNs for implementing cross-domain dataflow forwarding, a dataflow from h1 in AS1 to h6 in AS6 was constructed by Scapy in the experimental topology, and the experiment was repeated 30 times, and the results were averaged. The average latency of each controller is shown in Figure14, where the controller latency is the time elapsed from the time the controller receives the packet_in request for the dataflow to the time it issues the flow_mod messages.Figure 14
The average latency of each controller.The experimental results show three characteristics. First, both baseline and BPFC-SDNs have higher latency in the source domain and lower latency in the transmission domain. The source domain latency of baseline and BPFC-SDNs is higher than the transmission domain latency by about 1.58 ms and 3.77 ms, respectively. This is because both baseline and BPFC-SDNs need to calculate the global interdomain path segments in the source domain based on the global network view, while the transmission domain only needs to calculate the intradomain forwarding path based on the path segment distributed to it. Therefore, when a new dataflow is received, the source domain controller latency of baseline and BPFC-SDNs is always higher than that of the transmission domains. Second, the source domain controller latency of BPFC-SDNs is higher than that of baseline, while the transmission domain latency is basically the same as that of baseline, about 2.64 ms. This is because baseline calculates routes directly based on interdomain topology, while the source domain controller of BPFC-SDNs needs to perform policy collaboration and calculate interdomain path segments based on global collaboration policy. Therefore, the extended policy forwarding control function of BPFC-SDNs introduces a latency of about 2.2 ms in the source domain compared to baseline. Third, the controller latency in the source domain of baseline and BPFC-SDNs is higher than that of PbSA, but the controller latency in the transmission domain is lower than that of PbSA. This is because PbSA needs to calculate policy routes in each domain. As described in [5], PbSA computes only the next hop (AS level) in each domain according to policies.


## 7.3.3. End-to-End Flow Establishment Latency

To evaluate the impact of BPFC-SDNs on the data forwarding latency and the scalability of BPFC-SDNs, experiments were conducted to construct dataflows from h1 in AS1 to h2 in AS2, h3 in AS3, h4 in AS4, h5 in AS5, and h6 in AS6 by Scapy to test the end-to-end flow establishment latency of baseline and BPFC-SDNs across different numbers of domains. We then compared the flow establishment latency of baseline and BPFC-SDNs with that of PbSA. The end-to-end flow establishment latency refers to the time from the source host to the destination host for the first packet of the dataflow. Each cross-domain dataflow was sent 30 times repeatedly, and the dataflow establishment latencies were averaged as shown in Figure15. At the same time, the latency growth rates are shown in Figure 16. The number of SDN domains 2, 3, 4, 5, and 6 in Figures 14 and 15 represents that the flow is transferred through (AS1 and AS2), (AS1, AS2, and AS3), (AS1, AS2, AS3, and AS4), (AS1, AS2, AS3, AS4, and AS5), and (AS1, AS2, AS3, AS4, AS5, and AS6), respectively.Figure 15
The flow establishment latency.Figure 16
The growth rate of flow establishment latency. (a) Latency growth rate of BPFC-SDNs compared to baseline. (b) Latency growth rate with the varying number of domains.
(a)(b)The experimental results show three features. First, as shown in Figure15, the end-to-end flow establishment latency of BPFC-SDNs increases linearly with the number of domains but within an acceptable range. This indicates that BPFC-SDNs can meet the latency sensitivity requirements of forwarding control. Secondly, Figure 16(a) shows that compared with baseline, under the same network scale, the latency increase rate of BPFC-SDNs gradually decreases with the increase of the number of domains, from 25% for two domains to 9% for six domains. This indicates that BPFC-SDNs have good scalability because with the number of domains increasing, the latency increase rate tends to decrease. Third, with the growth of the number of domains, the latency difference between BPFC-SDNs and PbSA gradually decreases, from 1.53 ms slower than PbSA for two domains to 1.4 ms faster than PbSA for six domains, as shown in Figure 15. This is due to the fact that PbSA performs policy token upload, policy retrieval, interdomain route calculation, and other operations in each domain, so the transmission latency of each domain in PbSA is always larger than that of BPFC-SDNs. It is also calculated that with the growth of the number of domains, the latency growth rate of BPFC-SDNs is always lower than that of PbSA, as shown in Figure 16(b), indicating that BPFC-SDNs have better scalability compared with PbSA.


## 7.3.4. The Impact of the Policy Scale on the Flow Establishment Latency

The policy scale affects policy collaboration, which in turn affects forwarding control latency. Therefore, in the experiment environment of six domains, the effect of the policy scale on end-to-end flow establishment latency is tested by setting different numbers of policies in each domain.The experimental results in Figure17 show that the flow establishment latency of BPFC-SDNs is gradually increasing with the increase of the policy scale. For each increase of 50 policies per domain, the flow establishment latency increases by about 0.5 ms on average, and the average latency growth rate is about 2%.Figure 17
The flow establishment latency with the varying number of policies.


## 7.3.5. The Throughput of BPFC-SDNs

Compared to baseline, BPFC-SDNs have an extended policy forwarding control function. To evaluate the impact of the policy forwarding control function on system throughput, cross-domain dataflows were injected into the network at different rates via custom packet sending scripts of Scapy in baseline and BPFC-SDNs, respectively. The experimental environment was six domains, each containing 100 policies, and each experiment lasted 10 s and was repeated 10 times. The experiments measured the flow establishment rate at different flow request rates, and the results are shown in Figure18.Figure 18
The flow establishment rate with the varying flow request rate.The experimental results show that the throughput of the baseline is about 4589 flow/s, and the throughput of BPFC-SDNs is about 3932 flow/s, which is about 14.3% lower than that of the baseline. Therefore, BPFC-SDNs sacrifice some network performance while providing secure and trusted cross-domain forwarding control functions. This is mainly because of the performance bottleneck brought by the extended policy collaboration function in the source domain. At present, there are special methods to improve the efficiency of policy retrieval [48]. In addition, due to the irrelevance of domain information and policies in each domain, attribute mapping and policy matching can be performed in parallel. These will be helpful in improving the throughput of BPFC-SDNs.


## 8. Summary

In this article, a novel forwarding control mechanism called BPFC-SDNs was proposed for the SDN interdomain. An attribute-based policy paradigm was defined for fine-grained control of the flow forwarding path. Blockchain was integrated into the SDN control plane to provide secure, trusted, and high-efficiency domain information and policy sharing among SDN domains. Furthermore, we devised a separate forwarding control functional model to avoid the high latency and complicated calculation introduced by blockchain. Finally, the BPFC-SDNs were verified, and the results indicate that the forwarding control function is valid and satisfies the latency sensitivity.



---

*Source: 1104565-2023-01-25.xml*