# Facing the Reality: Validation of Energy Saving Mechanisms on a Testbed

Authors: Edion Tego; Filip Idzikowski; Luca Chiaraviglio; Angelo Coiro; Francesco Matera
Journal: Journal of Electrical and Computer Engineering
 (2014)

---

## Abstract

Two energy saving approaches, called Fixed Upper Fixed Lower (FUFL) and Dynamic Upper Fixed Lower (DUFL), switching off idle optical Gigabit Ethernet (GbE) interfaces during low traffic periods, have been implemented on a testbed. We show on a simple network scenario that energy can be saved using off-the-shelf equipment not explicitly designed for dynamic on/off operation. No packet loss is experienced in our experiments. We indicate the
need for faster access to routers in order to perform the reconfiguration. This is particularly important for the more sophisticated energy saving approaches such as DUFL, since FUFL can be implemented locally.


---

## Body

## 1. Introduction

Variation of traffic over day and night in backbone networks offers the opportunity to save energy by deactivating some network devices (or their parts) in low-demand hours. Various approaches have been proposed in the literature (see [1]) for choosing the devices to be deactivated. However there is little work done on the actual implementation validating the potential problems that energy saving schemes may introduce. The challenges (corresponding to steps schematically depicted in Figure 1) include (
1
) accurate monitoring of traffic data; (
2
) timely triggering network reconfiguration; (
3
) fast calculation of the desired (energy-efficient) network configuration; (
4
) the reconfiguration itself including signaling and the time needed to activate or deactivate the devices and potentially to reroute traffic. Consequently, issues such as network stability, increased delay, jitter, or even packet loss may occur in the network, which is particularly crucial in the backbone.Figure 1
Steps needed for energy saving (traffic rates in Gbps, IP link capacities in number of GbE links).We validate the feasibility of implementation of two algorithms referred to as FUFL and DUFL [2–4] on a testbed. Our experiments show that it is possible to automatically and remotely switch on and off network interfaces in a dynamic manner using off-the-shelf equipment.The structure of the paper is as follows. We provide an overview of the work related to experimental activities for green core networks in Section2. The network scenario and methodology with algorithms' description are presented in Sections 3 and 4, respectively. Results are reported in Section 5. Eventually, Section 6 concludes this work.


## 2. Related Work

There is limited amount of related work dealing with implementation of energy saving mechanisms through selective on/off switching of network elements during periods of low load. To the best of our knowledge, it is basically limited to the following two activities.


### 2.1. MiDORi

Extensive work has been performed within the MiDORi (Multi (layer, path, and resources) Dynamically Optimized Routing) Network Technologies project [5]. The focus of the project is set on energy saving in the GbE network by selectively powering off network interfaces under hop-limit and bandwidth constraints. Powering on/off the whole transit routers or their parts is also considered. Starting with [6], the authors propose a solution which effectively creates all on/off combinations of links in the network. The solution that can carry the whole traffic and consumes the lowest power (considering moving some virtual routers between network nodes in order to power off also nodes) is selected. Beeler's algorithm is compared with the proposed any-order pattern algorithm, which additionally guarantees the hop number of the path lower than a given maximum and disjoint multiroute link divergence for reliable communications. While the algorithms are centrally executed by the Path Computation Element (PCE), the authors consider also link on/off control protocols. Extensions to Generalized Multiprotocol Label Switching (GMPLS) (related to Open Shortest Path First (OSPF), Resources Reservation Protocol (RSVP), and Link Management Protocol (LMP)) are proposed.Further publications related to MiDORi provide extensions of [6]. In particular, a prototype layer 2 (L2) switch is introduced in [7]. A depth-d algorithm is proposed and evaluated in a simulative way in [8]. The algorithm searches for the optimal configuration of the logical topology, where d determines the maximum number of links that are attempted to be switched off.The following contributions are made in [9]. First, clear steps for the energy saving in the MiDORi architecture are described, that is, (
1
) traffic monitoring; (
2
) calculation of energy-efficient logical topology by PCE; (
3
) reconfiguration of the network. Second, experiments on a 6-node 7-link network using the depth-d algorithm are conducted, and results similar to the ones from [8] are presented together with the calculation times in the range of 0.01–1
0
5 s for networks with 10–100 nodes and d in the range of 1–4. In [9] the authors report also total current of the prototype L2 switches (2.765–2.831 A) and mention Ethernet Virtual Local Area Networks (VLANs) as the way of controlling traffic paths.More details of the GbE L2 switch are provided in the block diagram presented in [10]. The switch can count traffic of each Label Switched Path (LSP) (VLAN) and each GbE link. The power consumption of the switch can be read via a command and a current meter. The presented switch has eight GbE links and is controlled remotely (power on/off state of each link and each fabric) using telnet via a Linux based control card which is one of the few parts constantly powered up. The authors demonstrated MiDORi on a fully meshed 6-node network testbed using the depth-1 algorithm with generic QoS restrictions. Six traffic generators/receivers were used; however the traffic assumptions were not detailed except for the fact that low traffic to high traffic ratio equals 1 : 5. The following steps are distinguished (extension from [9]). Step (
1
) is reading the traffic counters of each VLAN by the PCE and calculating average values. Step (
2
) is execution of the depth-1 algorithm at the PCE to obtain the logical topology and VLAN paths. Step (3
-
1) is powering on/off links in all switches (remotely by the PCE) according to the topology from Step (
2
). Step (
3
-
2
) is reconfiguration of the VLAN network topology according to the path calculation from Step (
2
).Execution of Steps(
1
)
–
(
3
-
2
) is repeated everyX minutes; however the authors do not report values assigned toX in [10]. Parallel and serial control of the switches were considered, with the parallel control taking significantly less time during both the traffic increase and traffic decrease (233.7–243.9 s versus 61.8–68.7 s, for serial versus parallel control, resp.). The results show that the calculation of the logical topology (Step 2) takes marginal time (0.004–0.006 s). Duration of Steps (
1
), (
3
-
1
), and (
3
-
2
) takes 23.8–112.7 s in the serial control and 7.2–29.0 s in the parallel control.The concept of Self-Organized Network (initially mentioned in [10]) is tackled in [11] using the depth-d algorithm again. The authors point out that the MiDORi GMPLS supports multiple layers, multiple paths, and multiple resources. They explain again the OSPF extension (relation between physical links and Traffic Engineering (TE) links), LMP extension (power on/off control function using the LMP ChannelStatus message with Ack and IP Control Channel always up), and RSVP extension (power control request in the Admin_Status object for LSP status flag). The authors mention the 16-port GbE switch, which they developed additionally to the 8-port switch presented in [10]. Demonstration on a 5-node 7-link network is performed showing that total switch power consumption can be reduced from 283.1 W to 276.1 W. The results of the reconfiguration times from [10] are also summarized in [11]. Additionally, the authors point out that their prototype switch does not support a “make-before-break” VLAN reconfiguration, and therefore data disruption occurs over the 29 s of VLAN reconfiguration. Eventually, the authors mention a MiDORi GMPLS optical switch, which they developed. It is also controlled via telnet by the PCE implemented on a small Linux box. The optical switch allows the authors to demonstrate the multilayer GMPLS signaling between a Lambda Switch Capable layer and a Layer 2 Switch Capable layer.In [12], the authors show the energy consumption (in Wh without specifying the considered time period and details of the traffic data) on a 4-node full mesh network. The energy saving reaches up to 23.8%.The experiments with the multilayer network using GbE switches and the optical switch from [11] are continued in [13] using the extension of the GMPLS. Namely, 4 Ethernet switches out of the considered 6 are connected to the optical switch. The demonstrated power saving (9.4 W corresponding to 6 ports) is low but shows that the MiDORi network technology is potentially feasible in a high speed and power consuming interface located in a large scale network environment.Eventually, the results are summarized in [14], which directly extends [6]. It includes Beeler's algorithm, the any-order pattern algorithm, their simulative evaluation on the National Science Foundation (NSF) network loaded with uniformly generated internode traffic, the GMPLS extensions (OSPF, RSVP, and LMP), the 8-port GbE switch development, and the same results as in [10] (fully meshed 6-node network testbed).The GMPLS extensions developed within the MiDORi project have been proposed to Internet Engineering Task Force (IETF) [15].


### 2.2. Experiments on the CARISMA Testbed

Researchers from Alcatel-Lucent Bell Labs and Universitat Politècnica de Catalunya (UPC) proposed the extension of GMPLS in [16] and performed experiments on the CARISMA testbed. More specifically, the authors of [16] propose to introduce a new bit “S” to the Resource Reservation Protocol-Traffic Engineering (RSVP-TE) Path and Resv messages. The bit “S” used jointly with the already existing bit “A” differentiates the following states of an OptoElectronic (OE) device (such as colored line card, transponder, or regenerator): up, idle, down, and damaged (see also [17]).The proposed extension has been evaluated on the CARISMA testbed available at UPC premises in Barcelona. The testbed was configured according to a Pan-European network composed of 16 nodes and 23 links with 10 bidirectional 100 Gbps wavelengths per link. 20 add/drop transponders and 10 regenerators were used at each node. The testbed was loaded with uniformly distributed lightpath requests according to the Poisson model with average holding time equal to 3 hours. The load was varied between 40 and 80 Erlang.Differentiated provisioning of connection requests is considered in [18] for gold, silver, and best-effort traffic. In this case, the “A” and “S” bits of the ADMIN Status object of the RSVP-TE Path message are used in the following way: (i) “A” = 0 and “S” = 0 indicate the OE devices in the up state which must be used to allocate gold requests; (ii) “A” = 0 and “S” = 1 indicate the OE devices in idle state which are needed to allocate silver requests; (iii) “A” = 1 and “S” = 0 indicate the OE devices in down state which can be used to allocate best-effort requests. Differently to [16], availability of regenerators and wavelengths on links is disseminated over the network using the proposed extension of the GMPLS Open Shortest Path First-Traffic Engineering (OSPF-TE) protocol. A new sub-TLV (Type Length Value) (named TSP Status) is introduced in the OSPF-TE opaque Link State Advertisements (LSAs) containing the number of up, down, and idle transponders in a node (a regenerator corresponds to two transponders). This sub-TLV is inserted into a Node Information Top Level TLV (type 5, see Figure 1 of  [18]). Using the OSPF-TE opaque LSAs, the PCE can populate its Traffic Engineering Database with wavelength and regenerator availability information, which is used for computation of end-to-end routes.The same topology as in [16] is used for the experimental study on the CARISMA testbed [18]. The service class distribution is 20, 30, and 50% for gold, silver, and best-effort traffic, respectively. Different shares of resources are reserved for different classes of traffic. Prereservation of resources is implemented in the PCE to avoid contention of resources among different lightpaths under establishment. Results on the blocking ratio, number of OE devices in up/idle/down states, and power consumption per active LSP are reported.The experimental activities on a testbed reported in [16, 18] were restricted to protocol information exchanges. Due to unavailability of transponders, the idle-up and down-up state transition times were assumed and not measured. The assumed transition times equal 20 ms and 60 s, respectively.


### 2.3. Our Contribution

We extend the related work described above in the following way. First, we implement the energy-saving schemes on the off-the-shelf equipment, demonstrating that the energy-saving is possible straightaway, even without extending GMPLS, even though FUFL and DUFL can be used also in the GMPLS environment. Second, we explicitly consider triggering events for calculation of new network configuration and consequently potential network reconfiguration for energy saving. Third, we implement different energy-saving algorithms (FUFL and DUFL) than the ones implemented in the MiDORi and CARISMA testbeds. FUFL is particularly interesting for the network operators [4]. Fourth, we implemented the “make-before-break” mechanism. Eventually, we used different traffic schemes focusing on evaluation of deactivation of parallel GbE links constituting one logical link. While very little information is provided about traffic assumption in MiDORi studies, the traffic defined as the number of lightpath requests and not “bps to be transported” is used in [16, 18]. Finally, the work in [16, 18] does not consider routing of IP traffic over the logical topology.


## 2.1. MiDORi

Extensive work has been performed within the MiDORi (Multi (layer, path, and resources) Dynamically Optimized Routing) Network Technologies project [5]. The focus of the project is set on energy saving in the GbE network by selectively powering off network interfaces under hop-limit and bandwidth constraints. Powering on/off the whole transit routers or their parts is also considered. Starting with [6], the authors propose a solution which effectively creates all on/off combinations of links in the network. The solution that can carry the whole traffic and consumes the lowest power (considering moving some virtual routers between network nodes in order to power off also nodes) is selected. Beeler's algorithm is compared with the proposed any-order pattern algorithm, which additionally guarantees the hop number of the path lower than a given maximum and disjoint multiroute link divergence for reliable communications. While the algorithms are centrally executed by the Path Computation Element (PCE), the authors consider also link on/off control protocols. Extensions to Generalized Multiprotocol Label Switching (GMPLS) (related to Open Shortest Path First (OSPF), Resources Reservation Protocol (RSVP), and Link Management Protocol (LMP)) are proposed.Further publications related to MiDORi provide extensions of [6]. In particular, a prototype layer 2 (L2) switch is introduced in [7]. A depth-d algorithm is proposed and evaluated in a simulative way in [8]. The algorithm searches for the optimal configuration of the logical topology, where d determines the maximum number of links that are attempted to be switched off.The following contributions are made in [9]. First, clear steps for the energy saving in the MiDORi architecture are described, that is, (
1
) traffic monitoring; (
2
) calculation of energy-efficient logical topology by PCE; (
3
) reconfiguration of the network. Second, experiments on a 6-node 7-link network using the depth-d algorithm are conducted, and results similar to the ones from [8] are presented together with the calculation times in the range of 0.01–1
0
5 s for networks with 10–100 nodes and d in the range of 1–4. In [9] the authors report also total current of the prototype L2 switches (2.765–2.831 A) and mention Ethernet Virtual Local Area Networks (VLANs) as the way of controlling traffic paths.More details of the GbE L2 switch are provided in the block diagram presented in [10]. The switch can count traffic of each Label Switched Path (LSP) (VLAN) and each GbE link. The power consumption of the switch can be read via a command and a current meter. The presented switch has eight GbE links and is controlled remotely (power on/off state of each link and each fabric) using telnet via a Linux based control card which is one of the few parts constantly powered up. The authors demonstrated MiDORi on a fully meshed 6-node network testbed using the depth-1 algorithm with generic QoS restrictions. Six traffic generators/receivers were used; however the traffic assumptions were not detailed except for the fact that low traffic to high traffic ratio equals 1 : 5. The following steps are distinguished (extension from [9]). Step (
1
) is reading the traffic counters of each VLAN by the PCE and calculating average values. Step (
2
) is execution of the depth-1 algorithm at the PCE to obtain the logical topology and VLAN paths. Step (3
-
1) is powering on/off links in all switches (remotely by the PCE) according to the topology from Step (
2
). Step (
3
-
2
) is reconfiguration of the VLAN network topology according to the path calculation from Step (
2
).Execution of Steps(
1
)
–
(
3
-
2
) is repeated everyX minutes; however the authors do not report values assigned toX in [10]. Parallel and serial control of the switches were considered, with the parallel control taking significantly less time during both the traffic increase and traffic decrease (233.7–243.9 s versus 61.8–68.7 s, for serial versus parallel control, resp.). The results show that the calculation of the logical topology (Step 2) takes marginal time (0.004–0.006 s). Duration of Steps (
1
), (
3
-
1
), and (
3
-
2
) takes 23.8–112.7 s in the serial control and 7.2–29.0 s in the parallel control.The concept of Self-Organized Network (initially mentioned in [10]) is tackled in [11] using the depth-d algorithm again. The authors point out that the MiDORi GMPLS supports multiple layers, multiple paths, and multiple resources. They explain again the OSPF extension (relation between physical links and Traffic Engineering (TE) links), LMP extension (power on/off control function using the LMP ChannelStatus message with Ack and IP Control Channel always up), and RSVP extension (power control request in the Admin_Status object for LSP status flag). The authors mention the 16-port GbE switch, which they developed additionally to the 8-port switch presented in [10]. Demonstration on a 5-node 7-link network is performed showing that total switch power consumption can be reduced from 283.1 W to 276.1 W. The results of the reconfiguration times from [10] are also summarized in [11]. Additionally, the authors point out that their prototype switch does not support a “make-before-break” VLAN reconfiguration, and therefore data disruption occurs over the 29 s of VLAN reconfiguration. Eventually, the authors mention a MiDORi GMPLS optical switch, which they developed. It is also controlled via telnet by the PCE implemented on a small Linux box. The optical switch allows the authors to demonstrate the multilayer GMPLS signaling between a Lambda Switch Capable layer and a Layer 2 Switch Capable layer.In [12], the authors show the energy consumption (in Wh without specifying the considered time period and details of the traffic data) on a 4-node full mesh network. The energy saving reaches up to 23.8%.The experiments with the multilayer network using GbE switches and the optical switch from [11] are continued in [13] using the extension of the GMPLS. Namely, 4 Ethernet switches out of the considered 6 are connected to the optical switch. The demonstrated power saving (9.4 W corresponding to 6 ports) is low but shows that the MiDORi network technology is potentially feasible in a high speed and power consuming interface located in a large scale network environment.Eventually, the results are summarized in [14], which directly extends [6]. It includes Beeler's algorithm, the any-order pattern algorithm, their simulative evaluation on the National Science Foundation (NSF) network loaded with uniformly generated internode traffic, the GMPLS extensions (OSPF, RSVP, and LMP), the 8-port GbE switch development, and the same results as in [10] (fully meshed 6-node network testbed).The GMPLS extensions developed within the MiDORi project have been proposed to Internet Engineering Task Force (IETF) [15].


## 2.2. Experiments on the CARISMA Testbed

Researchers from Alcatel-Lucent Bell Labs and Universitat Politècnica de Catalunya (UPC) proposed the extension of GMPLS in [16] and performed experiments on the CARISMA testbed. More specifically, the authors of [16] propose to introduce a new bit “S” to the Resource Reservation Protocol-Traffic Engineering (RSVP-TE) Path and Resv messages. The bit “S” used jointly with the already existing bit “A” differentiates the following states of an OptoElectronic (OE) device (such as colored line card, transponder, or regenerator): up, idle, down, and damaged (see also [17]).The proposed extension has been evaluated on the CARISMA testbed available at UPC premises in Barcelona. The testbed was configured according to a Pan-European network composed of 16 nodes and 23 links with 10 bidirectional 100 Gbps wavelengths per link. 20 add/drop transponders and 10 regenerators were used at each node. The testbed was loaded with uniformly distributed lightpath requests according to the Poisson model with average holding time equal to 3 hours. The load was varied between 40 and 80 Erlang.Differentiated provisioning of connection requests is considered in [18] for gold, silver, and best-effort traffic. In this case, the “A” and “S” bits of the ADMIN Status object of the RSVP-TE Path message are used in the following way: (i) “A” = 0 and “S” = 0 indicate the OE devices in the up state which must be used to allocate gold requests; (ii) “A” = 0 and “S” = 1 indicate the OE devices in idle state which are needed to allocate silver requests; (iii) “A” = 1 and “S” = 0 indicate the OE devices in down state which can be used to allocate best-effort requests. Differently to [16], availability of regenerators and wavelengths on links is disseminated over the network using the proposed extension of the GMPLS Open Shortest Path First-Traffic Engineering (OSPF-TE) protocol. A new sub-TLV (Type Length Value) (named TSP Status) is introduced in the OSPF-TE opaque Link State Advertisements (LSAs) containing the number of up, down, and idle transponders in a node (a regenerator corresponds to two transponders). This sub-TLV is inserted into a Node Information Top Level TLV (type 5, see Figure 1 of  [18]). Using the OSPF-TE opaque LSAs, the PCE can populate its Traffic Engineering Database with wavelength and regenerator availability information, which is used for computation of end-to-end routes.The same topology as in [16] is used for the experimental study on the CARISMA testbed [18]. The service class distribution is 20, 30, and 50% for gold, silver, and best-effort traffic, respectively. Different shares of resources are reserved for different classes of traffic. Prereservation of resources is implemented in the PCE to avoid contention of resources among different lightpaths under establishment. Results on the blocking ratio, number of OE devices in up/idle/down states, and power consumption per active LSP are reported.The experimental activities on a testbed reported in [16, 18] were restricted to protocol information exchanges. Due to unavailability of transponders, the idle-up and down-up state transition times were assumed and not measured. The assumed transition times equal 20 ms and 60 s, respectively.


## 2.3. Our Contribution

We extend the related work described above in the following way. First, we implement the energy-saving schemes on the off-the-shelf equipment, demonstrating that the energy-saving is possible straightaway, even without extending GMPLS, even though FUFL and DUFL can be used also in the GMPLS environment. Second, we explicitly consider triggering events for calculation of new network configuration and consequently potential network reconfiguration for energy saving. Third, we implement different energy-saving algorithms (FUFL and DUFL) than the ones implemented in the MiDORi and CARISMA testbeds. FUFL is particularly interesting for the network operators [4]. Fourth, we implemented the “make-before-break” mechanism. Eventually, we used different traffic schemes focusing on evaluation of deactivation of parallel GbE links constituting one logical link. While very little information is provided about traffic assumption in MiDORi studies, the traffic defined as the number of lightpath requests and not “bps to be transported” is used in [16, 18]. Finally, the work in [16, 18] does not consider routing of IP traffic over the logical topology.


## 3. Network Scenario

Even though the original methods FUFL and DUFL [2, 3] were proposed for IP-over-Wavelength Division Multiplexing (WDM) backbone networks, they can be applied also to other types of networks, as pointed out in [19, 20] for FUFL. Differently from [2, 3], optical GbE links are used instead of lightpaths in this work, which is determined by the testbed that we have access to.


### 3.1. Testbed

We used the testbed located at the Institute of Communications and Information Technology (Istituto Superiore delle Comunicazioni e delle Tecnologie dell’Informazione ISCOM) of the Italian Ministry of Economic Development. The testbed scheme is shown in Figure2 [21]. The core part is composed of four Juniper M10/M10i routers (J1–J4) interconnected using 1 Gbps long haul optical links connecting Rome to Pomezia (total distance of 50 km). Three Cisco 3845 edge routers (C1–C3) are deployed at the access part of the network by means of GbE optical links. Finally, the testbed is completed with Gigabit Passive Optical Network (GPON) access networks composed of an Optical Line Terminal (OLT) and up to eight Optical Network Terminals (ONTs), offering a shared bandwidth equal to 1.244 Gbps. To guarantee an end-to-end minimum bandwidth in the backbone path, we use the technique described in [21] that allows us to assign a guaranteed bandwidth between two endpoints of the network by means of different tagging techniques, that is, VLAN and Virtual Private Local Area Network Service (VPLS).Figure 2
Testbed scheme located at Fondazione Ugo Bordoni (FUB) in Rome.Figure3 shows the testbed in the configuration used in this work. A central Personal Computer (PC) is used for network management and control. It is connected directly over Fast Ethernet (FE) links to the IP router. Traffic monitoring is communicated by means of Simple Network Management Protocol (SNMP). Triggering of calculation of a new network configuration based on the monitored traffic as well as the calculation of the new network configuration itself is realized using bash scripting. Eventually, network reconfiguration is performed by logging via telnet into the IP routers and executing commands to perform rerouting and activation/deactivation of GbE interfaces.Figure 3
Photo of the testbed configured for the experiments.


### 3.2. Base Logical Topology

We focus on the core part consisting of one Juniper M10 and two M10i IP routers interconnected by 50 km of fiber cable into a bidirectional physical ring. The logical topology is composed of the IP routers interconnected by GbE optical links. All parallel GbE optical links between a node pair form a logical link. The logical topology is controlled in a centralized way [22].We configure the testbed in order to obtain a simple scenario that allows demonstration of FUFL and DUFL. The base logical topology together with all traffic demands (source-target) is presented in Figure4. The nodesA–D andH–K represent traffic generators and sinks attached to nodesE andG. The part representing a core network consists of the nodesE,F, andG interconnected by three logical links, each formed by two GbE optical links.Figure 4
Base logical topology.


### 3.3. Traffic and Routing

Traffic and its routing over the base logical topology have been chosen in an artificial manner so that FUFL and DUFL operation can be demonstrated on the available testbed. The Ethernet Testing Platform Spirent SPT-3U, Anritsu MD1230B, and a Linux PC have been used to generate and terminate traffic. Two types of traffic are used: (i) random traffic with specified minimum and maximum values and (ii) sine-like traffic with specified minimum and maximum values, as well as period length in seconds. The sine-like traffic consists of halves of sine periods and of idle periods (as indicated in Figure5(c)), which is determined by the traffic generators.FUFL and DUFL results on the testbed (mind differenty-scales in (c) and (d)).
(a)
FUFL: total power consumption (lefty-axis) and power saving (righty-axis)
(b)
DUFL: total power consumption (lefty-axis) and power saving (righty-axis)
(c)
FUFL: monitored traffic
(d)
DUFL: monitored trafficThe maximum value of the sine-like traffic that our traffic generators can produce is 100 Mbps. Corresponding value for the random traffic is 1000 Mbps. A summary of the traffic inserted into the network is provided in Table1. Traffic is generated in both directions.Table 1
Traffic demands (bidirectional) in the network.
Traffic demand
Traffic type
Min (Gbps)
Max (Gbps)
Period (s)
A–H
Random
0.97
1
—
B–I
Sine-like
0
0.1
200
C–J
Sine-like
0
0.1
200
D–K
Random
0.97
1
—The IP routing in the base network indicated in Figure4 has been chosen so that all the logical links carry traffic and that the load exceeds the capacity of a single GbE optical link. It gives us the opportunity to see what happens with the traffic on a logical link when the whole logical link or just one out of two parallel GbE links is switched off in the low demand hour.The inefficient utilization of the logical links is caused by the limitations of the traffic generators. This constitutes no obstacle for showing the operation of energy-saving approaches, but explicitly provides potential for switching off the GbE interfaces.


### 3.4. Power Consumption

Power consumption of interfaces has been measured offline using Precision Power Analyzer N4L PPA2530 and a method similar to the one from [23]. Namely, power consumption of an IP router was measured twice, that is, when a GbE interface was active and when it was inactive (further measurements of power consumption of an IP router with all interfaces physically removed showed a difference of less than 0.5 W with respect to the router with an inactive GbE interface installed). The subtraction of these two values determined the power consumption of the GbE interface (11.07 W for the M10 router and 8.9 W for the M10i router). Power consumption of Juniper's M10 and M10i routers with all interfaces shutdown equals (according to our measurements) 186.15 W and 112.5 W, respectively. The power consumption values reported above are lower than the ones reported in [23] for Cisco 7507 and in [11] for the MiDORi Ethernet switch.


## 3.1. Testbed

We used the testbed located at the Institute of Communications and Information Technology (Istituto Superiore delle Comunicazioni e delle Tecnologie dell’Informazione ISCOM) of the Italian Ministry of Economic Development. The testbed scheme is shown in Figure2 [21]. The core part is composed of four Juniper M10/M10i routers (J1–J4) interconnected using 1 Gbps long haul optical links connecting Rome to Pomezia (total distance of 50 km). Three Cisco 3845 edge routers (C1–C3) are deployed at the access part of the network by means of GbE optical links. Finally, the testbed is completed with Gigabit Passive Optical Network (GPON) access networks composed of an Optical Line Terminal (OLT) and up to eight Optical Network Terminals (ONTs), offering a shared bandwidth equal to 1.244 Gbps. To guarantee an end-to-end minimum bandwidth in the backbone path, we use the technique described in [21] that allows us to assign a guaranteed bandwidth between two endpoints of the network by means of different tagging techniques, that is, VLAN and Virtual Private Local Area Network Service (VPLS).Figure 2
Testbed scheme located at Fondazione Ugo Bordoni (FUB) in Rome.Figure3 shows the testbed in the configuration used in this work. A central Personal Computer (PC) is used for network management and control. It is connected directly over Fast Ethernet (FE) links to the IP router. Traffic monitoring is communicated by means of Simple Network Management Protocol (SNMP). Triggering of calculation of a new network configuration based on the monitored traffic as well as the calculation of the new network configuration itself is realized using bash scripting. Eventually, network reconfiguration is performed by logging via telnet into the IP routers and executing commands to perform rerouting and activation/deactivation of GbE interfaces.Figure 3
Photo of the testbed configured for the experiments.


## 3.2. Base Logical Topology

We focus on the core part consisting of one Juniper M10 and two M10i IP routers interconnected by 50 km of fiber cable into a bidirectional physical ring. The logical topology is composed of the IP routers interconnected by GbE optical links. All parallel GbE optical links between a node pair form a logical link. The logical topology is controlled in a centralized way [22].We configure the testbed in order to obtain a simple scenario that allows demonstration of FUFL and DUFL. The base logical topology together with all traffic demands (source-target) is presented in Figure4. The nodesA–D andH–K represent traffic generators and sinks attached to nodesE andG. The part representing a core network consists of the nodesE,F, andG interconnected by three logical links, each formed by two GbE optical links.Figure 4
Base logical topology.


## 3.3. Traffic and Routing

Traffic and its routing over the base logical topology have been chosen in an artificial manner so that FUFL and DUFL operation can be demonstrated on the available testbed. The Ethernet Testing Platform Spirent SPT-3U, Anritsu MD1230B, and a Linux PC have been used to generate and terminate traffic. Two types of traffic are used: (i) random traffic with specified minimum and maximum values and (ii) sine-like traffic with specified minimum and maximum values, as well as period length in seconds. The sine-like traffic consists of halves of sine periods and of idle periods (as indicated in Figure5(c)), which is determined by the traffic generators.FUFL and DUFL results on the testbed (mind differenty-scales in (c) and (d)).
(a)
FUFL: total power consumption (lefty-axis) and power saving (righty-axis)
(b)
DUFL: total power consumption (lefty-axis) and power saving (righty-axis)
(c)
FUFL: monitored traffic
(d)
DUFL: monitored trafficThe maximum value of the sine-like traffic that our traffic generators can produce is 100 Mbps. Corresponding value for the random traffic is 1000 Mbps. A summary of the traffic inserted into the network is provided in Table1. Traffic is generated in both directions.Table 1
Traffic demands (bidirectional) in the network.
Traffic demand
Traffic type
Min (Gbps)
Max (Gbps)
Period (s)
A–H
Random
0.97
1
—
B–I
Sine-like
0
0.1
200
C–J
Sine-like
0
0.1
200
D–K
Random
0.97
1
—The IP routing in the base network indicated in Figure4 has been chosen so that all the logical links carry traffic and that the load exceeds the capacity of a single GbE optical link. It gives us the opportunity to see what happens with the traffic on a logical link when the whole logical link or just one out of two parallel GbE links is switched off in the low demand hour.The inefficient utilization of the logical links is caused by the limitations of the traffic generators. This constitutes no obstacle for showing the operation of energy-saving approaches, but explicitly provides potential for switching off the GbE interfaces.


## 3.4. Power Consumption

Power consumption of interfaces has been measured offline using Precision Power Analyzer N4L PPA2530 and a method similar to the one from [23]. Namely, power consumption of an IP router was measured twice, that is, when a GbE interface was active and when it was inactive (further measurements of power consumption of an IP router with all interfaces physically removed showed a difference of less than 0.5 W with respect to the router with an inactive GbE interface installed). The subtraction of these two values determined the power consumption of the GbE interface (11.07 W for the M10 router and 8.9 W for the M10i router). Power consumption of Juniper's M10 and M10i routers with all interfaces shutdown equals (according to our measurements) 186.15 W and 112.5 W, respectively. The power consumption values reported above are lower than the ones reported in [23] for Cisco 7507 and in [11] for the MiDORi Ethernet switch.


## 4. Methods

We explain the methods used in each step needed for energy saving according to Figure1.


### 4.1. Traffic Monitoring

The most intuitive approach to traffic monitoring is to perform the monitoring constantly. This approach is impossible in the simulative approaches performed so far, due to unavailability of input traffic data sets originating from measurements and covering traffic between all node pairs in the network over sufficiently long period of time with sufficient time granularity (IP packet arrival/departure level). Experimental activity overcomes this limitation. However, there is still some level of freedom in setting the traffic monitoring in a digital system such as a telecommunication network (testbed), namely, the time periodT
M over which the constantly monitored traffic data is averaged and provided for evaluation triggering calculation of a new network configuration.


### 4.2. Validation of Triggering Events

The second level of freedom determines the frequency at which the events triggering calculation of network configuration are validated. The related time period is denoted asT
L and should be set as small as possible in the digital system in order to mimic constant validation of triggering events. Please note that T
L is different than T
M. The latter determines the history which is taken into account when validating triggering events, while the former determines the frequency at which the validation is performed.With reference to Figure1, the “traffic monitoring” block provides with periodicity T
L input data (measure of the traffic load experienced on each logical link) for the “validation of triggering events” block. Thus, starting from the time instant t
1, corresponding to the first validation, the nth validation is performed at time t
n
=
t
1
+
(
n
-
1
)
T
L. Each validation is performed according to the measure of the average traffic load experienced during a period T
M. Specifically, the measure provided at time t
n corresponds to the average traffic load during the period (
t
n
-
T
M
,
t
n
). Notice that choosing a small value for T
L allows a prompt reaction to changing traffic conditions, which is particularly important during increasing traffic trend in order not to experience congestion within the network. On the other hand, the system should be as much stable as possible and not follow all tiny variations of traffic. For this reason, traffic load measures are provided as averages over the period T
M, which should be chosen sufficiently long so as to hide very high frequency traffic variations.Calculation of new network topology is triggered by violation of thresholds (W
A and W
D for FUFL and W
L and W
H for DUFL) as explained in detail in the following subsection. This step also takes into account the stability issue by including a hysteresis cycle within the threshold mechanism.


### 4.3. Calculation of Network Configuration

We focus on two classes of approaches to calculation of network configuration [2, 3], namely, FUFL and DUFL.


#### 4.3.1. FUFL

The first class is very simple and attractive for network operators [4]. It is fully distributed and involves neither changing of IP routing nor changing of the connectivity of the logical topology. The load on each GbE link constituting the logical link is monitored. A GbE link is switched off when load on the previous parallel GbE link goes down below W
D. It is switched on again when the load on the previous parallel GbE link goes above W
A. W
D and W
A are defined as utilization of a GbE link. The explanation above assumes bin-packing of traffic in parallel links [19, 20]. We have not verified the load-balancing mechanisms used in the Juniper routers. If other packing (load-distribution) strategies are used, traffic on the GbE link to be switched off is shifted to other active parallel GbE links. According to [24, 25], link aggregation implementation in Juniper routers uses the same load-balancing algorithm as that used for per-packet load balancing; that is, the router sends successive data packets over paths without regard to individual hosts or user sessions. It uses the round-robin method to determine which path each packet takes towards its target.


#### 4.3.2. DUFL

The second class DUFL is more complex, as it allows changing of IP routing, which in turn may increase the number of idle interfaces in the network and lead even to switching off whole logical links. A logical link nonexisting in the base network cannot be established though. There are many algorithms which fall into the class of DUFL (see, e.g., DAISIES [26] and Least Flow Algorithm [27]); however their comparison is out of the scope of this paper. The power savings in the simple base network topology from Figure 4 would be basically identical. A simulative comparison on a larger network is available in [1].For the sake of this work, we assume the following implementation of DUFL. The decision about an attempt to reroute traffic with the aim of deactivation or activation of a logical link is triggered by violation of the thresholdsW
L and W
H, respectively. Both W
L and W
H are defined as utilization of a logical link. The traffic demands routed viaE-F-G are attempted to be rerouted to linkE–G if aggregated demand on the logical linksE-F andF-G goes below W
L. Analogical rerouting attempt is performed when load of the logical linkE–G goes below W
L.Idle logical links with optical interfaces are switched off. The original logical topology and routing (Figure4) are restored when W
H is violated on any logical link.


### 4.4. Network Reconfiguration

The last step concerns the application of the newly computed network configuration in network devices. To perform this step, the management system opens a telnet session on routers which need their configurations to be changed and applies the needed changes. Specifically, routing is changed and network interfaces are switched on/off according to the computed network configuration. We ensure that rerouting is performed before a logical link is released when load decreases and after a logical link is established when load increases.


## 4.1. Traffic Monitoring

The most intuitive approach to traffic monitoring is to perform the monitoring constantly. This approach is impossible in the simulative approaches performed so far, due to unavailability of input traffic data sets originating from measurements and covering traffic between all node pairs in the network over sufficiently long period of time with sufficient time granularity (IP packet arrival/departure level). Experimental activity overcomes this limitation. However, there is still some level of freedom in setting the traffic monitoring in a digital system such as a telecommunication network (testbed), namely, the time periodT
M over which the constantly monitored traffic data is averaged and provided for evaluation triggering calculation of a new network configuration.


## 4.2. Validation of Triggering Events

The second level of freedom determines the frequency at which the events triggering calculation of network configuration are validated. The related time period is denoted asT
L and should be set as small as possible in the digital system in order to mimic constant validation of triggering events. Please note that T
L is different than T
M. The latter determines the history which is taken into account when validating triggering events, while the former determines the frequency at which the validation is performed.With reference to Figure1, the “traffic monitoring” block provides with periodicity T
L input data (measure of the traffic load experienced on each logical link) for the “validation of triggering events” block. Thus, starting from the time instant t
1, corresponding to the first validation, the nth validation is performed at time t
n
=
t
1
+
(
n
-
1
)
T
L. Each validation is performed according to the measure of the average traffic load experienced during a period T
M. Specifically, the measure provided at time t
n corresponds to the average traffic load during the period (
t
n
-
T
M
,
t
n
). Notice that choosing a small value for T
L allows a prompt reaction to changing traffic conditions, which is particularly important during increasing traffic trend in order not to experience congestion within the network. On the other hand, the system should be as much stable as possible and not follow all tiny variations of traffic. For this reason, traffic load measures are provided as averages over the period T
M, which should be chosen sufficiently long so as to hide very high frequency traffic variations.Calculation of new network topology is triggered by violation of thresholds (W
A and W
D for FUFL and W
L and W
H for DUFL) as explained in detail in the following subsection. This step also takes into account the stability issue by including a hysteresis cycle within the threshold mechanism.


## 4.3. Calculation of Network Configuration

We focus on two classes of approaches to calculation of network configuration [2, 3], namely, FUFL and DUFL.


### 4.3.1. FUFL

The first class is very simple and attractive for network operators [4]. It is fully distributed and involves neither changing of IP routing nor changing of the connectivity of the logical topology. The load on each GbE link constituting the logical link is monitored. A GbE link is switched off when load on the previous parallel GbE link goes down below W
D. It is switched on again when the load on the previous parallel GbE link goes above W
A. W
D and W
A are defined as utilization of a GbE link. The explanation above assumes bin-packing of traffic in parallel links [19, 20]. We have not verified the load-balancing mechanisms used in the Juniper routers. If other packing (load-distribution) strategies are used, traffic on the GbE link to be switched off is shifted to other active parallel GbE links. According to [24, 25], link aggregation implementation in Juniper routers uses the same load-balancing algorithm as that used for per-packet load balancing; that is, the router sends successive data packets over paths without regard to individual hosts or user sessions. It uses the round-robin method to determine which path each packet takes towards its target.


### 4.3.2. DUFL

The second class DUFL is more complex, as it allows changing of IP routing, which in turn may increase the number of idle interfaces in the network and lead even to switching off whole logical links. A logical link nonexisting in the base network cannot be established though. There are many algorithms which fall into the class of DUFL (see, e.g., DAISIES [26] and Least Flow Algorithm [27]); however their comparison is out of the scope of this paper. The power savings in the simple base network topology from Figure 4 would be basically identical. A simulative comparison on a larger network is available in [1].For the sake of this work, we assume the following implementation of DUFL. The decision about an attempt to reroute traffic with the aim of deactivation or activation of a logical link is triggered by violation of the thresholdsW
L and W
H, respectively. Both W
L and W
H are defined as utilization of a logical link. The traffic demands routed viaE-F-G are attempted to be rerouted to linkE–G if aggregated demand on the logical linksE-F andF-G goes below W
L. Analogical rerouting attempt is performed when load of the logical linkE–G goes below W
L.Idle logical links with optical interfaces are switched off. The original logical topology and routing (Figure4) are restored when W
H is violated on any logical link.


## 4.3.1. FUFL

The first class is very simple and attractive for network operators [4]. It is fully distributed and involves neither changing of IP routing nor changing of the connectivity of the logical topology. The load on each GbE link constituting the logical link is monitored. A GbE link is switched off when load on the previous parallel GbE link goes down below W
D. It is switched on again when the load on the previous parallel GbE link goes above W
A. W
D and W
A are defined as utilization of a GbE link. The explanation above assumes bin-packing of traffic in parallel links [19, 20]. We have not verified the load-balancing mechanisms used in the Juniper routers. If other packing (load-distribution) strategies are used, traffic on the GbE link to be switched off is shifted to other active parallel GbE links. According to [24, 25], link aggregation implementation in Juniper routers uses the same load-balancing algorithm as that used for per-packet load balancing; that is, the router sends successive data packets over paths without regard to individual hosts or user sessions. It uses the round-robin method to determine which path each packet takes towards its target.


## 4.3.2. DUFL

The second class DUFL is more complex, as it allows changing of IP routing, which in turn may increase the number of idle interfaces in the network and lead even to switching off whole logical links. A logical link nonexisting in the base network cannot be established though. There are many algorithms which fall into the class of DUFL (see, e.g., DAISIES [26] and Least Flow Algorithm [27]); however their comparison is out of the scope of this paper. The power savings in the simple base network topology from Figure 4 would be basically identical. A simulative comparison on a larger network is available in [1].For the sake of this work, we assume the following implementation of DUFL. The decision about an attempt to reroute traffic with the aim of deactivation or activation of a logical link is triggered by violation of the thresholdsW
L and W
H, respectively. Both W
L and W
H are defined as utilization of a logical link. The traffic demands routed viaE-F-G are attempted to be rerouted to linkE–G if aggregated demand on the logical linksE-F andF-G goes below W
L. Analogical rerouting attempt is performed when load of the logical linkE–G goes below W
L.Idle logical links with optical interfaces are switched off. The original logical topology and routing (Figure4) are restored when W
H is violated on any logical link.


## 4.4. Network Reconfiguration

The last step concerns the application of the newly computed network configuration in network devices. To perform this step, the management system opens a telnet session on routers which need their configurations to be changed and applies the needed changes. Specifically, routing is changed and network interfaces are switched on/off according to the computed network configuration. We ensure that rerouting is performed before a logical link is released when load decreases and after a logical link is established when load increases.


## 5. Results

We parameterize the methods described in Section4 in the following way. The thresholds are assigned with the following values: W
D
=
0.977 and W
A
=
0.985 for FUFL and W
L
=
0.4885 and W
H
=
0.9925 for DUFL. Both T
M and T
L are set to 10 s. The chosen values are determined by the generated traffic characteristic and for the sake of demonstration of the power saving approaches. Traffic variations close to the threshold values allow us to verify the methods without waiting long (corresponding to diurnal variation of traffic).As for the setting ofT
L (the time period between two successive validations of triggering events), we had to mind the time needed for reconfiguration (Step 4 in Figure 1). Specifically, T
L should be longer than the time needed for network reconfiguration, denoted as Δ
Step
4. The reason is that two concurrent attempts to switch on/off a network interface could be undertaken otherwise. Due to the nonnegligible values of Δ
Step
4 experienced in our experiments, we decided to overcome this problem by not performing the validation of triggering events during network reconfiguration. In this way, it was possible to keep T
L lower than Δ
Step
4 without experiencing any concurrent attempts to switch on/off an interface.Figures5(a) and 5(b) report the total power consumption and the power saving for the testbed running FUFL and DUFL on the logical topology, respectively. For clarity, the first 600 seconds are reported. The total power consumption corresponds to power consumed by all active GbE interfaces together with the routers according to the data from Section 3.4. Power consumption varies more frequently with DUFL than with FUFL, since our implementation of DUFL is more aggressive in turning off the network interfaces—it attempts to switch off the whole logical links. This in turn produces in general higher power saving compared to FUFL. The difference is minor due to the simple 3-node base logical topology and IP routing schemes that we use for this demonstration (see Section 3).Figures5(c) and 5(d) report the monitored traffic on logical links when FUFL and DUFL are applied, respectively. The figures report also the sine-like traffic injected to the network. As expected, all the logical links are always utilized with FUFL, and therefore each link has always aggregated traffic around 1 Gbps. On the contrary, the utilization of the links frequently changes with DUFL, since this algorithm reroutes the traffic and powers off the entire logical links. Therefore traffic on each logical link has a strong fluctuation between 0 and 2 Gbps.The timeΔ
S
tep
3 needed for calculation of network configuration (Step 3 in Figure 1) takes 0.15 s for both FUFL and DUFL. We measured also the time needed for network reconfiguration (Step 4 in Figure 1). It consists of (i) time consumed by telnet (opening a session), Δ
telnet: 11.54 s; (ii) time needed to power on a GbE interface, Δ
activate: 0.01 s; (iii) time needed to power off a GbE interface, Δ
deactivate: 0.01 s. We neglect the time that is needed to perform the rerouting in DUFL.We point out that the time values that we obtained are comparable with the ones reported in [10, 11], even though the testbeds differ significantly.The overall duration of Step 4 for a network consisting of a set of nodesV is calculated according to
(1)
Δ
Step
4
=
∑
i
∈
V
‍
(
x
i
·
Δ
activate
+
y
i
·
Δ
deactivate
)
+
z
·
Δ
telnet
,
where x
i and y
i denote the number of activated and deactivated interfaces at node i
∈
V, respectively, and z denotes the number of nodes (routers) that need to be accessed one after another.Clearly, in an operational network, the timeΔ
Step
4 required for the reconfiguration should be limited. In our case, it takes quite a lot of time to open a telnet session in order to reconfigure a router. If telnet authentication is done manually, the time to open a telnet session decreases a lot for the routers under consideration, depending on the complexity of the password. For example, the time that we measured with a manual authentication and a simple password of 6 characters was only 2.5 seconds. This amount of time depends on the implementation of telnet and on the operating system of the device. In our case the telnet sessions are opened one after another in order to perform the configuration of the interfaces for each node. In such a case, z
=
∑
i
∈
V
‍
z
i, where z
i determines the need to access node i
∈
V; namely,
(2)
z
i
=
{
0
,
if
x
i
+
y
i
=
0
,
1
,
otherwise
.
Note that z can be at most reduced to 1 if routers can be accessed in parallel. We show the time taken by network reconfiguration Δ
Step
4 in Figure 6 for FUFL and DUFL and for both the parallel and consecutive access of routers (automatic and manual authentication, resp.). Clearly, the required time to switch on/off the devices increases, when the number of affected interfaces is increased, since more routers need to be accessed and configured in our scenario. The theoretical time taken to switch on/off interfaces when manual authentication is assumed is significantly lower than the time measured during our experiments with automatic authentication. We think that this issue of long automatic authentication should be easily overcome in the operational network. Furthermore, local implementation of FUFL on routers would avoid this problem too.Time taken by network reconfiguration (Δ
Step
4).
(a)
FUFL
(b)
DUFLIn Figure7 we can observe the trends of the total monitored traffic and total power consumed by the network over time. For clarity we limit the timescale to the first 290 s. We can clearly see the direct impact of the total monitored traffic on the total power. In particular, the power tends to increase when the traffic increases, meaning that the algorithms are able to correctly react to the traffic variation. We can observe again that with DUFL the power varies more frequently, suggesting that the number of interfaces that are switched on/off frequently varies.Total monitored traffic and total power over time (mind differenty-scales).
(a)
FUFL
(b)
DUFLTo give more insight, we have collected the events that occur in the network. In particular, every time that one of the thresholds is violated (Step 2 in Figure1), one of the actions is taken according to the calculated (with FUFL or DUFL) network configuration (Step 3 in Figure 1). The occurrence of such events is reported in Figure 8. We show the first 290 s on the time axis in correspondence with Figure 7. Interestingly, we can observe that with FUFL (Figure 8(a)) most of the events do not require a change in the current network configuration, while with DUFL (Figure 8(b)) the configuration frequently varies over time. Additionally, the number of switched off interfaces is higher with DUFL than with FUFL (as expected).Occurring action over time.
(a)
FUFL
(b)
DUFLFinally, we observed minor increase of end-to-end packet delay (up to 30 ms during reconfiguration) and no packet loss based on the monitored traffic. Thus, we can conclude that our solutions are able to save energy while not deteriorating the quality of service for users in this simple scenario.


## 6. Conclusion

We demonstrated the operation of energy saving approaches FUFL and DUFL on a testbed with optical GbE interfaces. We showed that it was possible to dynamically adapt the network configuration to the changing load without losing traffic and with a minor increase in the packet delay. Moreover, we demonstrated that it was feasible to automatically and remotely activate and deactivate interfaces of commercial devices available today. We experienced relatively long time to reconfigure the network, which depends on how routers are accessed. We believe that this issue can be easily overcome in the operational networks using future devices designed for green networking. Furthermore, FUFL results can be directly translated to bigger networks, because this mechanism is local. Experimental validation of DUFL approaches is more challenging, since no testbed of a large size is available to us. Results of simulative studies on larger networks than our scenario can be found in [1, 4, 26, 28].In general, the mesh degree of the network influences power savings more than the network size [29, 30]. This is due to the fact that the mesh degree determines the number of possibilities to reroute traffic. Regarding the different methods for calculation of network configuration (Step 3 in Figure 1), our big picture study [1] showed that there is a noticeable difference between the simple local method FUFL and the more complex methods. However, power saving should be evaluated together with other evaluation criteria such as impact on QoS, network knowledge, or protection consideration (see Table 2 in [1]).



---

*Source: 806960-2014-03-27.xml*