To: Jon Koomey

From: Bruce Nordman

Re: Electricity Requirements for LBNL\'s Networking Hardware (LBNL
PUB-835)

Date: December 9, 1999

This memo summarizes a quick inquiry into the energy use of the
equipment making up LBNL's computer network infrastructure ("LBLnet").
This estimate does not include computers attached to the network other
than a few whose only function is to provide network services. It also
does not address the net effect on space conditioning, though that is
likely modest with LBNL's facilities and climate.

**Data Sources**

The network structure and equipment inventories are derived from an
interview with Ted Sopher and Ken Berg of the LBLnet Services Group in
November, 1999, and a detailed schematic drawing of the network that
they provided. Power data are primarily typical energy use figures from
the manufacturer along with nameplate power rating data for a few
devices.

The entire system is quite complex, but most of the complexity doesn\'t
affect energy use. LBLnet is always evolving, so this estimate is
necessarily a snapshot. The lab is highly networked, including several
computer-intensive research applications (the ALS and NERSC[^1]) so
probably has a higher networking power consumption per employee than
most organizations. The diversity of lab buildings and operations also
likely increases the intensity of network hardware in use.

The primary elements of LBLnet are Switches, Routers,
Transceivers/Repeaters, Network Servers, and other equipment. Network
devices (computers, printers, etc.) in a building are all connected to a
Subnet Switch[^2]. The switch transfers packets of data among the local
devices, and exchanges packets with the rest of the world via a
connection to the lab network's core. The core connections are mostly
fiber optic cables with a transceiver at each end to translate between
electrical (copper wire) and light (fiber optic) signals. In the core,
subnets are connected to one of five routers that can route packets to
the proper subnet within the lab or to the outside world. A number of
computers within the core area provide network services such as security
or 'name serving'. I count these as part of the network infrastructure
as they serve no purpose other than to support the network. Machines
such as web servers are not necessary for the functioning of the
network. Attention must be made to not double-count the energy use of
network support computers.

**Power Data**

There are several factors affecting the relationship between the
'nameplate' power listed on the back of a device and its real
consumption. In most cases these need to be multiplied together to get
the final rated-to-actual ratio. For reliability, some network hardware
and servers have multiple power supplies so that if one fails the
other(s) can provide all necessary power without interruption. While
they may operate in parallel in normal operation, the presence of
multiple supplies does not significantly increase overall energy use. A
second set of factors is the safety margins engineers built into what
the chassis is rated to manage and what the power supply could draw, and
between what the circuitry could draw and the power supply can deliver.
A third factor is the contrast between a maximally configured machine
and the average one. Larger switches and routers come as a box into
which a number of cards can be inserted to provide the number and type
of services desired. For Cisco products, 5, 7, and 13 are common numbers
of slots. The chassis and power supplies will be sized for a fully
loaded box of the most energy-intensive cards. The average box will have
only a portion of the slots filled and the average card will be less
energy-intensive than the maximum card. A fourth factor is that power
use depends on the level of network activity. For a table listing the
power use of specific cards types for a switch, Cisco notes that the
figures provided are "worst case" ones and that "Typical numbers are
approximately 30 percent below the numbers listed here"[^3].

As an example, the estimate below for the energy consumption of a Cisco
5513 switch serving 100 ports is 300 W. The nameplate rating for this
device is 16 A or 1.9 kW, giving a rated to actual figure of just 16%.
With a 5505 chassis, the rating is 9A for a percentage of 29%. Our
average 5513 switch probably has more than 100 ports, but it is unlikely
that the average demand gets anywhere near 50% of the rated demand.

**Subnet Switches**

The lab has about 90 subnets, with 250-1000 possible IP addresses for
each. This is about 40 people per subnet, and while some subnets are
dedicated to specific experimental uses, most are for general office
space so are used at well under their theoretical capacity. LBNL has
issued close to 10,000 IP numbers over the years, but many are not in
current use, so 5,000 (or 55 per subnet) is a likely in-use count. Since
network ports come in sets of 12, 24, or 48 per card, there are
naturally some extras. In addition, some ports are connected to wires
that have no device at the other end. To account for this, I have
increased the number of ports to 9,000 (100 per subnet).

A fully loaded large switch might be a Cisco 5513 with 11 48-port
boards, capable of serving 528 ports. Most of LBNL's switches are only
partially filled with cards. Actual consumption can be estimated from a
Cisco planning document[^4] listing maximum power requirements for
various models of chassis, supervisor cards, and port cards. The chassis
power levels range from 42 to 105 W, supervisor cards from 60 to 70 W,
and other cards mostly in the 70 to 110 W range.

Assuming 75 W for the chassis plus 70 W for the supervisor card results
in 145 W; subtracting 30% for the maximum-to-typical ratio gives 100 W
or 1 W per port for the power other than the port cards themselves. For
the port cards, most of LBL's are 5012 or 5225 cards. These draw 72/90 W
maximum; the number of ports is 48/24; the power per port is 1.5/3.8 W;
and after discounting by 30% it is 1.1/2.6 W per port. Taking 2 W as
typical, plus the 1 W for the balance of the system gives 3 W per port
for total switch energy. The 100 ports per subnet results in 300 W per
switch on average.

The 9,000 port figure for the lab as a whole amounts to about 2.5 ports
per person. At 3 W per port, this is 27 kW, or 240 MWh/year.

Not all ports are served by central building switches. A typical tiny
8-port switch (a \'hub\') is rated for 4W of low-voltage power. With
power supply losses this might be 6 W, or 50 kWh/year. There are about
400 of these at the lab, for a total of 2.4 kW or 20 MWh/year. These are
in addition to the subnet switches, and to the extent that there are at
least three ports used per hub, their ports offset ports on the
switches, so they reduce the total energy use rather than add to it.

While most subnets are served by the larger switches, there are about 20
Cisco Series 2900 switches at the lab. These are reported to consume
about 80 W[^5] during use, which is 2.9 W per port. There are also about
15 Cisco Series 1900 switches which use about 50 W[^6] or 2.1 W for each
of their 24 ports (for both of these it is unclear if this is rated or
actual power use).

**Ethernet Transceivers and Repeaters**

Transceivers convert between electrical and fiber optic cables. There
are less than 40 in the core network area, 40 corresponding ones
elsewhere, and perhaps another 20 others for a total of 100. They run
off of power supplies that are rated for 1 A, but probably use much
less. There are also about 100 repeaters (fiber and local) which
probably use similar power. Assuming 0.5A per device (60W), and 200
devices, the total is 12 kW, or 100 MWh/year. This is likely an
overestimate and may be a good metering opportunity.

**Routers**

There are six of these in the core (one for NERSC), plus one at the ALS,
for a total of seven. The routers are currently about half Cisco 7513
models with the rest smaller ones, with typical power use from 1,050 W
down to 550 W depending on the size[^7], with 850 W a likely average.
0.85 kW is 7.5 MWh/year each, with all seven about 6 kW or 50 MWh/year.

**Network Servers**

There are about 10 computers for security monitoring and about 10
providing various network services. None of the security machines have a
monitor, but some of the others do. A figure of 200 W for each is
generous, which would amount to 4 kW for the entire 20 machines, or 35
MWh/year.

**Other Equipment**

There is one switch, about 50 fiber transceivers or repeaters, and other
miscellaneous equipment. in the core. All the core network equipment is
on a Uninterruptible Power Supply (UPS) that is shared with the lab
phone system. The UPS has a capacity of 80 kVA of which about half, or
40 kW, is used by the network plus the core of LBNL's phone system. The
routers plus transceivers/repeaters total to only 10 kW under my
assumptions, leaving plenty of power for the several servers, switches,
and other equipment in the core area.

There are various miscellaneous devices, some in the core, as part of
LBLnet. I have not estimated energy use for these. Some such as the FDDI
GigaSwitches at NERSC are atypical. Some, such as monitoring devices,
probably use very little energy. The microwave towers might be energy
intensive individually, but with only three pairs in LBLnet, can\'t
amount to much. To be conservative I've added another 4 kW or 35
MWh/year for all of these.

The main UPS also has some ongoing power loss. Assuming it is 5% of 40
kW, this is 2 kW, or 20 MWh/year, bringing the total for other equipment
to 6 kW.

**Summary**

Table 1 shows the total for each of these categories, reflecting an
assumption of 3,800 people and 4,500 computers. There are about 3,400
employees at LBL but graduate students, and net visitors increase this.
We lack an accurate count of computers but there is clearly an average
of more than one per person.

Table 1. LBLnet Equipment: Average power and annual energy use.

  ---------------------------- ------------------------- ---------------- ---------------- --------------
                               **Average Demand\         **Per Person\    **Per Computer\  **Annual Use\
                               ***kW*                    ***W/employee*   ***W/Computer*   ***MWh/year*

  **Switches**                 27                        7.1              6.0              240

  **Transceivers/Repeaters**   12                        3.1              2.7              105

  **Routers**                  6                         1.6              1.3              50

  **Network Servers**          4                         1.1              0.9              35

  **Other Equipment**          6                         1.6              1.3              55

  **Total**                    55                        14.5             12.2             485
  ---------------------------- ------------------------- ---------------- ---------------- --------------

The annual energy per person is 130 kWh/year, and per computer is 110
kWh/year. In the San Francisco project[^8], we estimated that their
total office equipment energy use was just over 700 kWh/year per person
(this included PCs, monitors, printers, copiers, and fax machines).
While LBNL's office equipment use per person is likely considerably
higher than San Francisco's, using these two figures results in network
infrastructure energy use of less than 20% of the office equipment
total.

If every business or institution had a network infrastructure like
LBNL's, there would still need to be additional routers between them.
However, we should expect that this would not be much more than the
internal LBLnet router use and probably less, so would not greatly
increase the total. Some institutions use smaller routers in place of
local switches, but the resulting per port energy use is probably not
much greater than for LBNL's network design.

**Next Steps**

If we are interested in metering some of these devices, LBLnet staff are
willing to help out and also interested in the results. They emphasize
that we would need to monitor devices in actual use to capture the extra
power used by network traffic rather than simply the baseload power use
with no activity. The core routers would be problematic to interrupt
given their key role in laboratory communications, though if there were
interruptions planned for some other purpose, perhaps electricity
monitoring could be added to this. Monitoring a switch and a
transceiver/repeater would be considerably easier to do, and more
important given their larger contribution to total energy use based on
this estimate.

**Schematic of LBL network (to be added).**

[^1]: Advanced Light Source and National Energy Research Supercomputer
    Center.

[^2]: Large buildings sometimes having more than one subnet, or several
    smaller ones may share a subnet.

[^3]: Ibid. Ken Berg and Cisco documentation confirm that power use
    varies with the level of network activity.

[^4]: "Site Planning" for the Catalyst 5000 Family ---
    http://www.cisco.com/univercd/cc/td/doc/product/lan/**cat5000**/hardware/installg/03pspg.htm

[^5]: "Technical Specifications" for the Catalyst 2924---
    http://www.cisco.com/warp/public/cc/cisco/mkt/switch/cat/2900xl/prodlit/**2924**\_ds.htm

[^6]: "Technical Specifications" for the Catalyst 1900 Family ---
    http://www.cisco.com/univercd/cc/td/doc/product/lan/2820**1900**/1928v9x/19icg9x/19icspec.htm

[^7]: "Cisco 7500 Series" ---
    http://www.cisco.com/warp/public/cc/cisco/mkt/core/**7500**/prodlit/c7500_pa.htm

[^8]: Nordman, Bruce, Barbara Kresch, and Roger E. Picklum, "Guide to
    Reducing Energy Use in Office Equipment", March 20, 1999, Lawrence
    Berkeley National Laboratory.