Off-Nominal IP Networking Situations

by Wesley Eddy, Verizon Federal

for NASA Space Communication Architectures Working Group

1\. Introduction

Network architecture typically includes several types of infrastructure
nodes in addition to the end hosts that source and sink the majority of
application traffic on a network. These infrastructure nodes include:

-   Configuration Elements

    -   DHCP Server

    -   DHCP Relay Agent

    -   IPv6 Access Router

-   Performance Elements

    -   Link Accelerator / Transport Performance Enhancing Proxy

    -   Application Proxy / Cache

-   Connectivity Elements

    -   Router

    -   Network Address Translator

    -   Mobility Elements

        -   Home Agent

        -   Foreign Agent

        -   NEMO Router

        -   Mobility Anchor

    -   Directory Services Elements

        -   DNS Server

        -   SIP Server / Proxy

        -   LDAP Server

-   Security Elements

    -   Firewall

    -   Public Key Infrastructure Directory Server

    -   AAA Servers

        -   Radius

        -   Diameter

    -   IPsec Gateway

    -   SSL VPN Gateway

We have classified these infrastructure elements hierarchically by their
primary functions within the network architecture. Note that each
element does not necessarily exist within its own dedicated network
node, but often multiple elements are co-located within a node. For
instance, routers often perform DHCP server functions, and NATs often
have firewall capabilities. This means that the removal of a node from
the network may simultaneously eliminate multiple functional elements.
Since there are interdependencies between some of these elements as
well, cascading failures are also possible when certain elements become
impaired. In the remainder of this report, we consider each element
listed above and describe some common failure modes for the element, the
results of those failure modes, and possible reactions or proactive
designs that can make systems robust in the face of particular failures.

2\. Configuration Elements

Some network infrastructure elements assist other nodes in the
bootstrapping process of determining a usable network layer
configuration when freshly attaching to a network, and then maintaining
that configuration over time. We call these configuration elements,
whether they are providing information to local static nodes, to
currently local mobile nodes, or to remote nodes connecting via a
tunnel.

2.1. DHCP Server

DHCP (the Dynamic Host Configuration Protocol) is used in both IPv4 and
IPv6 to assign leased addresses to hosts for some amount of time. When
this time expires, the lease can either be renewed or the address can be
recycled and given to another host. DHCP also provides additional
configuration information to client machines, like the IP addresses of
default routers, and DNS or SIP servers.

DHCP for IPv4 is an IETF Draft Standard (RFC 2131) and is widely used.
For IPv6, DHCP is an IETF Proposed Standard (RFC 3315), and is also
widely used, although due to IP address autoconfiguration in IPv6, DHCP
is not always used for address assignment, but usually still provides
local service discovery in IPv6 networks. In some networks, dynamic
configuration (and DHCP) is not used at all, but instead IP addresses
are assigned manually and default routers, DNS servers, and other
configuration information is also manually configured. This is one way
to avoid problems related to DHCP server failure, but it has more basic
network management issues of its own (for instance, this precludes
seamless node mobility).

The loss of DHCP server functions has two effects on a network. First,
for currently configured hosts, the loss may not be noticed at all. IP
packets continue to flow both to and from hosts that already have
addresses. DNS lookups and other services also continue to function
since the addresses are already known. Currently configured hosts will
only experience problems once their DHCP lease expires and their
attempts to renew it with the failed server are fruitless, resulting in
neither DHCPACKs that would allow the lease to be renewed, nor DHCPNAKs
that would signal that the address has been recycled for use by another
host. In this case, per RFC 2131, the client host must return to its
initial, unconfigured state. Thus the length of the DHCP leases given
out by a DHCP server determine the amount of time that the existing
clients will be robust to the loss of the server, after which they will
fail.

The second effect of the loss of DHCP services is that new clients that
become connected to the network are unable to obtain configuration data.
In IPv4, this can result in a total loss of connectivity since there is
no autoconfiguration capability for globally usable IPv4 addresses. In
IPv6 however, if the router advertisements allow, nodes may be able to
use autoconfiguration to create their own globally unique IPv6
addresses. In this case, they will still lack knowledge of the addresses
used to reach local DNS and other network services, but this problem can
be dealt with separately (e.g. by hard-coding a known DNS server IP
address for use as a backup).

Split-scope DHCP server configurations are a technique for using
multiple DHCP servers for robustness to failures. Each server in a
split-scope configuration offers addresses from a different portion of
the address pool. Configurable delays can be used to prevent the DHCP
servers from contending with each other in offering addresses to clients
simultaneously. Some DHCP server implementations provide a database
replication feature that in the case of a failed server, allows
redundant servers to renew leases for clients of the failed server.

2.2. DHCP Relay Agent

A DHCP relay intercepts and forwards DHCP messages between clients and
servers that are not on the same subnets. A DHCP relay agent can run on
either a router or a non-routing host. DHCP relays are commonly used in
order to aggregate the DHCP services for multiple subnets to a single
DHCP server. The failure of a DHCP relay agent has identical results as
the failure of a DHCP server, although localized to the subnets served
by that particular relay, since distinct relays are typically configured
for distinct subnets.

The failure of DHCP relay agents can be mitigated simply by providing
multiple relay agents per subnet. This is facilitated by the ability to
run DHCP relay agents on both routers and non-routing hosts. When using
multiple relay agents per subnet, there is a configurable delay
parameter that should be set uniquely on each of the redundant DHCP
relay agents to stagger their activities during the non-unicast phase of
discovery.

2.3. IPv6 Access Router

The IPv6 Neighbor Discovery protocol (RFC 2461) can be used for
autoconfiguration of IP addresses, default routers, and other node
configuration parameters. Router Advertisement messages from local
Access Routers are the primary means for publishing the needed
information like usable prefixes and MTUs. Without these Router
Advertisements, it is not possible to initiate address autoconfiguration
(RFC 2462).

Loss of an access router results in both the loss of connectivity
through that router to the rest of the network, and loss of the Router
Advertisements and configuration data provided by the Access Router.
Without this information, new nodes coming onto the network may not be
able to obtain usable configurations, unless they keep a fallback
configuration statically cached, or unless link-local addresses are
sufficient for all applications. Nodes will fall back to DHCP for
configuration in the absence of Router Advertisements, but DHCP may not
always be available if Router Advertisements are being used. Restoration
of the Access Router should immediately heal hosts that are unable to
configure themselves without the Router Advertisements. If multiple
Access Routers are present on a link, then the Router Advertisements
from any one surviving Access Router are enough to allow new nodes to
configure themselves.

Nodes that are already configured before the loss of an Access Router
are also affected by the loss of an Access Router. Prefixes and default
routers obtained via Router Advertisements from Access Routers have
lifetimes associated with their use inside hosts. If no Router
Advertisements renew these, they will expire out of hosts'
configurations and new prefixes and default routers will have to be
selected. Changing the prefixes in use will affect in-progress transport
sessions and lookup mechanisms for reaching the host (e.g. DNS, SIP,
etc) that are configured with the old address. The IETF SHIM6 protocol
is a potential solution to this service disruption, but it is not yet a
finalized standard at the time of this writing.

3\. Performance Elements

Some infrastructure elements exist only to improve application
performance. Some operate at the transport layer, and some operate at
the application layer, but their effects and the impacts of their
failures are similar. RFC 3135 describes both transport and application
performance enhancing proxies, and discusses their architectural impact.

3.1. Link Accelerator / Transport Performance Enhancing Proxy

Several trade names are used to describe devices that perform in-network
transport layer performance enhancement. These usually are designed to
overcome some of TCP's shortcomings with regards to high delays and/or
high packet loss rates. They often split a TCP connection to confine
TCP's operation to either side of a challenging link, but they may
either transcode or encapsulate TCP headers into some other transport
protocol that is optimized for the particular environment. This may be
SCPS-TP, FAST TCP, or a UDP-based custom protocol, but the architecture
and effect is similar. Header and/or content compression may also be
used.

When a transport layer performance element fails, the result may either
be a loss of connectivity or simply a decrease in performance over the
link that the element was servicing. For instance, if the element is
part of the network's routing, when it fails, then the network can no
longer route through the link that it was assisting. In contrast, if the
element is invisible to the routing system and instead sniffs and
rewrites packets, then when it fails, as long as the modem can be
accessed through an alternate connection, packets can still be routed
across the link, but performance may be degraded. Since these elements
are often deployed in pairs (one on each side of a link), if the failure
of an element on one side results in a pass-through of unoptimized
traffic, then it is important that the opposite side also pass-through
that traffic rather than filtering it.

There are steps that can be taken to make the network configuration
robust to the failure of a link accelerator. If the accelerator is a
device that is routed through, then providing an alternate router that
can be failed-over to using the techniques in Section 4.1 can prevent
the loss of use of the link. If the stand-by also has link acceleration
capabilities, then a failure may not even result in a loss of
performance. Alternatively, if the accelerator is a transparent device
at the network layer, as long as it passes-through packets in the event
of a failure, communications with somewhat degraded performance can be
maintained, assuming the other side is also configured to pass-through
unaccelerated traffic.

The IETF's Informational guidance in RFC 3135 says that a user should
always have the option to choose end-to-end operation. The document also
describes how some performance elements may introduce data reliability
concerns, and make network debugging more difficult or confusing. This
implies that using a performance element (either transport or
application) should be an opt-in decision on a user's part on a
per-transport-connection basis, and should not be a default. If this is
the case, then users can manage their own risk/reward tradeoff for
sending a given data flow through the element.

3.2. Application Proxy / Cache

Performance elements that operate at the application layer can often
have more dramatic impacts on performance and also be easier to route
around (inside application code) in the case of a failure. These
elements support specific application protocols and assist by providing
some local capabilities like storage or reliability offloading. It may
be much easier to do application-level reliability in cases where link
disruptions are frequent than transport-based reliability is capable of
efficiently dealing with. Under these considerations, Delay (or
Disruption) Tolerant Networking (DTN) bundle agents are application
performance elements, although technically these are significantly
different than HTTP proxies or some other types of application
performance elements.

HTTP performance elements usually provide a local cache that can be
consulted in order to reduce the latency and capacity required to move
data end-to-end. They rely on either effective prefetch techniques or on
reuse of HTTP objects by application instances. A cache can also buffer
applications from the fact that the network has become globally
disconnected for a time, if it already contains most of the data that
users want. Some HTTP performance elements reduce the sizes of objects
by transcoding images or adjusting the source code of pages, which can
help if placed before a limiting link. A failed HTTP cache simply means
that local copies of objects become unavailable, but the global copies
are still reachable, albeit with higher delay.

DTN bundle agents improve application performance through local
store-and-forward operations. Since DTN contains some routing
capabilities, it may be possible to route around failed bundle agents,
although this does not seem to be in practice yet. Configuring
applications to fail-over sending operations with alternate bundle
agents is possible and not difficult if multiple local bundle agents are
available. For receiving operations though, DTN EIDs in existing schemes
seem to be registered with specific bundle agents, and failure of those
agents would result in inability to receive data. It would be possible
to have a receiver register a multicast EID with several bundle agents
and then have all sends to that single receiver use multicast bundles,
although this also currently seems to be beyond the capability of
existing DTN bundle agent software.

In general application performance elements may be easier to mitigate
failures for than transport agents, since user applications tend to
voluntarily communicate with them, whereas transport elements are
usually not visible to end users. This increased visibility actually
assists in detecting and reacting to failures, and is desirable.

4\. Connectivity Elements

There are several concepts that we lump together under the context of
connectivity elements. In addition to devices like physically connect
subnetworks to each other, we include devices that maintain the
reachability of mobile components or that provide resolution services
between different namespaces. The common thread between all of these
types of elements is that the service they provide allows data flows to
be set up between end node applications that are programmed to use
specific static destination addresses or names, even though the network
may be dynamic over time.

4.1. Router

Assuming a router is "active" -- that other nodes have entries with it
as a next-hop in their routing tables \-- failure of a router results in
some level of packet loss until routing tables can be updated with
alternate entries. This may occur either through timeout of the cached
entry due to the IPv6 Neighbor Discovery protocol, or due to the lack of
HELLO messages or other advertisements in a routing protocol (e.g. RIP,
OSPF, or IS-IS). There are several ways to provide redundancy and
failover for routers.

Virtual Router Redundancy Protocol (VRRP) is an IETF Draft Standard that
allows hosts to be configured with a static default route, and uses
signaling and election between multiple routers to provide automatic
fail-over for the packet forwarding service pointed to by a default
route. RFC 3768 specifies VRRP for IPv4 addresses, but although the IETF
VRRP Working Group published an Internet-Draft discussing VRRP for IPv6,
but it was not further published as an RFC. VRRP for IPv4 is available
in several off-the-shelf systems, and the protocol itself is relatively
simple to implement. Cisco's Hot Standby Router Protocol (HSRP) is very
similar to VRRP, and is described in RFC 2281, but VRRP is more
carefully designed (based on the content of the RFCs) and more widely
supported.

Dynamic routing protocols do not provide redundancy of a single router
address, but rather select amongst redundant paths through routers with
different addresses to provide a similar type of availability for local
packet forwarding functions. Three commonly used dynamic routing
protocols are RIP, OSPF, and IS-IS, each of which has various advantages
and disadvantages. A full trade-off analysis is beyond the scope of this
document, it suffices that multiple protocol solutions exist to use a
mesh of routers to protect against the failure of any single router
within that mesh. Compared to VRRP, dynamic routing protocols can be
used when the router addresses themselves do not hold significance. This
depends on the network architecture. In some cases, a VRRP-like solution
can be obtained by using anycast routes to the protected address or
prefix. The optimal solution depends heavily on the local network
architecture.

4.2. Network Address Translator

Network Address Translators (NATs) are used to translate between IP
packet addresses from private prefix spaces into publicly routable
addresses for traffic in one direction and back in the other direction.
Since a NAT is functionally similar to a router that also modifies
packets, the failure of a NAT is similar in effect to the failure of a
router, and can be mitigated in a similar fashion (e.g. using VRRP to
reach redundant NATs). One complication is that since IP addresses are
embedded in the payloads of many higher-layer protocols, NATs often keep
state and perform application-layer gateway (ALG) operations to decode
traffic and translate the IP addresses embedded in packet payloads.
Redundant NATs that serve applications requiring ALG functionality need
some NAT-to-NAT protocol or mechanism for sharing state. There is no
standard for this, but non-standard non-interoperable vendor solutions
are available.

4.3. Mobility Elements

Mobile IP and its extensions define several pieces of network
infrastructure whose failures are discussed in this section.

4.3.1. Home Agent

The Home Agent is the basis of Mobile IP, both version 4 (MIPv4) and
version 6 (MIPv6). The Home Agent is an element deployed in an area with
stable network connectivity called the Home Network. The Home Agent
serves a number of Mobile Nodes by holding addresses for them that
correspond to the well-connected part of the network. When the Mobile
Nodes are attached to the Home Network, but are temporarily attached
elsewhere in the network topology, with other addresses, the Home Agent
relays packets sent to their permanent Home Addresses in the Home
Network to their temporary Care-of Addresses at their current location.

When using MIPv4 (as specified in RFC 3344), all packets sent from a
Corresponding Node anywhere in the network to the Home Address of a
Mobile Node go through that Mobile Node's Home Agent. Failure of the
Home Agent thus results in a complete loss of connectivity to the Mobile
Node via its Home Address for both new and existing connections. The
Mobile Node can still send packets to Corresponding Nodes, so it is
possible that unidirectional transport protocols like some content
dissemination protocols using erasure coding could be uninterrupted.
However, common applications that use TCP or other means of active
chatty feedback and acknowledgement will fail if the Home Agent ceases
to function. For practical reasons involving in-network filtering of
packets based on their source addresses (to combat spoofing), many MIPv4
implementations use bi-directional tunneling through the Home Agent, in
which case failure of the Home Agent results in loss of connectivity in
both directions. The Mobile Node could only initiate connections using
its Care-of Address in this event.

In MIPv6 (RFC 3775), the situation is somewhat different. The Home Agent
is used for initial packets from a Corresponding Node to a Mobile Node,
but if both Nodes support the Route Optimization extensions, then later
packets flow directly between the two parties. Existing connections
using this mode could survive the loss of a Home Agent for some period
of time. If the Care-of Address changes while the Home Agent is offline,
the Return Reachability Check depends on the Home Agent and would fail,
breaking the connection. Bi-directional tunneling is used in the NEMO
extensions to MIPv6, and in this case loss of a Home Agent has the same
effect as it does when using bi-directional tunneling in MIPv4. Route
Optimization extensions that are functional with NEMO are being
developed, so bi-directional tunneling may not always be the only mode
NEMO supports.

People considering deploying Mobile IP protocols for critical
communications have viewed the Home Agent as a single point-of-failure.
This has stalled or prevented Mobile IP's recommendation for deployment
in some of these cases. The MIPv6 specification allows for multiple Home
Agents to be present in the Home Network, but does not specify how to
transfer redirection service for a Mobile Node between Home Agents. The
capability is intended more for load balancing rather than fault
tolerance. Some proposals for Home Agent redundancy capabilities have
been presented, but none currently standardized. The most well-developed
of these is the Inter Home Agents (HAHA) protocol, which defines
messages for sharing registration state for Mobile Nodes between Home
Agents and uses periodic "hello" signaling between equivalent Home
Agents to detect liveness. The HAHA protocol could potentially allow
Mobile IP deployment without any fear of Home Agent failure, assuming it
advances in the standards development process. Too little implementation
experience and test results are available today to be certain.

4.3.2. Foreign Agent

Mobile Nodes in MIPv4 can use either a co-located Care-of Address, which
means that a Mobile Node has a distinct Care-of Address, or a Foreign
Agent Care-of Address, which allows many Mobile Nodes to share the same
Care-of Address. The advantage of the Foreign Agent Care-of Address is
that if IP addresses are scarce, then it provides some savings. The
disadvantage is that it requires additional infrastructure. The Foreign
Agent is an infrastructure element that advertises itself to Mobile
Nodes and notifies them of usable Foreign Agent Care-of Addresses. It
then acts as a tunnel endpoint that decapsulates packets that are
redirected to it from Home Agents.

The failure of a Foreign Agent affects all of the Mobile Nodes that are
currently relying on it to decapsulate their packets. Nodes will still
be able to send packets out, but not receive them coming in. It is not
uncommon for the Foreign Agent to also be a default router for the
Mobile Nodes, so its failure might imply other problems for outgoing
traffic as well, although the Foreign Agent does not have to be the
Mobile Node's default router, this is just a popular way to consolidate
infrastructure. There are no protocols or proposals for failover between
multiple Foreign Agents that we are aware of. Using co-located Care-of
Addresses exclusively may be a more robust solution when such addresses
are available.

4.3.3. NEMO Router

A NEMO Mobile Router hides mobility from nodes using one or more Mobile
Network Prefixes behind it. This is accomplished using a tunnel between
a Mobile IP Home Agent in the right topological location for the prefix
and the NEMO Mobile Router's Care-of Address representing its current
location. On the wire, NEMO operations are a small extension to Mobile
IP, and redundancy mechanisms like HAHA can be used to effectively
protect the Home Agent side of a NEMO tunnel. Protecting the NEMO Mobile
Router side of the tunnel requires different solutions.

The effects of a failed NEMO Mobile Router on the nodes behind it is
identical to the failure of a router, since these nodes usually interact
with the NEMO Mobile Router as only a router, and have no knowledge of
its NEMO functions. The nodes also do not have topologically-correct
Care-of Addresses that they could use for communication to bypass the
failed NEMO Mobile Router, since the NEMO Mobile Router masks
topological changes from them.

NEMO is designed with the potential for multiple Mobile Routers to serve
a Mobile Network Prefix and provide load balancing or failover for that
prefix. Failover scenarios and technical solutions are described in
several drafts that have been discussed in the IETF NEMO working group.
No specification has been formally adopted, but active work on the
problem continues and experimental code can be found.

4.3.4. Mobility Anchor

In protocols like Hierarchical Mobile IPv6 (RFC 4140) additional
infrastructure elements called Mobility Anchor Points (MAPs) are used to
provide "local Home Agent" functionalities that provide a reduction in
the MIPv6 signaling latency and overhead. This is accomplished by using
a tree of routes over time from a MIPv6 Home Agent to a Mobile Node
through various MAPs. Mobility of the Mobile Node within the tree of
MAPs is only signaled all the way to the Home Agent when the top-level
MAP in use changes.

Failure of a MAP is thus similar to failure of a Home Agent. Packets
destined for the Mobile Node through the MAP are not able to be
redirected to it because the MAP is unavailable. Packets destined
directly to the Mobile Node through its current Care-of Address are
still capable of reaching it, and outgoing packets that do not go
through a tunnel through the MAP also still allow communication.

Redundancy and failover for MAPs could be implemented in a similar way
as it is for Home Agents in the HAHA protocol. At this time, there is no
specification for such any robustness mechanism for MAP services,
however, and we are not aware of any planned work in the IETF to produce
such a specification (HMIPv6 itself is Experimental). If the HAHA
protocol proceeds to be developed further in the IETF, it is possible
that only minor additions to it would allow MAPs to use HAHA for this
purpose.

4.4. Directory Services Elements

For human-usability, most applications employ some mapping service from
human-readable names (like domain names) into machine-usable names (like
IPv6 addresses). Since these services are then required to initiate
communications, their unavailability needs to be considered.

4.4.1. DNS Server

Most IP-based applications initiate connections by first performing a
Domain Name System (DNS) lookup of a hostname. The hostname is resolved
recursively, and several DNS servers may actually be involved. A query
to a local DNS server may be handled in one of three ways: (1) the
request may be for a locally-administered name, in which case a response
can be immediately returned, (2) the request may be for a
remotely-administered name with a fully reply requested, in which case
the DNS server itself contacts other remote DNS servers on behalf of the
client, in order to resolve the name, or (3) the request may be for a
remotely-administered name with iterative resolution specified by the
client, in which case the server simply returns the address of the next
DNS server that the client should contact for more information.

Failure of a local DNS server has several consequences. Local client
hosts who are configured to use that DNS server will have their requests
time-out, which is a slow way to detect failure. This will stifle both
local and remote communications that are based on host names rather than
IP addresses. Applications that use IP addresses directly will function
without any difference. Remote clients who are attempting to resolve a
name within the subdomain managed by the local DNS server will also
experience timeouts and failures.

There are several ways that DNS server failures can be mitigated. The
DNS protocol is designed to accommodate multiple servers within a
domain. Hosts are usually configured with a list of DNS servers rather
than a single server, and the default host behavior is to try each known
DNS server until a response is obtained. This does involve timeouts for
each request if the first server on the list has failed, but it
maintains services. Rules for selecting from, sorting or resorting the
list of DNS servers that a host knows about are a matter of
implementation, and might be configured to shotgun requests to multiple
servers in order to limit the latency of a lookup when a server has
failed. In general, this may not be advisable, but for isolated
deployments, it might be an option.

IP anycast addresses are another popular option for making DNS services
robust to server failures. Using anycast, all the DNS clients can be
configured with a small number of addresses, but a number of redundant
DNS servers can answer to those addresses, with the routing system
determining which DNS server to use at a particular time and place in
the network. Anycast local DNS is desirable because it involves less
latency and intelligence in clients in responding to failed DNS servers.

Another means for robustness to DNS failures is to provide a local cache
of mappings, for instance in the common /etc/hosts file on Unix-like
systems. The drawback to this approach is that it is more static than
using a DNS server, and so if DNS mappings are dynamic or are added
frequently, the local hosts tables for all users will have to be updated
periodically. Since local host tables are commonly consulted before DNS
servers, it may be prudent to limit entries to only the most critical
and stable hostname mappings. Otherwise an erroneous local host entry
could be used instead of a correct DNS response (that is never requested
because of the presence of the host table entry).

4.4.2. SIP Server / Proxy

There are three types of infrastructure nodes used by the Session
Initiation Protocol (SIP) used for session initiation by several
applications (namely Voice over IP). In practice, these three types of
nodes are often implemented within a single element, but they may also
be separate, depending on deployment goals. The three types of SIP
infrastructure elements are:

-   SIP Proxy Servers interact directly with local clients for location
    registration and handling location and negotiation tasks for
    incoming and outgoing session requests.

-   SIP Registrar and Location Servers serve as databases of location
    information for clients/users of local SIP Proxy Servers. These only
    communicate directly with SIP Proxy Servers and do not use the SIP
    protocol for this communication. Often, this component is handled by
    a traditional database server.

-   SIP Redirect Servers are queried by SIP proxy servers (using the SIP
    protocol) in order to find appropriate SIP Proxy Servers in other
    domains that local clients have requested sessions within.

When SIP Proxy servers fail, then local clients are unable to initiate
any new sessions through them with either local or remote SIP clients.
Since SIP servers do not participate in the actual session, existing
sessions are mostly unaffected, similar to the case of DNS server
failure. If the failed SIP Proxy is referred to by remote SIP Redirect
Servers, then new sessions from remote clients to local clients will
also be unable to be initiated. When SIP Registration Servers fail, the
impact is passed through the SIP Proxy Servers to the SIP clients, with
similar effects as in the case of a failed SIP Proxy Server. Failed SIP
Redirect Servers do not affect the establishment of local-to-local
sessions or incoming remote-to-local sessions, but do prevent new
local-to-remote sessions from being setup.

Some SIP clients can be configured with multiple local SIP Proxy
addresses. The clients can then register with multiple SIP Proxies and
failover between them when they sense that local requests to one are
failing. Dynamic DNS SRV records for the domain's preferred SIP server
can be used to ensure that SIP Redirect Servers for incoming session
requests from other domains reach a live SIP Proxy.

Since not all SIP clients support registering with multiple servers,
client-based failover may be slow, and using DNS for redirection has its
own problems, it is advisable to use a separate SIP Register and
Location Server that the local SIP Proxies mirror their registrations
to. A mechanism like VRRP or IP anycast can be used to ensure that a
live SIP Proxy is reachable at a static IP address within the domain.
Robustness of the SIP Registration and Location Server is provided by
traditional database server reliability techniques (e.g. server
clustering).

4.4.3. LDAP Server

Lightweight Directory Access Protocol (LDAP) servers are used by some
common applications as well as by some other infrastructure services for
storage and lookup of directory entries. For instance, email clients
typically support the use of LDAP to find the email addresses of
individuals. Some DHCP, DIAMETER, and RADIUS servers support the use of
LDAP for their record-keeping purposes. Offloading the data to LDAP like
this can make these other infrastructure services more robust to element
failures, but the LDAP service itself must also be robust.

LDAP client configurations support listing of multiple LDAP servers,
similar to the way that DNS clients are configured with multiple DNS
servers. The client can then initiate failover use of secondary LDAP
servers if the primary LDAP server becomes unresponsive. Alternatively,
anycast addresses could be used for LDAP servers. In this case, the
redundant LDAP servers sharing an address have to also share security
credentials if transport security for LDAP is used. Since LDAP supports
both read and write operations, ensuring synchronization of the
databases behind LDAP primary and secondary servers is necessary. Since
LDAP is primarily defined as a front-end access mechanism to X.500, this
issue may be handled by the backend in some implementations and uses of
LDAP.

5.0. Security Elements

Security elements mainly serve to deny network access to unwanted
parties. The several types of security elements discussed here have
differing failure modes due to the type and amount of local state that
each one manages.

5.1. Firewall

A firewall blocks undesirable traffic from moving into a network. There
are typically multiple levels of firewalls within a network. There are
two possible failure modes for a firewall. Since a network is usually
configured such that all traffic goes through a firewall, then failure
of that firewall can result in loss of connectivity through that path
(fail-closed). The routing system may repair this automatically and
route through another backup firewall. Another possibility is that the
firewall's loss could open the network up to undesirable packets
(fail-open). This can be troublesome in some environments because the
network behind the firewall may be constrained in resources or otherwise
sensitive to the imposed traffic. For instance, the presence of the
firewall may have caused administrators to be lax in securing hosts on
the network. In general, letting the routing system failover between
firewalls that fail-closed seems to be a better solution than
configuring the network to fail-open when a firewall is lost.

Some specialized firewall devices have redundancy features that allow
identical redundant firewalls to share state with each other and
fail-over. These features are not based on standards, however, and only
apply to stateful firewalls. Failover for stateless firewalls can be
handled fully by the routing system without disrupting connections,
whereas stateful firewalls can't be failed-over without forfeiting
existing connections, in the absence of these specialized non-standard
features. The decision between stateless and stateful firewall
techniques should depend on local security considerations; we simply
note that smooth failover is simpler and more interoperable when using
stateless firewalls.

The fail-open / fail-closed terminology here mimics that used in
configuring firewalls for how to handle the case where no matching rule
is found. Fail-open firewall configuration rules allow a packet to enter
if it does not match any rules that would otherwise cause it to be
denied, whereas fail-closed rules deny packets that are not explicitly
allowed by other rules. The configuration of firewall rules is outside
the scope of this document and depends on the local security
considerations, but it should be noted that failures due to
misconfiguration of firewall rules can have different effects depending
upon whether fail-open or fail-closed configuration rules are employed.

5.2. Public Key Infrastructure Directory Server

When using certificate-based security protocols, certificates themselves
are often transferred inline within the protocols (e.g. SSL and IKEv2).
Aside from this, for validating certificates, it is sometimes necessary
to contact a Public Key Infrastructure (PKI) Directory Server in order
to obtain certificate revocation lists and additional certificates. The
PKI Directory Server then becomes a piece of critical infrastructure,
since without this service, some certificate-based authentication
attempts will not be completable.

The PKI Directory Server for a domain is located in a number of ways,
including through a field in a certificate, and through DNS SRV records
(RFC 4386). In the event of a Directory Server failure, either using
dynamic DNS to point the SRV record to a live backup PKI Directory
Server or using a VRRP-like or clustering mechanism to maintain the
service at a static IP address are both workable solutions. Client-based
time-out and failover between alternate entries do not seem to be
feasible due to the lack of secondary server parameters.

Of course, minimizing the amount of certificate retrieval that is
required from the PKI Directory Server is desirable, and the PKI
architecture and applications should be engineered with this in mind.
For instance, pre-configuring clients with the public keys for several
certificate authorities is advantageous, in the cases of both
hierarchical and mesh certificate authority architectures. Publication
of certificate revocation lists via "push" might also be used to
eliminate some of the need for clients to request these lists from PKI
directory servers, at the expense of increased bandwidth usage.

5.3. AAA Server

Authentication, Authorization, and Accounting (AAA) servers have several
uses in controlling and recording access to network resources. For
example, in mobile networks, when devices join a network, AAA elements
may verify their authentication and authorization information for
activities on the network, and begin accounting for their usage.
Similarly, SIP servers may interact with AAA elements in order to
account and bill for calls initiated with SIP.

Two IETF protocols used for AAA functions are RADIUS and DIAMETER. There
are some significant differences between these, and DIAMETER is favored
due to being more carefully designed with modern networking needs in
mind. Technically, one large difference between the two is that RADIUS
uses UDP, while DIAMETER uses TCP or SCTP for signaling.

Since AAA systems are critical to network access and billing, AAA
elements are usually redundantly deployed in practice, for both failover
and load balancing purposes. The AAA transport profile in RFC 3539
discusses failover for AAA elements. It points out that when using TCP,
failover times can be very slow, whereas when using SCTP, the failovers
can be much faster due to SCTP's multihoming feature and heartbeat
mechanism. This document also points out that RADIUS does not include a
failover mechanism in its specifications, and so implementations vary
widely in their failover behavior. The application layer watchdog
discussed at length in RFC 3539 is what DIAMETER implementations use and
so DIAMETER failover behavior is more consistent in practice. The common
practice of providing redundancy of AAA elements is a good start at
building a robust AAA system, and using DIAMETER with SCTP is the best
current practice, as it combines the fastest failover mechanism with the
lowest risk of premature failover.

5.4. IPsec Security Gateway

In IPsec VPN configurations, one or more Security Gateway elements are
used as endpoints of IPsec tunnels between the private network and
remote users. A remote user configures his client with the IP addresses
of the Security Gateways, and the IKEv2 protocol is used to establish
Security Associations between the client and each Security Gateway. A
Security Association implies shared state, including (in normal
practice) a key for symmetric cryptography, metadata on the key,
parameters specifying the cryptographic algorithms in use, and
parameters describing the traffic that the Security Association applies
to. Establishing a Security Association for application traffic from
scratch involves several end-to-end IKEv2 exchanges. The exact number
depends on the configuration options and algorithms in use (e.g.
certificates, EAP, etc.), but usually involves at least 4 round-trip
times.

Failure of an IPsec Security Gateway element results in loss of service
into the private network for all of the VPN clients that were using
Security Associations with the Security Gateway. For as long as the
Security Association stays valid (or until the keying material expires),
there is no need for IKEv2 exchanges with the Security Gateway. Since
the Security Gateway's normal behavior is to silently perform IPsec
operations and forwarding of VPN client packets, at the IPsec level, the
clients do not necessarily automatically detect failed Security
Gateways. Since the use of IPsec to reach the VPN is mostly transparent
to the applications, to them it looks like failures of their peers.
Application-level efforts to restore sessions will fail as long as the
packets are routed through the tunnel to the failed VPN gateway.

For the earlier version of IKE, RFC 3706 described a method that can be
used to periodically perform a liveness check on an IKE peer. In the
IKEv2 specification (RFC 4306), options used to perform liveness checks
are specified in the base standard. These can be used to frequently poll
Security Gateways, but frequent polling of the IKEv2 daemon on the
Security Gateway increases the workload of the device that is frequently
already high. Since detection can be slow due to a low polling rate, and
initialization of Security Associations with backup Security Gateways is
also slow (due to multiple IKEv2 exchanges), client-based failover is
time consuming, possibly taking several minutes.

It may be possible to use IP anycast to reach the IPsec Security
Gateways into a private network, and use some non-standardized mechanism
for exchanging Security Association state between the redundant Security
Gateways in order to provide for fast failover that is transparent to
the VPN client IKEv2 peers and applications. A mechanism to operate
between redundant gateway elements and synchronize Security Associations
is being preliminarily worked on in the IETF, but not yet fully
specified. Non-standard and non-interoperable mechanisms are currently
available from some vendors.

5.5. SSL VPN Gateway

SSL VPNs currently vary widely in features and functionality. Since the
security state associated with the SSL connection and the transport
state for the underlying TCP connection are both lost if an SSL VPN
gateway fails, there is no easy way to perform a failover that is
transparent to the client. Some clustering solutions that allow for the
needed state to be shared between servers will work, but implementing
this involves more server changes than just network configuration.
Client-based detection of a failed SSL VPN gateway can be done through
the existing mechanisms for HTTP and TCP timeout and re-connection
through a backup VPN gateway can be initiated. This failover might even
be user based, since often SSL VPN software is simply a web browser,
where "configuring" an SSL VPN Gateway for use may simply consist of
setting a bookmark to it, unlike IPsec VPNs that often have slightly
more detailed configurations. Differences in IPsec VPN and SSL VPN
Gateway failover techniques are due to the layering difference between
IPsec and SSL, since IPsec requires only mirroring of IP-layer and IPsec
state between backups, but SSL requires IP-layer, transport-layer, SSL,
and potentially application-gateway-layer state between backups. Thus
failover for SSL VPNs is more difficult to automate from the
server-side.

6\. Unidirectional Scenarios

In cases where connectivity is either unidirectional, or very nearly so
due to limited link capacity or other factors, many Internet protocols
that normally detect and respond to failures cease to function well. In
fact, in cases of asymmetric links, protocol behaviors intended to aid
in robust operations (like keepalive and polling) often exacerbate the
asymmetry by wasting precious capacity in the tight direction. In other
cases where connectivity in one direction fails for some period of time,
but communications are still possible in the other direction, most stock
protocols will actually stop communicating over the link that is still
available due to the lack of feedback. For instance, any protocol using
TCP or SCTP for transport will stall waiting for acknowledgements.

If connectivity between two routers becomes unidirectional, then dynamic
routing protocols may cause packets to stop being forwarded even in the
working direction. Since most dynamic routing protocols use HELLO
messages for detecting liveness of next-hops, after short timer
expiration, the routes associated with the still-working direction fail
because no HELLO was received in the reverse (failed) direction. If
static routing is used, then packets can still flow in the open
direction, but of course the downside is that a full link failure can't
be responded to using static routing. Perhaps a good combination in some
situations is to use dynamic routing by default, but with administrative
intervention (either from a human or control program), install a static
route in the working direction when bidirectional connectivity is lost,
but unidirectional connectivity is otherwise assumed to work.

In cases where a node has an IP address at which it can be reached, but
not reach other nodes from, then mobility protocols like Mobile IP and
NEMO fail because the return reachability proceedure from the Home Agent
to the mobile element fail. Session negotiations using SIP also fail,
even if the intention is only to setup a unidirectional flow of voice
traffic, for example, because the signaling protocol is bidirectional.
Operations through IKE to handle key expiration also fail, since
requests to use new symmetric keys can't be responded to, and the use of
asymetric cryptography to establish other symmetric keys is also
impossible without bidirectional connectivity. This causes the Security
Associations to expire and blocks transmission of packets that would
normally go through them. IKE dead peer detection also uses a poll and
response mechanism to timeout the IKE Security Association itself, which
fails in the case of unidirectional connectivity.

In all of the cases listed in this section, the loss of connectivity in
only one direction results in a loss of connectivity in both directions
due only to protocol mechanisms that are not capable of inferring
unidirectional connectivity. Often the implementations could be made to
work in unidirectional situations by modifying the code to ignore some
timer expirations or other lack of responses, but this would preclude
using fully commercial off-the-shelf solutions.

7\. Conclusions

Each piece of network infrastructure adds some capability to the network
that assists users in some way. To guard against failure of a piece of
infrastructure, the two basic approaches that have been suggested in
this study are: (1) providing backup configurations to users that take
over in the case of a detected infrastructure failure, and (2) using
redundancy of infrastructure elements to allow for single elements to
fail.

The first approach is not always possible to effectively take. For
instance, if an application is written to always use DNS to look up a
name before failing over to a local name resolver, the application
performance may suffer due to the latency of its attempts to contact a
failed DNS server.

The second approach is also not always effectively possible, due to the
facts that each piece of infrastructure that is added entails some
additional cost or overhead, that multiple elements can still fail
simultaneously, and that each element adds complexity and administrative
configuration management issues to the network.

For these reasons, before deploying any piece of network infrastructure,
or any applications that rely on a piece of network infrastructure, a
careful analysis should be made of the effects of the failure of that
infrastructure element. The effectiveness of backup configurations and
redundancy should then be considered. If it is not possible to mitigate
the failure of an element effectively enough, then the network and
application design should be altered to not rely on its services,
otherwise Murphy's Law applies.