Regular Papers

High Availability Server Platform forOperations Support System
Nobuhiro Kimura†, Akira Yamada, Hikaru Seshake,
and Toshihiro Nishizono
Abstract
In the near future, operations support systems (OSSs) will be required to have higher availability. To
cope with this, we propose three methods: (1) distributed server restart, (2) domain division restart by
HA-SP, our high availability server platform, and (3) service recovery by application process functions.
Our methods concern the platform of a computer system and require no changes in the application
processes, so they are applicable to other kinds of systems. Applying these methods to an OSS for theintelligent network service of IMT2000 (International Mobile Telecommunications 2000) reduced theservice down-time by 40%.

1. Introduction
Recently, with the growth of IP networks, the number of network elements (NEs) has increased rapidly
and so has the number that one operations support
system (OSS) [1] can manage. OSSs used to be used
only by network operators to monitor and control the
network for network service maintenance. Nowadays,
however, not only carrier operators but also network
service users control the network to open or change
network services or monitor the network to check theservice level agreement (SLA) between users and thecarrier. This trend is leading to an increase in the number of OSS users, so any interruption of OSS services
is becoming as serious as a network service interruption: the availability of OSS needs to be as high as thatof the network, as shown in Fig. 1.
These days, computer systems, including OSSs, arecomposed of commercially available servers, such asworkstations or personal computers for the following
reasons.
• Systems can be built at a low cost.
• Prevalent servers are made of state-of-the-art† NTT Network Service Systems Laboratories
Musashino-shi, 180-8585 Japan
E-mail: kimura.nobuhiro@lab.ntt.co.jp

40

technology.
We propose three methods that shorten the service
down-time to speed up service offerings and maintainthe operation service quality by using an OSS consisting of two or more ordinary servers, based onstate-of-the-art technology. In general, clustering
technology [2]-[4] is used to shorten service downtime. Our methods minimize the service down-timeand enable complete system restoration by improving
the clustering technology.
2. Current clustering technology
Clustering technology can be applied to ordinary
servers that use state-of-the-art technology. It treats a
set of servers as one system. When some processes
fail due to the failure of an application process orwhen all processes on one server fail due to the failure of hardware, this technology detects the failure
immediately and restores the failed process.
Suppose that a system is composed of two servers.
When trouble occurs in one server, the failed processes are restored in either the same server or the otherserver as shown in Fig. 2. Trouble that may occur inthe system is classified into two types.
• failure of an application process• failure of hardware, for example the server’s
CPU or memory
NTT Technical Review

Regular Papers

Network service users

Carrier operators

Increasing
Goal
Shorten the service down-time

OSS

NEs
Increasing

Fig. 1. Background.

Process A
Server 1

Server 2

Process A
Server 1(1) Software failure

Server 2
Restore the failed process on the same server

ActiveActiveActiveProcess A

Process B

Process A

Server 1

Server 2

Server 1

Process A

StandbyServer 2

Process A

Process B
Server 1(2) Hardware failure

Server 2

Server 1

Server 2

Restart services on another server

Fig. 2. Current clustering technology.

Vol. 1 No. 4 July 2003

41

Regular Papers

When a process on server 1 fails, clustering technology restarts it on the same server. On the other
hand, when some hardware on server 1 fails, all theprocesses running on server 1 are restored on server 2
if server 1 is judged to be malfunctioning.
Server 2 can be active or standby. If it is on standby, it is started when server 1 fails. In this case, theperformance of server 2 is the same as that of server
1. If server 2 is already active, running other processes, and the set of processes A is switched over to server 2, the performance of server 2 will drop because it
is running two sets of processes. But when the system
returns to normal, system resources can be usedeffectively on both servers again.

process X.
An individual process only interacts with a sharedprocess, which provides primitive services shared byan individual process. In this model, for example, if
process B fails, service 2 cannot be provided until
process B has been restored. However, services 1 and3 can both be provided continuously.
On the other hand, if process X fails, none of thethree can be provided because all three processes
need to use its primitive services. Moreover, in a large
system that has many processes, there is no guaranteethat other processes will not also be affected byprocess X failing. In other words, we cannot predict
how the failure of a shared process will affect services
in such a system.

3. Study model
4. Requirements
Generally, a service is provided through the cooperation of many processes. Figure 3 shows some relationships between services and processes. Cooperation between processes A and X provides service 1,that between B and X provides service 2, and thatbetween C and X provides service 3. Processes can beclassified into two groups: individual processes(processes A, B, and C) and shared processes such as
Service 1

Service 2

Service 3

Process A

Process B

Process C

There are two requirements for a high-availability
OSS.
1. Complete system restoration.
However severe the trouble, the system should berestored completely to provide services.
2. Minimal service down-time.
Failure of the OSS monitoring a network masks

Individual process
Shared process
ServiceCan be provided

ServiceCannot be provided

Process X

If process B fails, service 2 cannot be provided.

If process X fails, none of the three can be provided.

Service 1

Service 2

Service 3

Service 1

Service 2

Service 3

Process A

Process B

Process C

Process A

Process B

Process C

Process X

Process X

Fig. 3. Study model.

42

NTT Technical Review

Regular Papers

failures of the network. Therefore, OSS service
interruption affects not only OSS users but also network users. Therefore the service down-timeshould be as short as possible.
To meet the above requirements, we propose thefollowing three methods. The first is the “distributed
server restart method”, which restarts all the processes affected by the failure to enable complete system
restoration. The second is the “domain divisionrestart method” to localize the processes to be restarted. These two methods are individually provided in
our high availability server platform called HA-SP.
HA-SP can localize the process to be restarted. However, some service down-time is inevitable while afailed process is being restored. To shorten the service stoppage, our third method, the “service recovery method”, achieves cooperation between the platform and application process functions. This methodminimizes the time of service switchover from afailed server to another server. These methods concern the platform of a computer system and do notneed any changes to applications.
5. High Availability Server Platform (HA-SP)
5.1 Distributed server restart method

The distributed server restart method can restarteither individual processes or all processes as shownin Fig. 4. When an individual process A fails (top leftof Fig. 4), only the failed process is restarted. If ashared process fails, individual processes A, B, and C
cannot interact with the shared process until it has
been restored. Even when a shared process has been
restored, some processes may not be able to repeat
events that involve interaction with the sharedprocess. Thus, all processes are restarted to reestablish stable relationships between a shared process andindividual processes (bottom left of Fig. 4).
On the other hand, when a server fails, the system
is restored as follows. If all the processes on a failed
server 1 are individual ones, processes on otherservers provide services continuously and the failed
processes are restarted individually on other servers(top right of Fig. 4). If there is at least one sharedprocess on a failed server 2, not only the processes on
the failed server but also processes on other servers
are unable to provide services (bottom right of Fig.
4). In this case, the system isolates server 2 and all theprocesses are restarted on servers 1 and 3.
The process monitoring function is required to
implement this method. The operating system (OS)assigns a process identity (PID) to a process when it

Software failure

Hardware failure

Individual process is restarted.

Individual process is restarted.

Process A

Process B

Server 1

Server 2

Process C

Process A

Process B

Process C

Sharedprocess

Process A

Server 3

Server 1

Server 2

Server 3

Sharedprocess

All processes are restarted.

Process A

Process B

All processes are restarted.

Process C

Sharedprocess
Server 1

Server 2

Server 3

Process A

Process B

Process C

Process B

Sharedprocess

Sharedprocess

Server 1

Server 2

Server 3

Fig. 4. Distributed server restart method.

Vol. 1 No. 4 July 2003

43

Regular Papers

starts. And OS manages PID correspondence with theprocess name until the process fails or stops. If it fails
or stops, OS deletes the PID. This OS function
enables the process monitoring function to get the
PID when the process starts and continuously supervise it to detect process failure. Therefore, thismethod requires no change in the supervision of theprocess.
5.2 Domain division restart method
Because the distributed server restart method
restarts all processes when a shared process fails, itsuspends all the services provided by the system. To
minimize service down-time, the processes to berestarted should be localized to isolate the effect of
this method.
The domain division restart method divides a system into N domains, each having L servers as shownin Fig. 5. A shared process runs in each domain andprocesses related to the shared process run in therespective domains. Thus, each service can beprocessed within one domain. When a shared process
in one domain fails, all the processes in that domain
are restarted. However, processes in other domains
need not be restarted and can provide services continuously. Consequently, the domain division restart
Distributed serverrestart method

Process A

method can localize the processes to be restarted
within the domain.
5.3 Restart phase
Combining the distributed server and domain division restart methods, we define four restart phases asshown in Fig. 6.
Phase 1 restarts individual processes. It is applied
when an individual process fails.
Phase 2 restarts all processes in one domain.
Phase 3 restarts the OSs of all servers in one
domain. Either phase 2 or 3 is applied when a sharedprocess fails.
Phase 4 restarts the OSs of all servers in all
domains.
The recovery time increases in the order from 1 to
4 and the likelihood of recovery increases in the sameorder. When a failure occurs, an adequate restartphase that takes the shortest time for restarting should
initially be selected to correct the failure. However, if
that phase does not restore the failure, the method
escalates up the restart sequence. If the system stillcannot be restored by the “all restart phase”, thisphase is repeated. In almost all cases, this method canrestore a system completely and can shorten service
down-time.

Process B

Process C

Process D

Server 3

Server 4

Sharedprocess
Server 1

Server 2
N×L

Domain division
Domain divisionrestart method

Domain 1

Domain 2

Process A
Implement the sharedprocesses in eachdomainServer 1

Process B

Process C

Sharedprocess

Sharedprocess

Server 2

Server 1

L

Domain N

Process D

Server 2
L

Fig. 5. Domain division restart method.

44

NTT Technical Review

Regular Papers

Ordinary condition

Phase 1: Individual process restart.

Phase 2: All process domain restart.

Phase 3: All server domain restart.

Trouble occurs
Phase 4: All restart.

Trouble is fixed
Escalation

Fig. 6. Restart phase.

6. Service recovery method
The server should initialize all the processes in adomain when restoring services by applying phase 2
or 3 in the domain division restart method. In thatcase, the recovery time becomes long depending on
the number of processes. Moreover, the server
reboots the OS in phase 3. The recovery time
becomes long with the reboot time depending on thenumber of processes, or the characteristics of the OS.
Here, we propose a rapid service recovery methodthat reduces the influence of the number of processes
and hardware and OS characteristics.
Figure 7 shows a system composed of two domains,
each having two servers. An active process and astandby process are located at the same time in different domains. The active process actually provides
services while the standby process provides services
only when the active one fails. Generally, before
processes provide services, they have to be started
and initialized. We defined the active and standby
processes as follows.
• Active process: a process that has been started
and initialized• Standby process: a process that has been started but not initialized
This method simultaneously starts an active process
and a standby process in the different domains. Whileactive process A provides service in domain 1, thismethod can complete the OS reboot and start thestandby process in domain 2. If the server in domainVol. 1 No. 4 July 2003

1 fails, the server in domain 2 restores failed process
A rapidly just by initializing standby process A. The
server can skip the OS reboot and can start processes
while restoring process A. Therefore, switching from
active to standby processes hardly depends on thenumber of processes or OS characteristics.
Another merit of our proposal is low usage of system resources. Because the standby process is only
started but not initialized, it consumes memory but noCPU resources. However, if more rapid service
recovery is needed, active processes for the same services can be implemented in each domain.
7. Example of restart
These three methods enables complete and rapid
service recovery. When a shared process fails indomain 1, all or a group of processes are restarted toachieve complete system recovery by the distributed
server and domain division restart methods, as shownin the upper half of Fig. 8.
Thus, the more processes running in domain 1, thelonger the restart takes. Therefore, these two methods
should successfully shorten the down-time of services in domain 1. When the domain division restartoccurs in domain 1, all standby processes (processes
A and B) in domain 2 become active via the server
recovery method. In turn, processes A and B indomain 1 restart as standby processes as shown in thelower half of Fig. 8.
Unlike the general clustering method, which
45

Regular Papers

Running active and standby processes in different domains

Domain 1

Domain 2

Process A

Process B

Process A

OS

OS

OS

OS

ServerServerServerServerActive process

Process B

Change standby
process to active
process

Standby process
Process A

Start

Process A

Initialize

Start

Initialize

Fig. 7. Service recovery method.

Domain 1
All processes restart.
Domain 1

Process

ActiveProcess

Standby

Domain 2

OS

OS

OS

OS

Process A

Process B

Process C

Process D

Sharedprocess

Sharedprocess

Process D

Process C

Process B

Processes restart as standby ones.

Process A

All standby processes become active.

Domain 1

Domain 2

OS

OS

OS

OS

Process A

Process B

Process C

Process D

Sharedprocess

Sharedprocess

Process D

Process C

Process B

Process A

Fig. 8. Example of restart.

46

NTT Technical Review

Regular Papers

restores service but does not repair the failed domain
or server, our domain division restart method restoresdomain 1 automatically. For this reason, if a failure
occurs in domain 2 afterward, domain 1 can restore it
without interruption.

In this example, the OSS supervises 40 nodes.
Process A (which manages NEs 1 to 20) runs indomain 1 and process B (which manages NEs 21 to40) runs in domain 2 to balance the load. And theirstandby processes run in different domains so the service recovery method can be used. During initialization, active processes A and B send M-set requirements to EFD on the NE and are ready to receive
events from NEs. However, the standby processes,
which are not initialized, do not receive events fromNEs. Process X is duplicated and runs in both
domains 1 and 2 because the supervision processes
share it.
If domain 1 fails, active processes A and X indomain 1 stop. And standby process A in domain 2 isinitialized immediately and it sends the M-set
requirement to EFD in NEs 1 to 20. Using this example system, we measured the service down-time
before and after applying our three methods. The
results are shown in Fig. 10.
In this OSS, there are about 80 processes for downloading firmware, updating NE files, and so on.
Before we applied our three methods, there was only
one shared process (process X) located in one of theservers. Each supervision process was also single andlocated in each server. When the shared process
failed, all the processes were started and initialized.

8. NE supervision of OSS
Figure 9 shows an OSS example applying the threemethods. The OSS is used for controlling and supervising NEs in an intelligent network service of
IMT2000 (International Mobile Telecommunications
2000). The NE supervising function consists of acommunication process (process X) and supervision
processes (processes A and B). The communication
process establishes connections between the NEs andthe OSS. It also delivers messages from an OSS
application process to an NE, and vice versa. The
supervision process performs actions such as testingan NE, according to the messages it receives. In thismodel, NE and OSS communicate with each other bycommon management information protocol (CMIP)
and events from NE are filtered by an event forwarding discriminator (EFD) [5], [6] on the NE. In oursystem, NEs send an M-event-report after the OSS
has set the discriminator condition in EFD using theM-set requirements.
Domain 1 or server 1 fails

Active process
Domain 1 (server 1)

Domain 2 (server 2)Standby process

Process A

Process B

Process X

Process A

Process B

Process X

NE-A supervision

NE-B supervision

NEs 1 to 20

NEs 21 to 40
Change a process
NE-A supervision

Domain 1

NE-B supervision

Domain 2
Process A

Process B

Process X

NEs 1 to 20

NEs 21 to 40

Fig. 9. OSS applying our three methods.

Vol. 1 No. 4 July 2003

47

Regular Papers

1.0

1.0
Initializing
individual processes

Restart standby processes
0.6

Starting
individual processes

Initializing individual
processes

Starting
shared processes

Switching to activefrom standby
Domain #1

NE-A
supervision

NE-B
supervision

NE-A
supervision

Service down-time
before applying our threemethods

Domain #2
NE-A
supervision

NE-B
supervision

Service down-timeafter applying our threemethods

Fig. 10. Service down-time.

The services were interrupted until all restoration had
finished. As shown in the left of Fig. 10, the service
down-time consists of the starting of shared process,starting of individual processes, and initializing ofindividual processes. After we applied the threemethods, the service down-time was shorter, as
shown in the left of Fig. 10. The shared process failure in domain 1 did not affect the services for NE-B
supervision because the shared process existed indomain 2. This means that the service down-time for
NE-B was zero. For NE-A, compared with the left ofFig. 10, the down-time for starting the individual andshared processes was eliminated and replaced by theshorter time for switching the process from standby
to active. As a result, the down-time was reduced by40%.
While switching the process from standby to activein domain 2, the three methods restore domain 1.
Therefore, if a failure occurs in domain 2 afterward,
domain 1 can restore it without interruption. The time
to restart standby process in domain 1 is shown asdashed line in the right of Fig. 10. This time don’t
affect the service down-time for NE-A and NE-B.
9. Conclusionthat uses three new methods. The distributed serverrestart method can restart any process. The domain
division restart method divides a system into N
domains to limit the effect of the distributed serverrestart method. The domain service recovery method
runs an active process and a standby process in different domains, and switches the process in anotherdomain from standby to active if an active process
fails.
These methods have been integrated into a highavailability server platform called HA-SP that is
applicable to various systems. The distributed serverand domain division restart methods are independentof applications. The service recovery method isimplemented through application functions. They
were first applied to the OSS for the intelligent network of IMT2000. In this system, we could reduce
the service down-time by 40%. We also applied HASP to the OSS for VoIP (Voice over IP) service provisioning [7], [8] and to the OSS for equipment management of an MPLS-based IP network. We wereable to make these OSSs highly available without
modifying the application programs. These methods
enable a system to be restored by only restarting thefailed process.

We proposed a high-availability distributed OSS
48

NTT Technical Review

Regular Papers

References
[1]
[2]
[3]
[4]
[5]
[6]

[7]

[8]

Ishikawa, “High reliable switching node operation support system
based on server,” APCC/OECC, Oct. 1999.
http://www.sun.com/service/products/suncluster/index.html
http://www.hp.com/techservers/clusters/
http://www.veritas.com/vcsguided/index.html
ITU-T Recommendation X.734.
H. Seshake, T. Oimatsu, and N. Akiyama, “DATA Communication
Platform in Distributed Operation System Based on TMN,” IEEE,
Vol. 2, pp. 15-19, Apr. 1996.
T. Ichijo, K. Yamane, M. Aso, and Y. Hibino, “The OSS architecture
for VoIP service ordering based on the MSF network,” APSITT2001,
pp. 259-263, Kathmandu, Nepal/Atami, Japan, Nov. 2001.
T. Furukawa and K. Yamane, “A Study on Service Order of VoIP Service Including SLA Information,” APNOMS 2002, Jeju Island,
Korea, pp. 120-131, Sep. 2002.

Nobuhiro Kimura
Research Engineer, the First Promotion Project, Network Software Service Project, NTT
Network Service Systems Laboratories.
He received the B.E. and M.E. degrees in materials engineering from Nagoya University,
Nagoya, in 1993 and 1995, respectively. In 1995,
he joined NTT Network Service Systems Laboratories, Tokyo, Japan. He worked on the development of congestion control functions for thesystem, the OSS for IP networks, and the highavailability system platform. He is currently
working on IP control and management service
research.

Akira Yamada
Engineer, the First Promotion Project, NTT
Network Service Systems Laboratories.
He received the B.E. and M.E. degrees in computer science from Nihon University in 1994 and2000, respectively. Since joining NTT Network
Service Systems Laboratories in 2000, he has
worked on OSSs. His research interests include
integrated operations systems and recently enterprise information portals.

Hikaru Seshake
Senior Research Engineer, the First Promotion
Project, NTT Network Service Systems Laboratories.
He received the B.S. and M.S. degrees in science from Tokyo Institute of Technology, in1990 and 1992, respectively. In 1992, he joined
the Switching Systems Laboratories (now NTT
Network Service Systems Laboratories), Tokyo,
Japan. He worked on the development of congestion control functions for the system, adaptation
technology for enterprise application integration,and the high-availability system platform. He iscurrently working on IP control and management
service research.

Toshihiro Nishizono
Senior Research Engineer, Supervisor, Network Software Service Project, NTT Network
Service Systems Laboratories.
He received the B.S., M.S., and Ph.D. degrees
in information engineering from Kyushu University, Fukuoka, in 1978, 1980, and 1990, respectively. In 1980, he joined the Electrical Communication Laboratories, Nippon Telegraph andTelephone Public Corporation (now NTT),
Tokyo, Japan. He worked on the development of
DDX packet switching systems and ATM
switching systems. From 1990 to 1993, heworked in Advanced Telecommunication
Research Laboratories, Kyoto, Japan, on software specifications and distributed operating
systems. From 1998 to 2001, he was engaged ininternational communication service deployment
and IP service development in NTT Communications, Tokyo, Japan. He is currently working
on IP control and management service research.

Vol. 1 No. 4 July 2003

49