Facilitating failover to a secondary file server in a highly available file system

One embodiment of the present invention provides a system for elevating a secondary file server to act as a new primary file server in a highly available file system. Upon determining that a primary file server in the highly available file system has failed, the system promotes the secondary file server to become the new primary file server. During this process, the new primary file server scans file objects to look for a file lock indication. Upon finding a file lock indication, the new primary file server converts an associated file identifier into a virtual node. Otherwise, conversion of file identifiers into virtual nodes is delayed until the first time a file is subsequently accessed by the new primary server, thereby speeding up the failover process.

BACKGROUND

1. Field of the Invention

The present invention relates to highly available file systems. More specifically, the present invention relates to a method and apparatus for facilitating a failover to a secondary file server upon failure of a primary file server in a highly available file system.

2. Related Art

Highly available file systems allow a client to continue accessing files from a storage device when a primary file server for the storage device fails or otherwise becomes inaccessible. Highly available file systems typically make use of multiple file servers that are coupled to one or more multi-ported storage devices. During operation, one file server is designated as the primary file server. This primary file server services all requests to access files on the storage device. The other file servers act as backup secondary file servers, which do not normally access the storage device. The primary file server periodically sends checkpoint information to the secondary file servers. This allows the secondary file servers to maintain copies of state information from the primary file server.

When the primary file server subsequently fails or otherwise becomes inaccessible, a secondary file server is promoted to act as a new primary file server. This process of promoting a secondary file server to act as a new primary file server is referred to as a “failover.” Promoting the secondary file server allows operations in progress to complete or to be automatically retried.

In some operating systems, a primary file server maintains file objects, which contain a pointer to a virtual node (vnode). A vnode holds state information that is used to access the file on the underlying file system. The secondary file server also maintains file objects. However, instead of including pointers to vnodes, these file objects include pointers to file identifiers (FIDs). FIDs can be used to identify a file on the storage device. However, an FID must first be converted to a vnode before the file can be accessed.

Hence, during a failover operation, the new primary file server must convert FIDs to vnodes. This can be an extremely slow process because a large number of FIDs may have to be converted to vnodes, and furthermore, each conversion may require an access to the storage device. During this conversion process, the new primary file server cannot accept new file requests. This can cause long waits for client applications that need to access the file system, which result in timeouts or other failures.

What is needed is a method and an apparatus that facilitates failover to a secondary file server in a highly available file system without the problems described above.

SUMMARY

One embodiment of the present invention provides a system for elevating a secondary file server to act as a new primary file server in a highly available file system. Upon determining that a primary file server in the highly available file system has failed, the system promotes the secondary file server to become the new primary file server. During this process, the new primary file server scans file objects to look for a file lock indication. Upon finding a file lock indication, the new primary file server converts an associated file identifier into a virtual node. Otherwise, conversion of file identifiers into virtual nodes is delayed until the first time a file is subsequently accessed by the new primary server, thereby speeding up the failover process.

In one embodiment of the present invention, the file lock indication is stored within a file object referenced by a list of file objects.

In one embodiment of the present invention, the system converts a file identifier into a corresponding virtual node during a subsequent access to the file object by the new primary file server.

In one embodiment of the present invention, converting the associated file identifier into the virtual node involves locating a file referenced by the associated file identifier and reading the file.

In one embodiment of the present invention, the system periodically sends a checkpoint from the primary file server to the secondary file server.

In one embodiment of the present invention, the checkpoint includes the file identifier associated with the virtual node.

In one embodiment of the present invention, a checkpoint is sent for each state change at the primary file server.

DETAILED DESCRIPTION

Highly Available File System

FIG. 1Aillustrates highly available file system100in accordance with an embodiment of the present invention. Highly available file system100includes primary server104and secondary server106. Primary server104and secondary server106can generally include any nodes on a computer network including a mechanism for servicing requests from a client for computational and/or data storage resources. Note that highly available file system100may include more than one secondary server.

Primary server104and secondary server106are coupled to disk112. Disk112can include any type of system for storing data in non-volatile storage. This includes, but is not limited to, systems based upon magnetic, optical, and magneto-optical storage devices, as well as storage devices based on flash memory and/or battery-backed up memory. Disk112can include a redundant array of inexpensive (or independent) disks (RAID) system. Moreover, disk112includes multiple ports for communicating with file servers. Note that disk112may include more than the two ports shown in FIG.1A.

Primary server104and secondary server106include failover mechanisms108and110, respectively. Failover mechanisms108and110form a distributed management system for disk112, which operates as described below.

Primary server104and secondary server106are also coupled to network102. Network102can generally include any type of wire or wireless communication channel capable of coupling together computing nodes. This includes, but is not limited to, a local area network, a wide area network, or a combination of networks. In one embodiment of the present invention, network102includes the Internet.

Failover Mechanisms

FIG. 1Billustrates the structure of failover mechanism110in accordance with an embodiment of the present invention. Failover mechanism110includes failure detector114, file identifier converter120, and checkpoint mechanism122.

Failure detector114operates in conjunction with the corresponding failure detectors in other servers to detect when primary server104fails. Techniques for detecting failure of primary file server104are know in the art and will not be discussed further herein. Upon detecting failure of primary server104, failover mechanism110promotes secondary server106to be the new primary server.

After secondary server106has been promoted to be the new primary server, failover mechanism110scans a file object list as described below in conjunction withFIGS. 3 and 5to determine if the associated file object includes a lock indication. A file objects that includes a lock indication has its file identifier immediately converted into a virtual node (vnode). In contrast, a file object without a lock indication delays having its file indicator converted into a vnode until the associated file is subsequently accessed.

Checkpoint mechanism122within primary server104sends checkpoints to a corresponding checkpoint mechanism122within secondary server106so that secondary server106is aware of file operations in progress within primary server104. These checkpoints include a file identifier (FID) for each file in use by primary server104. Note that primary server104maintains a vnode for each file rather than an FID.

FIG. 2Aillustrates the process of checkpointing in a highly available file system in accordance with an embodiment of the present invention. During operation, primary server104receives a file access request from network102and accesses disk112through I/O channel204to respond to this request.

Primary server104periodically sends checkpoints202to secondary server106so that secondary server106can maintain data related to file operations in progress on primary server104. If primary server104fails, secondary server106makes use of the data provided in checkpoints202to become a new primary server.

Failure of Primary Server

InFIG. 2B, primary server104has failed and secondary server106has been promoted to be a new primary. Upon being promoted to be a new primary server, secondary server106converts FIDs associated with locks into vnodes and then retries uncompleted file operations if necessary. FIDs not associated with locks are not converted into vnodes until a subsequent access to the file occurs. Note that secondary server106communicates with disk112across I/O channel210.

Data Structures

FIG. 3illustrates a number of data structures within a highly available file system in accordance with an embodiment of the present invention. Note thatFIG. 3includes portions of file system client302, primary server104, and secondary server106. File system client302includes virtual node data308related to a file within the highly available file system. File system client302receives data from the associated file and updates the associated file through interface318. Interface318accesses file object310within primary server104. File object310, in turn, accesses file system vnode314using backpointer322within file object310. Note that file object list304includes a pointer to every file object on primary server104, including file object310.

Primary server104sends checkpoints202to secondary server106. Secondary server106uses checkpoints202to maintain file object312. Note that file object312is substantially the same as file object310except that backpointer324within file object312points to file identifier316rather than a vnode. File object list306includes a pointer to each file object within secondary server106.

File Object

FIG. 4illustrates file object310in accordance with an embodiment of the present invention. File object310includes has_locks402and back_object404. Has_locks402is a Boolean variable indicating whether the related file has been locked by an application. Typically, open files are locked only when an update operation is in progress, therefore, has_locks402is typically false. Back_object404is a pointer pointing to the file system vnode314. The respective back_object pointer in file object312points to file identifier316. Note that the has_locks Boolean variable within file object312has the same state as the has_locks Boolean variable402within file object310.

Converting a Secondary Server

FIG. 5is a flowchart illustrating the process of promoting a secondary file server to be a new primary file server in accordance with an embodiment of the present invention. The system starts when secondary server106determines that primary server104has failed (step502). In response to this determination, secondary server106is promoted to be a new primary server (step504). During this process, the new primary server stops accepting new I/O requests (step506).

After stopping the new I/O requests, the new primary server inspects the file objects in a loop to determine which files have their has_locks Boolean variables set to true. The first action in the loop is to get a file pointer from the file object list (step508). Next, the new primary server determines if the has_locks Boolean variable is true (step510). If so, the file identifier converter converts the file identifier to a vnode (step512). Note that converting the file identifier involves accessing the disk to read the file. After this conversion, or if the has_locks Boolean is false, the new primary server determines if there are more pointers in the file object list (step514). If so, the process returns to step508to retrieve the next file object pointer. Note that the new file server converts only files with has_locks Boolean variables set to true, which can save a considerable amount of time during the failover process.

After scanning the entire file object list, the new primary file server retries any outstanding I/O requests (step516). Finally, the new primary file server resumes accepting new I/O requests (step518).

FIG. 6is a flowchart illustrating the process of converting a file identifier into a vnode in accordance with an embodiment of the present invention. This process takes place the first time a file is accessed by a new primary file server after a failover. During this first access, the primary file server retrieves a back pointer from a file object associated with the file (step602). Next, the primary file server determines if the back pointer points to a vnode or to an FID (step604). If the back pointer points to an FID, the primary file server converts the FID into a vnode (step606). After the vnode is created, the system resumes operation (step608).