Patent Publication Number: US-11650961-B2

Title: Managing replica unavailability in a distributed file system

Description:
TECHNICAL FIELD 
     The present disclosure relates generally to distributed file systems. More specifically, but not by way of limitation, this disclosure relates to managing replica unavailability in a distributed file system. 
     BACKGROUND 
     A distributed file system stores directories and files in a hierarchical structure across multiple disks of multiple storage servers. Distributed file systems are different from, for example, object storage in which data is stored in a flat non-hierarchical namespace. One popular type of distributed file system is the Gluster file system, or GlusterFS. 
     Distributed file systems store data in bricks. A brick is a storage area on a physical disk (e.g., hard drive) or a virtual disk. Multiple bricks can be aggregated into a storage volume, or “volume,” such that a single volume serves as a logical collection of two or more bricks spanning one or more servers in the distributed file system. 
     Volumes can be mounted as regular directories on client devices. This enables applications on the client devices to request that various file operations (e.g., read or write operations) be executed on the data in the volumes. And if a volume spans multiple bricks on multiple storage servers, the storage servers can collaborate to carry out the file operations requested by the applications. 
     In some cases, a volume can include one or more replicas (e.g., copies) of data stored in a brick. These replicas can be stored in their own bricks within the volume. The replicas can serve as backups in case the original data becomes unavailable, for example, due to the corruption or failure of the brick in which the original data is stored. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG.  1    is a block diagram of an example of a system for managing replica unavailability in a distributed file system according to some aspects. 
         FIG.  2    is a block diagram of another example of a system for managing replica unavailability in a distributed file system according to some aspects. 
         FIG.  3    is a flow chart of an example of a process for managing replica unavailability in a distributed file system according to some aspects. 
     
    
    
     DETAILED DESCRIPTION 
     Some distributed file systems require that all replicas of a piece of data be stored in the same volume as the piece of data. This creates numerous advantages, such as better isolation, easier administration, and the ability to easily apply the same properties to all of the data in the volume. But this also creates numerous problems. For example, if a replica becomes unavailable (e.g., due to a problem with a corresponding brick), the distributed file system may attempt to create another replica in the volume in order to fill the void. But if the distributed file system is unable to create the new replica (e.g., due to there being insufficient space in the volume), the distributed file system will be left without recourse. This loss of redundancy renders the distributed file system more susceptible to corruption and errors, and may result in reduced data availability and throughput. 
     Some examples of the present disclosure overcome one or more of the abovementioned problems by creating a copy of a piece of data in a designated memory location, or “hand-off location,” that is within the distributed file system but outside the volume in which the piece of data is stored. The distributed file system can create the copy in the memory location upon detecting that one of the replicas within the volume is unavailable, in order to supplement the system until the replica becomes available again. Once the replica within the volume becomes available again, the distributed file system can delete the copy in the designated memory location (e.g., to conserve resources). This can help ensure that the distributed file system maintains its redundancy in the face of replica failures, which reduces errors and improves data availability. 
     As a particular example, the distributed file system can be a modified version of GlusterFS in which adjustments have been made to at least the automatic file replication (AFR) module to implement various aspects of the present disclosure. In this example, the distributed file system can store data in brick A of a volume. The distributed file system can also store a replica of the data in brick B of the volume. If brick B becomes unavailable (e.g., due to hardware failure or corruption), the modified AFR module can automatically copy of the data to a memory location outside of the volume. The memory location may or may not include another volume within the distributed file system. In some examples, the memory location is backed by one or more high-performance disks, such as solid-state hard drives. The copy of the data in the memory location can effectively serve as another replica of the data, which the distributed file system can then use to perform various tasks. 
     For example, the distributed file system can load balance read requests for the data by fulfilling some of the read requests using the data stored in brick A and others of the read requests using the copy stored in the memory location outside the volume. This can significantly increase the speed with which read requests are handled by the system. 
     In some examples, the distributed file system can maintain consistency between the data stored in brick A and the copy stored in the memory location. For example, in response to a write request to modify the data in brick A, the modified AFR module can not only modify the data in brick A, but also correspondingly modify the copy in the memory location to maintain consistency between both locations. 
     At a future point in time, the replica in brick C may become available again. If the replica in brick B becomes available, the modified AFR module can automatically synchronize the replica in brick B with the copy in the memory location, to ensure consistency between the two. For example, the modified AFR module can overwrite the replica in brick B with the memory location&#39;s data. The modified AFR module can then delete the copy from the memory location, thereby freeing up the memory location for subsequent use. Thus, the memory location serves as a temporary location for storing a temporary copy of the data in brick A. 
     These illustrative examples are given to introduce the reader to the general subject matter discussed here and are not intended to limit the scope of the disclosed concepts. The following sections describe various additional features and examples with reference to the drawings in which like numerals indicate like elements but, like the illustrative examples, should not be used to limit the present disclosure. 
       FIG.  1    is a block diagram of an example of system for managing replica unavailability in a distributed file system  100  according to some aspects. The distributed file system  100  can include any number and combination of nodes (e.g., storage nodes). In this example, the distributed file system  100  includes nodes  102   a - d , such as storage servers. At least some of the nodes  102   a - d  include bricks for storing data. For example, node  102   a  includes bricks  104   a - 1  through  104   a - 3 , node  102   b  includes bricks  104   b - 1  through  104   b - 4 , and node  102   c  includes bricks  104   c - 1  through  104   c - 2 . But in other examples, each of the nodes  104   a - d  can have other amounts and combinations of bricks. 
     Multiple bricks can be combined to form a volume. For example, bricks  104   a - 1 ,  104   b - 3 , and  104   c - 2  collectively form a single volume  108 . Other combinations of bricks may also form other volumes in the distributed file system  100 . 
     In this example, volume  108  includes data  106  and two replicas of the data  106 . These replicas are designated Replica A (on node  102   b ) and Replica B (on node  102   c ) in  FIG.  1   . But other examples can involve more or fewer replicas of the data  106  on more or fewer nodes in the distributed file system  100 . 
     At some point in time, one or both of Replicas A and B may become unavailable. This can slow down the distributed file system  100 , reduce its ability to respond to read requests for the data  106 , and reduce redundancy, which can result in a variety of problems. To avoid these problems, the distributed file system  100  can detect that one or both of Replicas A and B are unavailable and responsively take corrective action. 
     For example, the distributed file system  100  can determine that brick  104   b - 3  has gone offline, rendering Replica A unavailable. In response, the distributed file system  100  can create a copy  112  of the data  106  in a memory location  110  that is outside of the volume  108  (but still within the distributed file system  100 ). The memory location may be a preset location specifically designated for storing copies of the data  106  when replicas of the data  106  become unavailable. For example, the preset location may be a user-designated location specifically for these purposes. The memory location  110  can be a physical memory location (e.g., a hard drive) or a virtual memory location. For example, the memory location  110  can include another volume formed from one or more bricks on node  102   d . As another example, the memory location  110  can be a location in random access memory (RAM). The copy  112  can serve as a temporary replica usable by the distributed file system  100 . 
     After creating the copy  112  of the data  106 , the distributed file system  100  can use the copy  112  in various ways. For example, the distributed file system  100  can balance a workload between the data  106  and the copy  112 , such that at least one part of the workload is handled using the data  106  and another part of the workload is handled using the copy  112  in the memory location  110 . In one such example, the distributed file system  100  can receive read requests from client device(s)  114 , where a read request involves reading the data  106  from the distributed file system  100 . The distributed file system  100  can respond to the read requests by distributing them at least partially among node  102   a  and node  102   d , such that node  102   a  responds to its subset of read requests using the data  106  in brick  104   a - 1  and node  102   d  responds to its subset of read requests using the copy  112  in the memory location  110 . This can significantly speed up the distributed file system&#39;s ability to respond to the read requests. 
     In other examples, the distributed file system  100  can receive write requests from the client device(s)  114 . A write request involves somehow modifying the data  106 , such as by adding information to the data  106  or removing information from the data  106 . In response to a write request, the distributed file system  100  can modify the data  106  accordingly. The distributed file system  100  may also modify some or all available replicas, such as Replica B, to keep the replica(s) in synch with the data  106 . The distributed file system  100  can further modify the copy  112  in the memory location  110  to keep the copy  112  in synch with the data  106 . This can help ensure that the available replicas and/or the copy  112  in the memory location  110  are always up-to-date. 
     At some point, Replica B may become available again. The distributed file system  100  can detect that Replica B is available and automatically synchronize Replica B with the copy  112  (or the data  106 ) to ensure consistency between the two. For example, the distributed file system  100  can overwrite Replica B using the copy  112  in the memory location  110 . After updating Replica B, the distributed file system  100  can delete the copy  112  from the memory location  110 , thereby freeing up the memory location  110 . 
     While the above example involves the creation of one temporary copy  112  in one memory location  110 , other examples can involve the creation of any number any combination of temporary copies in any number and combination of memory locations external to the volume  108  (e.g., but still within the distributed file system  100 ). For example, if the distributed file system  100  detects that both Replica A and Replica B are unavailable, the distributed file system  100  can generate one copy of the data  106  in one memory location and another copy of the data  106  in another memory location. These copies can serve as temporary proxies for Replicas A and B while the replicas are unavailable. The distributed file system  100  can then use these copies as discussed above. For example, the distributed file system  100  can use both copies to perform load balancing. As another example, the distributed file system  100  can execute write requests at least partially by updating the copies, in order to maintain consistency between the data  106  the copies in the memory locations. 
       FIG.  2    is a block diagram of another example of a system  200  for managing replica unavailability in a distributed file system  100  according to some aspects. The system  200  includes a processing device  202  communicatively coupled with a memory device  204 . In some examples, the processing device  202  and the memory device  204  can be part of a node, such as one of the nodes  102   a - d  of  FIG.  1   . 
     The processing device  202  can include one processing device or multiple processing devices. Non-limiting examples of the processing device  202  include a Field-Programmable Gate Array (FPGA), an application-specific integrated circuit (ASIC), a microprocessor, etc. The processing device  202  can execute instructions  206  stored in the memory device  204  to perform operations. In some examples, the instructions  206  can include processor-specific instructions generated by a compiler or an interpreter from code written in any suitable computer-programming language, such as C, C++, C#, etc. 
     The memory device  204  can include one memory device or multiple memory devices. The memory device  204  can be non-volatile and may include any type of memory device that retains stored information when powered off. Non-limiting examples of the memory device  204  include electrically erasable and programmable read-only memory (EEPROM), flash memory, or any other type of non-volatile memory. In some examples, at least some of the memory device can include a medium from which the processing device  202  can read instructions  206 . A computer-readable medium can include electronic, optical, magnetic, or other storage devices capable of providing the processing device  202  with computer-readable instructions or other program code. Non-limiting examples of a computer-readable medium include magnetic disk(s), memory chip(s), ROM, random-access memory (RAM), an ASIC, a configured processor, optical storage, or any other medium from which a computer processor can read the instructions  206 . 
     In some examples, the processing device  202  can detect that a replica  210  of data  106  in a volume  108  of a distributed file system  100  is unavailable. In response to detecting that the replica  210  is unavailable, the processing device  202  can create a copy  112  of the data  106  in a memory location  110 . The memory location  110  is external to the volume  108  and may (or may not) exist within the distributed file system  100 . The processing device  202  can then use the memory location  110  to perform various tasks. For example, the processing device  202  can receive a write request  208  involving modifying the data  106  in the volume  108 . In response, the processing device  202  can execute the write request  208  at least partially by modifying the copy  112  in the memory location  110 . For example, the processing device  20  can modify both the data  106  in the volume  108  and the copy  112  in the memory location  110 , for example, to maintain consistency between the two. 
     In some examples, the processing device  202  can implement some or all of the steps shown in  FIG.  3   . Other examples can include more steps, fewer steps, different steps, or a different order of the steps than is shown in  FIG.  3   . The steps of  FIG.  3    are discussed below with reference to the components discussed above in relation to  FIG.  2   . 
     In block  302 , a processing device  202  detects that a replica  210  of data  106  in a volume  108  of a distributed file system  100  is unavailable. For example, the processing device  202  can detect that the replica  210  is unavailable by attempting (e.g., periodically attempting) to access the replica  210 . If the processing device  202  cannot access the replica  210 , then the processing device  202  can determine that the replica  210  is unavailable. As another example, the processing device  202  can detect that the replica  210  is unavailable by attempting to access a brick in which the replica  210  is stored. If the processing device  202  cannot access the brick, then the processing device  202  can determine that the replica  210  is unavailable. 
     In block  304 , the processing device  202  creates a copy  112  of the data  106  in a memory location  110  that is within the distributed file system  100  and external to the volume  108 . The processing device  202  can create the copy  112  in response to detecting that the replica  210  is unavailable. The memory location  110  can include one or more storage areas on one or more disks. 
     In block  306 , the processing device  202  executes a write request  208  for modifying the data  106  in the volume  108  by modifying both (i) the data  106  in the volume  108 , and (ii) the copy  112  in the memory location  110 . In some examples, the data  106  and the copy  112  are both modified in the same way to ensure consistency between the two. 
     The foregoing description of certain examples, including illustrated examples, has been presented only for the purpose of illustration and description and is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. Numerous modifications, adaptations, and uses thereof will be apparent to those skilled in the art without departing from the scope of the disclosure. For instance, any example(s) described herein can be combined with any other example(s) to yield further examples.