Patent ID: 12192282

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT(S)

Referring toFIGS.2-3and4A-4B, the subject system10is a data migration and storage system which includes a number of compute nodes. The compute nodes in the present system may be either in the form of storage server nodes12or client nodes24, or their combination. The storage server nodes (also referred to herein as peer nodes, servers nodes, servers, or storage nodes)12, as well as the client nodes24, may be arranged in groups (also referred to herein as clusters)14. Both clients and servers cooperate to perform complex computations of various types in numerous areas of applications. The servers also store the results of the computations on the storage devices for short- or long-term storage and retrieval when needed. The clients24are represented by local clients13and remote clients26, as will be detailed in further paragraphs.

As shown inFIGS.2-3and4A-4B, the compute nodes12are operatively connected through a network communication channel50. The communication channel50is formed between two compute nodes12(or between the node12and client13,26) to communicate messages therebetween. A node sending messages will be further referred to herein also as an initiating node54, and a node receiving messages will be further referred to herein also as a target node56. Each of the initiating node54and the target node56may be a server node12, a client node13,26, and the mix thereof.

The subject Multi-Path sub-system52is embedded in the system10to support messages transmission over multiple pathways routed on network fabric types of the same or different types between the initiating node54and target node56. Specifically, the subject Multi-Path sub-system52is configured for supporting whole messages transmitting protocol where the whole message is created and sent over a “selected” pathway. If the message sends correctly, the transmission is completed, but if, however, the message is not sent in a correct manner, a different pathway is selected, and the message is sent over the different pathway which can be routed via the same or a different fabric type. The transmission of the whole message continues until either no pathways remain available or until the message is sent.

The Multi-Path sub-system52is envisioned to be applicable to numerous data migration and storage systems. In an exemplary embodiment of the numerous implementations of the system10, such may be also referred to as a RED system which will be detailed in further paragraphs. The RED system10may include one or a plurality of RED clusters14.

As shown inFIGS.2-3, the transmission of messages is attained in the present RED system10between multiple initiator interfaces16embedded in the initiator node54and multiple target interfaces17embedded in the target node56in the enhanced fashion provided by the Multi-Path sub-system52. This also supports load balancing the messages and RDMA (Remote Direct-Memory Access) operations over multiple connections60,62. AlthoughFIGS.2-3show only two network fabric types (Infiniband and Ethernet) with the connections60,62, it is to be understood that the number of network fabric types, as well as the number of connections (pathways) in the communications channel50between the nodes54,56is not limited by those shown inFIGS.2-3, and any number of pathways, as well as other network fabric types, are applicable in the present system10enhanced by the Multi-Path sub-system52.

The network communication channel50between the compute nodes12, as well as between nodes12and the clients13,24,26, may be arranged via a high-speed network, as well as via various intermediate architectural structures, commonly referred to herein as an operation supporting and enhancing layer, which may be implemented in the form of an intermediate storage layer (which may also be referred to as a burst buffer tier) which may serve as a mediator between the high performance computing architectures (such as clients26, and compute nodes12), or any other implementations.

The High Speed Network (HSN) functions as a high speed switch, and may be based on any of the network transport protocols, such as, for example, InfiniBand (IB), fiber channel (FC), gigabit Ethernet (GIGE), wireless LaN (WLAN), etc. In addition, the HSN may conform to the cascade, as well as Gemini architecture, may use optical fiber technology, may be proprietary, etc. Although the high speed network may include a single transport layer fabric type, the core feature of the subject system10is its ability to communicate messages between two nodes in the system (for example, between any initiating node54and any targeted node56in the RED cluster14) over the transport layer formed by different network fabric types.

The present Multi-Path sub-system52supports the data migration and data storage through a dynamic networking layer (channel)50which can load balance messages and RDMA (Remote Direct Memory Access) operations over many pathways (transport layers of various network fabric types). The operation of the present sub-system52also supports the ability to self-heal if pathways (transport layers) experience transient network faults.

Referring toFIG.2, which represents a no-core (no core affine) routing embodiment of the present system10, the network communication channel50in the present system10is established between the nodes (Node A and Node B), which are also referred to as the initiating node54and the target node56. The nodes54and56inFIG.2may be either client nodes, or server nodes, or combination thereof. The client is a computer or a process that accesses the services or recourses of another process or computer on the network. The server is a computer that provides service recourses, and that implements network services. The initiating node54(which may be a node12or a client node24) communicates with the target node56(which may be any of the storage nodes12or the client node24) in the system10.

As exemplified inFIG.2, the initiating node54and the target node56are connected through the Infiniband fabric64and the Ethernet fabric66. The initiating node54and the target node56are shown as having one Infiniband connection60to an Infiniband fabric64, and two Ethernet connections62to an Ethernet fabric66. Thus, five pathways are established for communication between the nodes54,56. It is to be understood however thatFIG.2is one of the numerous exemplary embodiments, and different number of connections, as well as alternative network fabric types, may be used as well, including, for example, FC, GIGE, WLAN, cascade, as well as Gemini architecture, etc.

As further depicted inFIG.2, each of the initiating node54and the target node56is built with numerous interfaces16,17, respectively, where the initiating interfaces16are embedded into the initiating node54while the target interfaces17are embedded in the target node56. Each interface16is able to “talk” to the interfaces17, while each interface17can “talk” to the interfaces16. Thus, the embodiment system shown inFIG.2provides five network paths over which the connections for communication between the nodes54and56can be accomplished.

Referring toFIG.3, another embodiment of the present system10is depicted which supports the core-routing Multi-Path solution, also referred to herein as a core-affine embodiment. In the embodiment of the system10depicted inFIG.3, the initiating node70includes initiating interfaces72and CPU cores74which are interconnected with the initiating interfaces72on the initiating node70. Likewise, the target node76includes target interfaces78and CPU cores80interconnected with the target interfaces78on the target node76.

Similar to the Multi-Path no-core routing embodiment shown inFIG.2, the Multi-Path core routing embodiment shown in inFIG.3has each node connected to various transport layers such as, for example, to the Infiniband fabric64and the Ethernet fabric66.

Specifically, as shown inFIG.3, each node70,76has an interface connected to the Infiniband fabric64via Infiniband connections60, while the interfaces in the initiating node70and the target node76are connected to the Ethernet fabric66via two Ethernet connections62. Each CPU core74and80in the initiating node70and the target node76, respectively, can establish connections over various pathways, between the two nodes70and76. Each initiating CPU cores74can directly access the initiating interfaces72via the connections71in the initiating node70, while each target CPU core80can directly access the local target interfaces78in the target node76via the connections77. Once a respective connection71,77is created between the CPU cores and their local interfaces, such connection is “anchored” to the CPU core, and only that CPU core can read-write over this connection. Such arrangement prevents locks and memory sharing, and thus, is important for the system performance.

The subject Multi-Path sub-system52is applicable to various applications which are both core-affine such as, for example, the HPC (High Performance Computing), shown inFIG.3, and those which are not core-affine, as shown inFIG.2. The balancing of messages/RDMA (Remote Direct Memory Access) operations and handling of network errors are all attained with a simple send/receive API (Application Programming Interface). Application only requires specification of the target node for a message and optionally, a CPU core number, in the case of the core affine target, shown inFIG.3.

An important advantage of the subject Multi-Path sub-system52is that it operates on Remote Procedure Call (RPC) “messages” (as opposed to the packets in the existing systems), for example, the Multi-Rail system which slices a message into packets, usually based on the MTU (message transmitting unit), which transmit over the same TCP/IP fabric corresponding to the packets headers.

In the subject Multi-Path sub-system52, the complete (or whole) messages (as opposed to the data packets) are load balanced over the interfaces. A single complete message may be sent over the same interface and therefore, via the connected fabric, however, a series of messages can each be sent over different interfaces to different fabric types. Thus, one message can be sent over, for example, a TCP/IP network and another message can be sent over, for example, the Infiniband network or Ethernet network, or any other fabric type transport layer which is used as the network communication channel50in the present system10. Also, is the communication pathway selected for transmission of the whole message in question fails, an alternative communication pathway is selected which may be configured in the same or a different fabric type transport layer.

The cluster14includes multiple server nodes12, and may include local client nodes13. Each node12may be operatively coupled to other nodes12residing within the same cluster14or residing within different clusters14with a shared network having multiple network interfaces16. In addition, each node12in the cluster14may be connected with the local client13(logically presented within the same cluster14). Alternatively, as required by the system operation, a server node12may also communicate with a remote client26residing external the cluster of the server node12.

As the subject system10is further exemplified as the RED system, the clusters will be further referred to as RED clusters14, and the nodes in the present disclosure will be further referred to as the RED storage's servers12, and/or clients13,24,26which have access to the corresponding RED storage servers12, and/or their combination. Any node12,13,24,26is configured for usage of the subject Multi-Path approach (also referred to herein as a Multi-Path sub-system) as will be detailed in further paragraphs.

In the RED cluster14, the traffic profile is defined as either a “client-to-server” or “server-to-server”. The RED storage server(s) associated with each node12are configured to support the communication between RED storage servers. However, the Multi-Path approach is not limited to “client-to-server” or “server-to-server” architectures, but is rather referred to as “node-node” architectures, where compute nodes12may be any mix of clients or servers.

As shown inFIGS.4A-4B, each node12in the RED cluster14may be configured with an Agent20(also referred to herein as an Agent Processing Unit) which supports a Management Process running on the node12which monitors the nodes12and their functionality. Each node12is configured with a plurality of storage devices18, which may also be referred to herein as Core Affined Storage Targets (CATs). The data storage system in the RED cluster14in the present system10exploits a large array of non-volatile memory (NVM) devices18which are arranged in a plurality of clusters14(or containers) where the nodes12may communicate with other nodes12in the same storage cluster (or container)14or with specified nodes in other RED clusters14.

Each compute node12is also configured with a corresponding Instance22(also referred to herein as an Instance IO Server Processing Unit) which supports an IO server process running on the compute node12. Instances22are generated (created) by the Agents20. Instances22“own” the local CATs18and supply IO services to clients24. As best shown inFIG.4A-4B, clients24are represented by the local clients13(local to the compute nodes12) and remote client(s)26.

As shown inFIGS.4A and4B, the RED system10further includes ETCD cluster30which is a highly available distributed configuration database. The ETCD cluster30stores cluster configuration32, runtime34, layout table36, agent registry38, and Inventory40. The ETCD Cluster30also includes databases ETCD31which, as best shown inFIG.4B, reside in respective compute nodes12.

Also depicted inFIGS.4A and4Bis a set of tool utilities42which are configured for communication with the Agents20and the ETCD cluster30to create, start, stop the RED cluster operation, OR query the status, etc.

FIG.4Bis representative of the physical view of an exemplary embodiment of the RED cluster14. In one of numerous exemplary embodiments, the compute nodes12in the RED cluster14may run various Linux containers using Docker/k8s. In this exemplary embodiment, the Agent20and the Instance22run in a single Linux container, with the Agent container controlling the access to the local CATs18. As shown inFIG.4B, in one of the possible embodiments, three ETCD servers31of the ETCD cluster30run on any three compute nodes12in the RED cluster14. The local client13and tools utilities42can run on any compute node12other than the RED cluster nodes12, to perform IO operations to the RED cluster14. As depicted inFIG.4B, the RED cluster14may be operatively coupled to a remote IO client26and the tool utilities42.

The Instance IO server processing unit22, also referred to herein as Instance, resides in the RED cluster14. The Instance IO server processor unit22is executed by different threads running within the Instance22. These threads may include a Cluster Manager thread which is responsible for handling the cluster management, Instances and CATs, Evicts and joins from and to the RED Cluster14, makes all the intelligent decisions to grow and/or shrink the Cluster, and communicates with ETCD cluster, as well as publishes the run time.

The Instance22further includes an Instance Manager thread which is responsible for Local Instance configuration management, and monitors and controls all of the needs of the Local Instance.

The Instance Manager is also responsible for, among other functions, controlling the openings and closures of the CATs, sampling the gossip operation from the Gossip Manager and publishes the Gossip to the CATs.

Reactor threads48are also included in Instance22. The Reactors48are bound to a specific RED Cluster and run different tasks. For example, Reactors run handler tasks (handler processing units) which are responsible for replication and routing requests from clients/servers to local CATs or remote Instances22. The reactor thread may run a task which may be specific to Gossip manager which is responsible for the scalable broadcasts of essential information.

Gossip Manager provides Gossip information to all sub-systems, while the Cluster Manager and Instance Manager may publish data and information over the Gossip messages.

The Instance22process also may include another reactor which runs the task for the CAT dedicated to another reactor. All activities involving CATs are executed by the reactor task running on the local core of CAT. Having this configuration, the RED system10avoids unnecessary locking by sending messages to intended handlers of the CAT. Each CAT18provides a persistent storage target for replicated data, hosts Intent Log, and includes a BepTree.

The operation of compute nodes12in the RED cluster14depends on the system application. They may function as servers, supercomputing clusters, etc., and have a capacity to “write” by outputting the data to an external memory, as well as “read” data from an external memory, or other type storage media.

The subject RED system presented in simplified form inFIGS.2-3and4A-4B, focuses on balancing complete messages rather than packets or connections. Each pathway60,62is given credits and a weight which are used to control the balancing of messages over the possible pathways, and it operates in a weighted “round-robin” format, which adapts dynamically.

In the case of the core-affine applications, shown inFIG.3(i.e., for the HPC embodiments and RED servers), the pathways which deliver messages to the destined CPU core are given the highest priority. A core is a CPU (Central Processing Unit) or processor which can independently perform, or process, all computational tasks and may be considered as a smaller CPU or a smaller processor within a big processor. CPU affinity enables binding or unbinding a process, thread, or multiple processes, to a specific CPU core or to a subset of CPU cores. As an example only, but not to limit the scope of the present invention to a specific CPU core, the following description will be presented in a manner that the process(es) will run from the specific core, i.e., will execute on the designated CPU(s). At the time of resource allocation, each task is allocated to its related processor in preference to others. Processor affinity takes advantage of the fact that remnants of a process that is run on a given processor may remain in that processor's state (for example, data in the cache memory) after another process was run on that processor. Scheduling a CPU-intensive process that has few interrupts to execute on the same processor may improve performance by reducing degrading events such as cache misses, but may slow down ordinary programs since they would need to wait for that CPU to become available again.

In the present system, the message order created by an initiating node54is not important. System10adapts and self-heals in a highly efficient manner and eases the implementation to a “fragile RPC” approach to the application.

Typically, applications communicate via7network switch layers, including (from top to bottom) application, presentation, session, transport, network, data link and physical layers. The prior art Multi-Rail system operates in the transport layer. The present system however uses the upper layers (application, presentation, and session), to specify to the initiating node54that they want to send a message to a specific peer (node), i.e., the target node56, and the subject Multi-Path system10selects the pathway the message will be transmitted over. This allows the pathways to be changed dynamically without affecting the upper layers.

The subject system10has the ability to route messages between the CPU cores74,80embedded in the nodes70,76, respectively (as shown inFIG.3) which the application is bound to. The HPC (High Performance Computing) uses a technology called MPI (Message Passing Interface) which is a standardized and portable message-passing standard design to function on parallel computing architectures. The MPI standard defines a syntax and semantics of library routines that are useful to wide-range of users writing portable message-passing programs in C++, and Fourtran. There are several open source MPI implementations, which fostered the development of a parallel software industry, and encouraged development of portable and scalable large-scale parallel applications. The MPI technology binds “ranks” of the application to specific cores. MPI has an infrastructure for communicating these ranks with each other regardless of what physical nodes the ranks run on.

In the RED system10, the servers12bind the reactors48(which are incorporated in the Instance in the node12for the RED cluster14as described in previous paragraphs) to the CPU cores and communicate in a similar fashion as MPI on the RED server14, but not on the RED client24side. Reactors48form a foundational framework for the data storage distributed systems, which create concurrent and distributable applications more easily, by providing correct and robust programming extractions. Based on the reactor model for the distributed programming, reactors allow writing location-transparent programs that can be easily sub-divided into modular components. Reactors make it easier to reason about concurrent programs. Separate reactors communicate by exchanging events through channels. At the same time, the reactor model is location-transparent, which means that they can develop and task the program on a single machine, and then seamlessly deploy it on multiple machines that are connected with the computer network.

Basically, the subject Multi-Path solution52takes into account “core routing” in the algorithm underlying the operation of the Multi-Path sub-system52. Each CPU core74(as depicted inFIG.3) can create connections over the various pathways which are established between the two nodes (the initiating node54and the target node56). This means that each CPU core74will directly access the local interfaces72, and, once the connection71is created, the local interfaces72“anchor” to that core74and only that core can read/write the connection.

The Multi-Path approach is of specific benefit to the composite nodes when:(a) low latency is critical, which can be achieved in the present arrangement with the multiple paths running in parallel;(b) resiliency is crucial which needs multiple instances of hardware, such as network interfaces (if one interface fails, other interfaces will take over);(c) scalability is critical and is not a fixed element (i.e., the workload can change dynamically over time and needs to be adapted; and(d) messages can arrive in a different order than they were sent out.

The present system10includes a support logic layer which is designed to support the operation of the present Multi-Path sub-system52. The support logic layer may be embedded either in each node including the nodes at the client26side (referred to herein as client nodes) and the nodes12in the RED cluster14(referred to herein as storage server nodes), or it may be a layer connected between the nodes26and the nodes12through the high-speed network.

One of the constituents of the support logic layer is the control channel “handshake” routine100, presented inFIG.5. In order to initiate the operation of the Multi-Path system, an initiating node54(for example, the client node13,24,26), establishes a “control channel” to the target node56(for example, the compute node12in the RED cluster14). The routine of establishing the control channel between the initiating node54and the target node56, for example, it may be a DNS look-up to obtain an address for the target node56to which the initiating node54chooses to be connected. Subsequent to obtaining the address of the target node56, the initiating node54sends control requests to the target node56.

Over the established control channel between the initiating node54and the target node56, a request is sent by the client node (initiating node)54to the target (server) node56telling it what “workload” the client node intends to place upon the server node (target node)56along with resource information including how many RPC (Remote Procedure Call) messages in flight that the client node would like to send. The RPC message protocol consists of two structures: the call message and the reply message. The client node26(also referred to herein as the initiator node54) makes a remote procedure call to the network server node12(target node56) and receives a reply containing the results of the procedure's execution. By providing a unique specification for the remote procedure, the RPC can match a reply message to each call (or request) message.

In one of the exemplary embodiments, the routine100for establishing the control channel “handshake” between the client node24(which may be a local client13or a remote client26) and the server node12, as shown inFIGS.5and3, begins in Step A, where the initiating node (i.e., the client node24) obtains the address of the target node requested by upper software layers and creates a control channel connection to the target node (i.e., the server12). In a subsequent step B, the client node24sends a desired workload specifier plus RPS messages in flight over the control channel. In the following Step C, the server (target) node receives the request and determines how many and which CPU cores on the server node12the client node26is able to connect to. A list of the target CPU core identifications (ids) along with a list of the target interfaces addresses on the server node12(alternatively combined in a single list) are returned in Step D to the client node24as a response to the initiating node's request.

Subsequently, in Step E, the client node24uses the target CPU cores IDs and the server's interfaces addresses list(s) to create connections over the various pathways to the target CPU cores80in the target node76, as depicted inFIG.3, in a laminar fashion (i.e., only one connection existing for each selected target CPU core). This set of connections represents the Multi-Path connection the client node24has with the server (target node)12.

During the Steps B and C of the control channel “handshake” routine100depicted inFIG.5, the Multi-Path sub-system52executes a routine110for generation of the list of target CPU cores at the server node12. The flow chart shown inFIG.6(in conjunction withFIG.3) details the logic used by the server node12(target node) when generating the list of the target CPU cores that the client can connect to. As shown inFIG.6, subsequent to the establishing of the control channel and accomplishing the control channel handshake in step A ofFIG.5, the logic proceeds to Step B in which the server node12receives workload specifier and RPCs in flight values from the initiating client node24.

From Step B, the logic flows to Step C where the target server node12determines what target CPU cores the client node24can connect to.

As depicted inFIG.6, from Step C, the logic passes to Step F where the routine110inputs a desired workload and current flow to the server node12. The logic uses a specific algorithm designed for each workload type to select local target CPU cores. These algorithms are pluggable and are designed to meet the cluster's specific needs. The RED system10has its own set of pluggable algorithms for the target CPU cores selection. The objective of the routine in step F is to determine what target CPU cores are needed for accomplishing the workload type in question and to balance the client's workload with the availability of the CPU cores in the target server node12.

In Step G, the target server node may select numerous versions of arrangements for the target CPU cores for which the initiating client node24can be connected to. The selection types may specify an arrangement with only one CPU core, or this may be one CPU core per NUMA core, or all cores, etc. NUMA is a Non-Uniform Memory Access architecture which is a multi-processing architecture in which memory is separated into close and distant banks. In NUMA, the memory on the same processor board as the CPU (local memory) is accessed faster than the memory on outer processor boards (shared memory), hence the “non-uniform” nomenclature. NUMA architecture scales well to high numbers of CPUs.

From Step C, the procedure advances to logic block where the server node12determines in step H if there is an available capacity in the target server node12for the client's message(s). If there is the capacity available for accepting the message(s), the logic flows to Step J in which the server12generates a response to the client node24containing the list of selected target CPU cores in the target server node12and the target interfaces addresses. However, if the server node12becomes overloaded as the result of the operation in question, it can trigger the client node24to execute the control handshake routine again to get a new set of target CPU cores and adapt the connection pool of target CPU cores to match the server's new criteria. Specifically, in the case of the target node overload, the logic advances from Step H to Step I where the server generates rejection response and sends the reply to the client in Step K. This way, the Multi-Path sub-system52is dynamically adapted to changing load conditions. When the client node24(initiating node54) has a Multi-Path set of connections, it binds messages destined to the server node12(target node56) based on the flow chart diagram shown inFIG.7(in conjunction withFIG.3). Optimally, if there is a connection60,62established from the sending initiating CPU core74to the target CPU core80(as exemplified inFIG.3), that connection will always be used because it provides the lowest latency, and consequently provides the best performance of the system. Otherwise, a Weighted Fair Queue algorithm is applied to select which of the existing connections will be used to send the message(s). This means that the message may need to be transferred between the CPU cores on either the initiating node70or server-side node76. Such a transfer still provides improved performance, and scalability, when taken with respect to using a lock mechanism to allow multiple cores to send/receive over the same connection.

Referring toFIGS.7and3, the routine120of binding messages destined to the target server node76is initiated by receiving the request for binding the message to a connection. Responsive to the receiving of the request for binding to a connection, the logic120flows to step L “Get Local Sending Core and Target Receiving Core” in which an initiating CPU core74and the target CPU core80are obtained. Subsequently, from step L, the logic flows to step M where the logic decides whether the connection exists between the two CPU cores obtained in the previous step L, i.e., the local sending core and the target receiving core. If in step M the logical decision is that there is no connection between the local sending core and the target receiving core, the procedure advances to Step N for applying Weighted Fair Queue Algorithm to Connections. Subsequently, the logic passes to Step O to decide whether the valid connection is chosen. If the connection is not chosen, the logic advances to Step P where the request is rejected.

Returning to Step M, if the logic decides that there is a connection which exists between the sending CPU core and the target receiving CPU core, the procedure advances to Step Q where the request for binding to connection is met for the chosen connection. From Step Q, the logic flows to the ending procedure to “Send Request or Reject”.

Returning now to Step O, if the logic decides that the valid connection chosen, the procedure flows to Step Q and the request to bind to the chosen connection is satisfied. If however in Step O, the logic decision is that connection is not chosen, the logic flows to Step P to reject the request.

Returning again to Step N, in order to execute the procedure of applying Weighted Fair Queue algorithm to connections in question, the logic inputs in step R the parameters which the algorithm requires for computations, including the weight of the connection(s), identification whether the connection(s) route(s) directly to the target CPU core, a number of RPC messages currently in-flight, and the depth of the send queue.

As presented in previous paragraphs with regard toFIGS.2-7, the Multi-Path solution is designed with the purpose of creating multiple connections between application cores across multi-interfaces on both the initiator and target nodes and for load balancing the message and RDMA operations over those multiple connections. The Multi-Path approach adapts dynamically to applications which are core-affine (like, for example, in HPC) and those which are non-core affine. The balancing of messages/RDMA operations and handling of network errors are attained with a simple send/receive API. The application needs only to specify the target node for a message and, optionally, a core number in the case of a core-affine target.

The Multi-Path approach to data migration focuses on balancing complete (whole) messages rather than packets or connections. Each pathway for a message is given a credit and a weight so that these two parameters are used to control the balancing of messages over the possible pathways, and this approach becomes a form of weighted “round-robin” which adapts dynamically to the workload. In the case of core-affine applications, i.e., HPC procedures and RED servers, pathways which deliver messages to the destined core are always given the highest priority.

The subject Multi-Path approach adapts dynamically to network errors by excluding failing pathways, as needed, and shifting traffic to remaining healthy pathways, where background recovery of pathways runs periodically for self-healing.

The present approach provides a highly efficient and scalable distributed storage system where message order is not of importance as the case with the subject RED system, which is capable of adapting and self-healing in efficient manner, and which makes it much easier to implement a “fragile RPC” approach to the application specific to the RED system.

Although this invention has been described in connection with specific forms and embodiments thereof, it will be appreciated that various modifications other than those discussed above may be resorted to without departing from the spirit or scope of the invention as defined in the appended claims. For example, functionally equivalent elements may be substituted for those specifically shown and described, certain features may be used independently of other features, and in certain cases, particular locations of elements, steps, or processes may be reversed or interposed, all without departing from the spirit or scope of the invention as defined in the appended claims.