Repeatable barrier synchronization object

A method is disclosed comprising: detecting an event that is generated within a storage system; identifying a plurality of barrier objects that are associated with the event, each of the plurality of barrier objects being associated with a different respective set function, each of the plurality of barrier objects being associated with a different respective wait function, and each of the plurality of barrier objects being associated with a different respective release function; calling the respective set function of each of the barrier objects; calling the respective wait function of each of the barrier objects, wherein at least one of the respective wait functions is called before another one of the respective wait functions has returned; reconfiguring the storage system after all of the respective wait functions have returned; and calling the respective release function of each of the barrier objects.

BACKGROUND

A distributed storage system may include a plurality of storage devices (e.g., storage arrays) to provide data storage to a plurality of nodes. The plurality of storage devices and the plurality of nodes may be situated in the same physical location, or in one or more physically remote locations. The plurality of nodes may be coupled to the storage devices by a high-speed interconnect, such as a switch fabric.

SUMMARY

According to aspects of the disclosure, a method for use in a storage system is disclosed, comprising: detecting an event that is generated within the storage system; identifying a plurality of barrier objects that are associated with the event, each of the plurality of barrier objects being associated with a different respective set function, each of the plurality of barrier objects being associated with a different respective wait function, and each of the plurality of barrier objects being associated with a different respective release function; calling the respective set function of each of the barrier objects; calling the respective wait function of each of the barrier objects, wherein at least one of the respective wait functions is called before another one of the respective wait functions has returned; reconfiguring the storage system after all of the respective wait functions have returned; and calling the respective release function of each of the barrier objects.

According to aspects of the disclosure, a system is disclosed comprising: a memory; and one or more processors operatively coupled to the memory, the one or more processors being configured to perform the operations of: detecting an event that is generated within a storage system; identifying a plurality of barrier objects that are associated with the event, each of the plurality of barrier objects being associated with a different respective set function, each of the plurality of barrier objects being associated with a different respective wait function, and each of the plurality of barrier objects being associated with a different respective release function; calling the respective set function of each of the barrier objects; calling the respective wait function of each of the barrier objects, wherein at least one of the respective wait functions is called before another one of the respective wait functions has returned; reconfiguring the storage system after all of the respective wait functions have returned; and calling the respective release function of each of the barrier objects.

According to aspects of the disclosure, a non-transitory computer-readable medium is disclosed that is configured to store one or more processor-executable instructions, which when executed by one or more processors cause the one or more processors to perform the operations of: detecting an event that is generated within a storage system; identifying a plurality of barrier objects that are associated with the event, each of the plurality of barrier objects being associated with a different respective set function, each of the plurality of barrier objects being associated with a different respective wait function, and each of the plurality of harrier objects being associated with a different respective release function; calling the respective set function of each of the barrier objects; calling the respective wait function of each of the barrier objects, wherein at least one of the respective wait functions is called before another one of the respective wait functions has returned; reconfiguring the storage system after all of the respective wait functions have returned; and calling the respective release function of each of the barrier objects.

DETAILED DESCRIPTION

Before describing embodiments of the concepts, structures, and techniques sought to be protected herein, some terms are explained. In some embodiments, the term “I/O request” or simply “I/O” may be used to refer to an input or output request. In some embodiments, an I/O request may refer to a data read or write request.

FIG. 1is a diagram of an example of a storage system100, according to aspects of the disclosure. The storage system100may include a content-based storage system and/or any other suitable type of distributed storage system. As illustrated, the storage system100may include one or more storage arrays110and a management system190. The operation of the storage arrays110and the management system190is discussed further below with respect toFIGS. 2-8.FIG. 2is a diagram a storage array110that is part of the storage system100, according to aspects of the disclosure. The storage array110may include a plurality of storage servers210that are coupled to one another in a network (e.g., a mesh network). The network may include an InfiniBand network, a TCP/IP network, and/or any other suitable type of network.

FIG. 3is a diagram of an example of the management system190, according to aspects of the disclosure. As illustrated, the management system190may include one or more processors310, a memory320,110device(s)330, and communications interface(s)340that are operatively coupled to one another. The processor310may include any of one or more general-purpose processors (e.g., x86 processors, RISC processors, ARM-based processors, etc.), one or more Field Programmable Gate Arrays (FPGAs), one or more application specific circuits (ASICs), and/or any other suitable type of processing circuitry. The memory320may include any suitable type of volatile and/or non-volatile memory. In some implementations, the memory320may include one or more of a random-access memory (RAM), a dynamic random memory (DRAM), a flash memory, a hard drive (HD), a solid-state drive (SSD), a network accessible storage (NAS), and or any other suitable type of memory device. The I/O device(s)330may include any suitable type of input and/or output device, such as one or more mice, one or more keyboards, one or more microphones, or one more display screens, for example. The communications interface(s)340may include any suitable type of communications interface, such as one or more Ethernet adapters, one or more Wi-Fi adapters (e.g., 802.1414 adapters), and one or more Long-Term Evolution (LTE) adapters, for example.

FIG. 4is a diagram of an example of the storage server210A, according to aspects of the disclosure. As illustrated, the storage server210may include a processor410, a memory420, a communications interface(s)430, and a plurality of storage devices that are operatively coupled to one another. The processor410may include any of one or more general-purpose processors (e.g., x86 processors, RISC processors, ARM-based processors, etc.), one or more Field Programmable Gate Arrays (FPGAs), one or more application specific circuits (ASICs), and/or any other suitable type of processing circuitry. The memory420may include any suitable type of volatile and/or nonvolatile memory. In some implementations, the memory420may include one or more of a random-access memory (RAM), a dynamic random memory (DRAM), a flash memory, a hard drive (HD), a solid-state drive (SSD), a network accessible storage (NAS), and or any other suitable type of memory device. The communications interface(s)430may include any suitable type of communications interface, such as one or more Ethernet adapters, one or more Wi-Fi adapters (e.g., 802.1414 adapters), and one or more Long-Tem Evolution (LTE) adapters, for example. In the present example, the storage devices440are solid state drives (SSD). However, alternative implementations are possible, in which at least one of the storage devices is a spinning hard disk (HD), a flash driver, a Read-Only Memory (ROM), a Random-Access Memory (RAM), and/or any other suitable type of volatile and non-volatile memory.

Consider a distributed storage system with nodes that are connected in a full-mesh network, managed by a system manager.

FIG. 5is a schematic diagram illustrating the operation of the storage system100, in accordance with one implementation. As illustrated, the storage system may be configured to execute a plurality of type-1 threads510, a plurality of type-2 threads, a plurality of barrier objects530, a plurality of resources540, an event generator550, and a client560.

The threads510may include any suitable type of thread that is configured to access one or more of the resources540. The threads520may also include any suitable type of thread. However, in some implementations, the threads510and the threads520may be mutually exclusive. For example, thread520A may be executed by the client560only when none of the threads510A-C is accessing the resource540A; thread520B may be executed by the client560only when none of the threads510A-C is accessing the resource540B; and thread520C may be executed by the client560only when none of the threads510A-C is accessing the resource540C. The threads510and520may be executed on one of servers210of the storage system. Any of the threads510and520may be executed on the same server210(e.g., by the same processor) or on different servers210(e.g., by different processors)

The resources540may include any suitable hardware or software resource of the storage system100. Examples of software resources may include file system journals, databases and/or any other suitable type of data structure. Examples of hardware resources may include one or more SSDs, one or more communications interface(s), one or more communications bus(es), and/or any other suitable type of hardware resource. According to the present example, each of the resources540may include a different file system journal of the storage system100.

The event generator550may include one or more processor-executable instructions that are configured to generate high-availability (HA) events. The HA events may include any suitable type of event that is generated in the storage system100, which triggers the suspension and/or execution of one or more threads within the storage system100. Examples of HA events may include process-restart events that are generated after a process failure in the storage system100, events that are generated when a failed node is removed from the system100, events that are generated when a new node is added to the storage system100, and/or any other suitable type of event. According to the present example, the event generator550is executed on the management system190. However alternative implementations are possible in which the event generator550is executed on another node of the storage system100. Although in the present example the event generator550is implemented in software, it will be understood that alternative implementations are possible in which the event generator550is implemented in hardware or as a combination of software and hardware.

The client560may include one or more processor-executable instructions that are configured to detect HA events and process the detected HA events. More particularly, the client may interact with the barrier object530based on detected HA events to synchronize the execution of the threads510and520. In some implementations, the client560may be configured to execute processes800-900, which are discussed further below with respect toFIGS. 8-9. According to the present example, the client560is executed on the management system190. However alternative implementations are possible in which the client560is executed on another node of the storage system100. Although in the present example the client560is implemented in software, it will be understood that alternative implementations are possible in which the client is implemented in hardware or as a combination of software and hardware.

According to aspects of the disclosure, each of the threads510may include one or more critical sections. A critical section, according to the present example, may include one or more processor-executable instructions that are configured to access one of the shared resources540. At the beginning of each critical section, the ENTER function of one of the barrier objects530may be called, and at the end of the critical section, the EXIT function of the same barrier object530may be called. In other words, each critical section in the threads510may be delineated by calls to the ENTER and EXIT functions of the same barrier object. As used throughout the disclosure, the term “critical section” is used synonymously with the term “flow.”

FIG. 6shows an example of a thread600, according to aspects of the disclosure. The thread600may be the same or similar to any of the threads510A-C. As illustrated, the thread600may include a plurality of portions610,620, and630. Portion610includes a critical section612that contains one or more processor-executable instructions that are configured to access the resource540A. A call614to the ENTER function of barrier object530A is placed before the beginning of the critical section612, and a call616to the EXIT function of the barrier object530A is placed after the end of the critical section612. Portion620includes a critical section622that contains one or more processor-executable instructions that are configured to access the resource540B. A call624to the ENTER function of barrier object530B is placed before the beginning of the critical section622, and a call626to the EXIT function of the barrier object530B is placed after the end of the critical section622. Portion630includes a critical section632that contains one or more processor-executable instructions that are configured to access the resource540C. A call634to the ENTER function of barrier object530C is placed before the beginning of the critical section632, and a call636to the EXIT function of the barrier object530C is placed after the end of the critical section632.

Returning toFIG. 5, each of the barrier objects530may implement a respective software barrier. The software barrier may have at least two states—namely a “set” state and a “released” state. The barrier implemented by any of the barrier objects530may be set by calling a BARRIER_SET function of this object. The barrier associated with any of the barrier objects530may be released by calling a BARRIER_RELEASE function of that object. According to the present example, the BARRIER_SET and BARRIER_RELEASE functions are called by the client560in response to HA events that are generated by the event generator550. When a flow in any of the threads510calls the ENTER function of one of the barrier objects530, the flow may either enter the barrier associated with that barrier object530or be suspended. When the ENTER function is called while the barrier is released, the flow (or its thread) may begin executing, and it is considered to be “inside the barrier.” When the ENTER function is called while the barrier is set, the flow (or its thread) may be suspended until the barrier is released (e.g., by the client560and/or a scheduler of the storage system100), and is considered “outside of the barrier.” After the barrier is released, any flows (or their threads) that have been suspended upon calling the ENTER function can be resumed.

According to the present example, access to the resource540A is controlled by the barrier object530A; access to the resource540B is controlled by the barrier object530B; and access to the resource540C is controlled by the barrier object530C. When one or more HA events are generated by the event generator HA450, the client560may interact the barrier objects530to lock the resources540for the threads510. While the resources540are being locked, one or more of the threads520may wait to be executed. After the locking of the resources540is completed, the waiting threads520may be executed (e.g., by the client560and/or a scheduler of the storage system100).

In some implementations, the client560may lock the resource540A (for access by the threads510) by executing a two-step sequence involving the barrier object530A: first, the client560may call the BARRIER_SET function of the barrier object530A to prevent new flows from entering the respective barrier of the barrier object530A, and then the client may call a BARRIER_WAIT function of the barrier object530A. The BARRIER_WAIT function of the barrier object530A may return after all flows that are currently inside the barrier of the barrier object530A have exited the barrier (e.g., by calling the EXIT function of the barrier object530A). In other words, the BARRIER function may return after a certain time delay has passed since it was called. After the BARRIER_WAIT function has returned, the barrier object530A (or the resource540A) is considered locked for access by the threads510, and one or more of the threads520may be executed. After the threads520have finished executing, the client560may release the barrier of the barrier object530A by calling the BARRIER_RELEASE function of the barrier object530A.

The client560may lock the resource540B (for access by the threads510) by executing a two-step sequence involving the barrier object530B: first, the client560may call the BARRIER_SET function of the barrier object530E to prevent new flows from entering the respective barrier of the barrier object530B, and then the client may call a BARRIER_WAIT function of the barrier object530B, The BARRIER_WAIT function of the barrier object530B may return after all flows that are currently inside the barrier of the barrier object530B have exited the barrier (e.g., by calling the EXIT function of the barrier object530B). In other words, the BARRIER function may return after a certain time delay has passed since it was called. After the BARRIER_WAIT function has returned, the barrier object530B (and/or the resource540B) is considered locked for access by the threads510, and one or more of the threads520may be executed. After the threads520have finished executing, the client560may release the barrier of the barrier object530B by calling the BARRIER_RELEASE function of the barrier object530B.

The client560may lock the resource540C (for access by the threads510) by executing a two-step sequence involving the barrier object530C: first, the client560may call the BARRIER_SET function of the barrier object530C to prevent new flows from entering the respective barrier of the barrier object530C, and then the client may call a BARRIER_WAIT function of the barrier object530C. The BARRIER_WAIT function of the barrier object530C may return after all flows that are currently inside the barrier of the barrier object530C have exited the barrier (e.g., by calling the EXIT function of the barrier object530C). In other words, the BARRIER_WAIT function may return after a certain time delay has passed that is based on the time it takes for all flows that are currently inside the barrier of the barrier object530C to exit the barrier. After the BARRIER_WAIT function has returned, the object530C (or the resource540C) is considered locked for access by the threads510, and one or more of the threads520may be executed. After the threads520have finished executing, the client560may release the barrier of the barrier object530C by calling the BARRIER_RELEASE function of the barrier object530C.

FIG. 7is a diagram of an example of a barrier object700, according to aspects of the disclosure. The barrier object700may be the same or similar to any of the barrier objects530A-C. In some implementations, the barrier object700may include member variables702-710and member functions722-730.

Variable702(NUM_INSIDERS) may be a counter variable that identifies the total count of flows that are currently inside the barrier implemented by the barrier object700. When member function722(ENTER) is called by a thread510, while the barrier is released, variable702may be incremented. When member function722(ENTER) is called by a thread510, while the barrier is set, variable702may be left unchanged, and the thread510may be suspended. When member function724(EXIT) is called by a thread510, variable702may be decremented.

Variable704(BARRIER_IS_SET) may be a flag variable that indicates whether the barrier implemented by the barrier object700is set. When the barrier is set, variable704may have a first value, and when the barrier is released, variable704may have a second value.

Variable706(NUM_SUSPENDERS) may be a counter variable that identifies a total count of unprocessed HA events generated within the system. When member function722(BARRIER_SET) is called by a thread510, variable706may be incremented. When member function730(BARRIER_RELEASE) is called, variable706may be decremented. In some implementations, the barrier implemented by the barrier object700may be released only when the value of variable706becomes zero after variable706is decremented.

Variable708(WORKERS_WAIT_QUEUE) may be a waiting queue where identifiers are stored for threads510that attempt to enter the barrier implemented by the barrier object700while the barrier is set. In other words, variable708(WORKERS_WAIT_QUEUE) may identify one or more threads510that are waiting for the barrier to be released. When a thread510calls the member function722(ENTER), while the barrier is set, the thread510may be suspended, and an identifier corresponding to the thread510may be added to the waiting queue.

Variable710(SUSPENDER_WAIT_QUEUE) may be a queue where identifiers are stored for threads520that are waiting for all flows inside the barrier implemented by the barrier object700to exit before the barrier is set. In other words, variable710(SUSPENDER_WAIT_QUEUE) may identify one or more threads520that are waiting for all flows (in the threads510) to exit the barrier. After the flows have exited, the barrier may be transitioned from the “released” state to the “set” state, and the threads520that are waiting in the queue may be executed.

Member function722(ENTER) may be invoked before a particular flow is executed. Executing the member function722, while the barrier implemented by the barrier object700is set, may cause the thread calling the member function722to be suspended (e.g., by the client560and/or a scheduler of the storage system100), and an identifier corresponding to the thread to be added to variable708(WORKER_WAIT_QUEUE).

Member function724(EXIT) may be invoked after a particular flow has finished executing. The member function724may be configured to decrement variable702(NUM_INSIDERS) when invoked.

Member function726(BARRIER_SET) may prevent threads that subsequently call member function722(ENTER) from entering the barrier implemented by the barrier object700. The member function724may be configured to transition the barrier object700from the “released” state to the “set” state by changing the value of variable704(BARRIER_IS_SET) to indicate that the barrier implemented by the barrier object700has been set. The member function724may be further configured to increment the value of variable706(NUM_SUSPENDERS) when called.

Member function728(BARRIER_WAIT) may be called in order to wait for all threads (or flows thereof) that are inside the barrier implemented by the barrier object700to exit. After member function728returns, it is guaranteed that there are no more threads remaining inside the barrier. Member function728may be configured to return only when there are no more threads (or flows thereof) inside the barrier.

Member function728(BARRIER_RELEASE) may be configured to decrement the value of variable706(NUM_SUSPENDERS) by 1. Furthermore, after decrementing the value of variable706, member function728may determine whether the value of variable706is equal to zero. If the value of variable706is equal to zero, member function728may: (i) transition the barrier object700from the “set” state to the “released” state by changing the value of variable704(BARRIER_IS_SET) to indicate that the barrier has been released, and (ii) release any threads510that are waiting in variable708(WORKER_WAIT_QUEUE). Releasing any of the threads510from variable708may include removing an identifier corresponding to the thread510from the variable708, and causing the client560(or a scheduler of the storage system100) to execute the thread510.

FIG. 8is a flowchart of an example of a process800for reconfiguring the storage system100, according to aspects of the disclosure. At step802, the client560detects that an HA is generated by the event generator550. In some implementations, the HA event may be generated when a node, on which a portion of a journal is stored, fails. At step804, the client560calls the BARRIER_SET function of the barrier object530A. At step806, the client560calls the BARRIER_SET function of the barrier object530B. At step808, the client560calls the BARRIER_SET function of the bather object530C. At step810, the client560calls the BARRIER_WAIT function of the barrier object530A. At step812, the client560calls the BARRIER_WAIT function of the barrier object530B. At step814, the client560calls the BARRIER_WAIT function of the barrier object530C. In some implementations, the BARRIER_WAIT function of the barrier object530C may be called before any of the calls to the BARRIER_WAIT functions of the objects530A-B have returned. At step816, the client560detects whether all of the invoked BARRIER_WAIT functions have returned. If all BARRIER_WAIT calls have returned, the process800proceeds to step818. Otherwise, if not all BARRIER_WAIT calls have returned, step816is executed again. At step818, the storage system is reconfigured. According to the present example, reconfiguring the storage system includes re-distributing the journal that is (partially) stored on the failed node among nodes in the storage system100that remain active. In some implementations, the journal may be re-distributed by executing one or more of the threads520that are identified in the SUSPENDER_WAIT_QUEUE of any of the barrier objects530A-C. At step820, the client560calls the BARRIER_RELEASE function of the bather object530C. At step822, the client560calls the BARRIER_RELEASE function of the barrier object530B. At step824, the client560calls the BARRIER_RELEASE function of the barrier object530A.

One advantage of using the barrier objects530for access control is that the barrier objects530can be set in parallel. As noted above, the barrier objects530employ a two-step mechanism for blocking access to the resources540, which involves: (i) calling the BARRIER_SET function of each of the barrier objects530to prevent new flows from entering the barrier implemented by the object, and (ii) calling the respective BARRIER_WAIT function of each of the bather objects530to wait out any flows that are currently inside the barrier implemented by the object. As noted above, after the BARRIER_WAIT function of any of the barrier objects530is called some time may pass until all flows inside the barrier of the barrier

have exited. Because the BARRIER_WAIT functions of the barrier objects530A-C are executed in parallel, the total waiting time associated with the barrier objects is equal to waiting time of the BARRIER_WAIT function that takes the longest to complete.

In some respects, the barrier objects530may provide an alternative to using conventional R/W locks to synchronize access to the resources540A-B. One advantage of the of the barrier objects530A-C over conventional R/W locks is that they can be manipulated in parallel by the client560, whereas conventional R/W locks cannot. A conventional R/W locks may provide a single LOCK function, which may prevent new flows from accessing a resource associated with the lock, and which may return only when all flows that are currently accessing the resource have finished. In other words, because each LOCK function returns only after all accesses to the LOCK function's respective resource have completed, the time it takes for each LOCK function to return may be roughly equal to the time it takes any of the BARRIER_WAIT functions to return. Thus, if access to each of the resources540were controlled via a separate R/W lock, the client560would have to make three consecutive calls to the respective LOCK functions of the RW locks. Each subsequent call can be placed only after the previous call has completed. Thus, the total time it would take for the three calls to be completed would be equal to the sum of the execution times of the three LOCK functions, which, in most circumstances, would be longer than the waiting time associated with the BARRIER_WAIT function that takes the longest to complete.

FIG. 9is a flowchart of an example of a process900for reconfiguring a storage system, according to aspects of the disclosure. In some implementations, the storage system may be the same or similar to the storage system100. In some implementations, the process900may be performed by a management system that is part of the storage system and/or any other suitable node of the storage system.

At step902, the management system detects that an event has been generated within the storage system. The event may be an HA event and/or any other suitable type of event. At step904, the management system identifies one or more resources of the storage system that are associated with the event. The one or more resources may be identified by using a table that maps each of a plurality of event identifiers to respective identifiers of resources that correspond to the event identifier. The one or more resources associated with the event detected at step902may be identified by performing a search of the table using an identifier of the event as a search key.

At step906, the management system identifies one or more barrier objects. Each of the barrier objects may correspond to one of the resources identified at step904. Each of the barrier objects may be identified by using a table that maps each of a plurality of resource identifiers to one or more corresponding barrier object identifiers that correspond to the resource identifier. Each of the harrier objects may be identified by performing a search of the table using an identifier of one of the resources identified at step904as a search key.

At step908, the respective BARRIER_SET function of each of the identified barrier objects is called. At step910, the respective BARRIER_WAIT auction of each of the identified barrier objects is called. At step912, a determination is made if all calls to BARRIER_WAIT functions have returned. If all calls to the BARRIER_WAIT functions have returned, the process900proceeds to step914. Otherwise, if not all calls to the BARRIER_WAIT functions have returned, step912is repeated.

At step914, the storage system is reconfigured by the management system. Reconfiguring the storage system may include any action that causes the state of the storage system to change. For example, reconfiguring the storage system may include copying data from one node in the storage system to another. As another example, reconfiguring the storage system may include adding or removing a node from the storage system. As yet another example, reconfiguring the storage system may include changing one or more configuration settings of the storage system. It will be understood that the present disclosure is not limited to any specific action for reconfiguring the storage system. In some implementations, the reconfiguration of the storage system may be performed by executing one or more threads in the storage system that are identified in the SUSPENDER_WAIT_QUEUE of any of the barrier objects identified at step906. At step916, after the storage system is reconfigured, the management system calls the respective BARRIER_RELEASE function of each of the barrier objects identified at step906.

According to the present example, the process900may be configured to redistribute one or more file system journals within the storage system. For example, a first barrier object may be associated with a first journal (hereinafter “journal A”), a second barrier object may be associated with a second journal (hereinafter “journal B”), and third barrier object may be associated with a third journal (hereinafter “journal C). Each of journals A-C may be distributed over a plurality of nodes of the storage system. When one of the nodes fails, an HA event may be generated, which triggers reconfiguration of the storage system. The reconfiguration of the storage system includes transitioning each of the barrier objects to the “set” state and waiting for all worker threads that are currently accessing the journals to finish. After the working threads are finished, and the barrier objects are set, the responsibility for one or more of journals A-C may be redistributed among the nodes in the storage system100that remain active.

FIGS. 1-9are provided as an example only. At least some of the steps discussed with respect toFIGS. 1-9may be performed in parallel, in a different order, or altogether omitted. As used in the context of “set function,” “wait function,” and “release function,” the term “function” may refer to one or more processor-executable instructions are configured to perform an operation. Although in one example, the “set function,” “wait function,” and “release function” may be implemented in software, it will be understood that alternative implementations are possible in which any of these functions may be performed in hardware or as a combination of software and hardware.

To the extent directional terms are used in the specification and claims (e.g., upper, lower, parallel, perpendicular, etc.), these terms are merely intended to assist in describing and claiming the invention and are not intended to limit the claims in any way. Such terms, do not require exactness (e.g., exact perpendicularity or exact parallelism, etc.), but instead it is intended that normal tolerances and ranges apply. Similarly, unless explicitly stated otherwise, each numerical value and range should be interpreted as being approximate as if the word “about”, “substantially” or “approximately” preceded the value of the value or range.

While the exemplary embodiments have been described with respect to processes of circuits, including possible implementation as a single integrated circuit, a multi-chip module, a single card, or a multi-card circuit pack, the described embodiments are not so limited. As would be apparent to one skilled in the art, various functions of circuit elements may also be implemented as processing blocks in a software program. Such software may be employed in, for example, a digital signal processor, micro-controller, or general-purpose computer.

Some embodiments might be implemented in the form of methods and apparatuses for practicing those methods. Described embodiments might also be implemented in the form of program code embodied in tangible media, such as magnetic recording media, optical recording media, solid state memory, floppy diskettes, CD-ROMs, hard drives, or any other machine-readable storage medium, wherein, when the program code is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the claimed invention. Described embodiments might also be implemented in the form of program code, for example, whether stored in a storage medium, loaded into and/or executed by a machine, or transmitted over some transmission medium or carrier, such as over electrical wiring or cabling, through fiber optics, or via electromagnetic radiation, wherein, when the program code is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the claimed invention. When implemented on a general-purpose processor, the program code segments combine with the processor to provide a unique device that operates analogously to specific logic circuits. Described embodiments might also be implemented in the form of a bitstream or other sequence of signal values electrically or optically transmitted through a medium, stored magnetic-field variations in a magnetic recording medium, etc., generated using a method and/or an apparatus of the claimed invention.

It will be further understood that various changes in the details, materials, and arrangements of the parts which have been described and illustrated in order to explain the nature of the claimed invention might be made by those skilled in the art without departing from the scope of the following claims.