Distributed flash memory storage manager systems

A flash memory storage system may include several modules of flash memory storage manager circuitry, each having some associated flash memory. The modules may be interconnected via the flash memory storage manager circuitry of the modules. The system may be able to write data to and/or read data from the flash memory associated with various ones of the modules by routing the data through the flash memory storage circuitry of the modules. The system may also be able to relocate data for various reasons using such read and write operations. The flash memory storage circuitry of the modules keeps track of where data actually is in the flash memory.

BACKGROUND

Larger data storage has been in increased demand in recent years. Data storage based on solid state flash memory offers compelling advantages in terms of read/write throughput, stability, shock and vibration resistance, etc., compared with traditional magnetic disk based storage. Some such solid state flash memory storage may need to be larger than others, and it can therefore be desirable to be able to use various numbers of identical or substantially identical modules to construct such flash memory storage systems in any of a wide range of sizes. It is also important for such flash storage and the associated memory access circuitry to be able to automatically keep track of where all data is in the memory so that the data can be efficiently and reliably accessed. The present disclosure facilities such aspects of electronic data memory construction and/or operation.

SUMMARY

In accordance with certain possible aspects of the disclosure, a plurality of memory circuits may each be connected to a respective one of a plurality of integrated circuits (“ICs”). Each of the ICs may be connected to at least one other of the ICs by inter-IC connections so that an IC exchanges memory circuit data with another IC via the inter-IC connections. Each of the ICs may include memory manager circuitry that comprises a logic block manager for maintaining a unique global identification (“ID”) for each block of data contained in any portion of any of the memory circuits, the global ID including a node ID identifying the IC that is connected to the memory circuit containing that block and a logical block number for that block. The memory manager circuitry for each IC may further comprise a translator for maintaining a mapping between (1) the logical block number of each block contained in the memory circuit connected to the IC that includes that translator, and (2) a physical portion ID of a portion of that memory circuit that contains that block. The memory manager for each IC may still further comprise a driver for receiving the physical portion ID from the translator of the IC that includes that driver and accessing the portion of the memory connected to that IC that is identified by that physical portion ID.

In accordance with certain other aspects of the disclosure, in memory circuits as summarized above, each of the ICs (“the source IC”) may include circuitry for transferring a block (“the transferred block”) accessed by the driver of the source IC to another of the ICs (“the destination IC”) for storage in the memory circuitry connected to the destination IC.

In such memory circuits the circuitry for transferring may employ the inter-IC connections.

In accordance with certain still other possible aspects of the disclosure, in memory circuits as summarized above, each of the ICs (“the source IC”) may further include circuitry for maintaining a count of how many times each of the other ICs requests a respective block contained in the memory circuit that is connected to the source IC, and circuitry for the transferring a block (“the transferred block”) (for which the count for one of the other ICs (“the destination IC”) exceeds a threshold value) from the memory circuit connected to the source IC to the memory circuit connected to the destination IC.

Still other possible aspects of the disclosure relate to managing access to a plurality of memory circuits, each of which is connected to a respective one of a plurality of integrated circuits (“ICs”). One of the ICs may be connected to at least one of the other ICs by inter-IC connections so that one IC exchanges blocks of memory circuit data with another IC via the inter-IC connections, each of the ICs (“the source IC”) including a memory manager. Each such memory manager may comprise circuitry for maintaining a count of how many times a given IC requests at least one block contained in the memory circuit that is connected to the source IC, and circuitry for transferring a block (“the transferred block”) (for which the count for one of the other ICs (“the destination IC”) exceeds a threshold value) from the memory circuit connected to the source IC to the memory circuit connected to the destination IC.

In such memory managers the circuitry for transferring may employ the inter-IC connections.

Further features of the disclosure, its nature and various advantages will be more apparent from the accompanying drawings and the following detailed description.

DETAILED DESCRIPTION

Illustrative embodiments of electronic data memory systems in which the present disclosure can be implemented and practiced are shown in Zhou et al. U.S. patent application Ser. No. 12/728,757, filed Mar. 22, 2010 (“the Zhou et al. reference”), which is hereby incorporated by reference herein in its entirety.FIGS. 1-11herein are repeated from the Zhou et al. reference and are briefly described below. More detailed description is contained in the text of the Zhou et al. reference.

FIG. 1shows an illustrative embodiment of an integrated circuit (“IC”)10that can be used as part of a distributed electronic data storage or memory system in accordance with this disclosure. (As used herein, terms like “circuit,” “circuitry,” “integrated circuit,” “IC,” and the like may refer to circuitry with or without software that runs on the circuitry and/or that controls various operations of the circuitry. As just one illustration of this, an “IC” (as that term is used herein) may include one or more processors with or without software that runs in and/or controls various operations of the processor(s).) The circuitry of IC10includes flash memory controller20, cache memory controller30, interface controller40, direct memory access (“DMA”) circuitry50, central processing unit (“CPU”) circuitry60, bus circuitry70, and controllable routing circuitry80. IC10is connected to flash memory channels120and cache memory130. Flash memory channels120are typically the relatively large, main memory for IC10. Cache memory130is typically a smaller, temporary memory for IC10. For example, cache memory130may be used for relatively short-term storage of data on its way to or from flash memory120.

Interface controller40can be used for connection of IC10to other circuitry (not shown) that may be thought of as external to the memory system of which elements10,120, and130are a part. For example, the memory system may store data supplied by that external circuitry. Similarly, the memory system may supply its stored data to the external circuitry. Connections140(to the external circuitry) may supply to IC10data write and/or data read instructions (requests or commands), as well as acting as the conduit for memory data exchange between IC10and the external circuitry.

Controller20controls writing data to and reading data from flash memory120. Controller30functions similarly for cache memory130. CPUs60provide overall control for IC10. DMA elements50support at least many aspects of memory writing and reading, with less or no involvement of CPUs60in such activities. Bus circuitry70provides connections between other circuit elements on IC10. Routing circuitry80provides controllable connections (1) between bus circuitry70and similar routing circuitry80in one or more other instances of IC10, and (2) between such other instances of IC10. In a memory system that includes such multiple ICs10, each IC is preferably constructed as shown inFIG. 1, and each IC is connected to its own flash memory120and its own buffer memory130. Accordingly, such a memory system may be referred to as a distributed memory system, and in general such a memory system may include any number of ICs10, etc., to provide memories having any of a wide range of sizes.

IC10is just one example of how this type of system component can be constructed in accordance with this disclosure. For example, in other embodiments of the disclosure, such an IC may omit some of the elements shown for IC10inFIG. 1, and/or such an IC may have other elements that are not shown for IC10inFIG. 1.

Routing circuitry80may be thought of as a crossbar switch (or at least being like a crossbar switch). In general, such routing circuitry80can connect any of circuitry80's ports (labeled P1-P9) to any other of circuitry80's ports (although there may be some inter-port connections that cannot be made). Inter-IC connections210are used to connect the “external ports” P4-P9of depicted IC10to similar ports of one or more other IC10instances in the distributed memory system.

FIGS. 2-4show some examples of distributed memory system topologies that can be used (although many other topologies are also usable). Each small circle10in each ofFIGS. 2-4represents one instance of an IC10. Each line210in each ofFIGS. 2-4represents an inter-IC connection. TheFIG. 2topology may be referred to as a two-dimensional (“2D”) mesh; theFIG. 3topology may be referred to as a three-dimensional (“3D”) mesh; theFIG. 4topology may be referred to as a triangle cube.

FIG. 5shows some additional details as to how routing circuitry80may be constructed. In this construction, each external port P4-P9is connected to serializer-deserializer (“SERDES”) circuitry512in physical layer circuitry510of routing circuitry80. Each SERDES circuitry512can convert signals between (1) serial form on inter-IC connections210and (2) parallel form for use inside circuitry80and elsewhere on IC10. (Internal ports P1-P3may be parallel ports, which do not require SERDES circuitry.) SYNC, align, and ACK/NAK circuitry522(in link layer circuitry520of routing circuitry80) performs synchronization (“SYNC”), alignment (“ALIGN”), and packet acknowledgement (“ACK/NAK”) functions for the signals coming from and/or going to each external port P4-P9. Packet routing circuitry532(in packet layer circuitry530of routing circuitry80) performs the actual routing of data packets between selectable different ones of ports P1-P9.

The organization of a typical data packet is shown inFIG. 6. For example, such a packet may include a header, which in turn includes an IC10identification (“ID”) and memory (“MEM”) address for the associated actual “data payload”. The data payload follows the header, and is in turn followed by cyclic redundancy check (“CRC”) or similar information for helping to make sure that the data payload has been correctly received. The IC10ID may also sometimes be referred to as the node ID.

FIG. 7shows that some translation may be needed between the information in the header of a packet and the action that packet routing circuitry532needs to take in order to get a packet from one IC10to another IC10. For example, in a system like that shown inFIG. 2, it may be necessary to send a packet from the upper right “node” (IC10) to the lower left “node” (IC10). This may be due to the data in the packet being stored in the flash memory120connected to upper right IC10, but being needed to satisfy a request for data received by interface40of the lower left IC10. The packet being discussed can be routed from the “source” or “owner” (upper right) IC10to the “destination” (lower left) IC10via the upper-most arcuate inter-IC connection210to the upper left IC10, and then via the left-most arcuate inter-IC connection210to the lower left IC10. The header for the packet may include the intended destination (IC10ID), but the packet routing circuitry532in the upper right IC10may need to translate that information into initial information to the effect that a way to get the packet to the lower left destination IC10is via the routing circuitry80of the upper left IC10. The routing table circuit740(FIG. 7) of the routing circuitry80of the upper right source IC10may therefore be programmed based on the topology of the system (e.g.,FIG. 2) to tell the associated packet routing circuitry532that when that circuitry532gets a packet that is to be routed to the lower left IC10, circuitry532should in fact route that packet to the upper left IC10. (The upper left IC will forward that packet on to the lower left destination IC.)

FIG. 8shows that packet routing circuitry532may include input and/or output buffer circuitry850if needed. Buffer circuitry850may be input buffer circuitry and/or output buffer circuitry for buffering data in each of the port channels of packet routing circuitry532.

FIG. 9shows an example of a possible physical layout of a distributed flash memory system (or a representative portion of such a system) in accordance with the disclosure. Element900is a printed circuit board (“PCB”). Six ICs10a-fare mounted on PCB900. Also mounted on PCB900are the flash memories120a-fand cache memories130a-fthat are connected to each of ICs10(e.g., via circuit traces on PCB900). Inter-IC connections210and external connections140(e.g., as inFIG. 1) may also be provided (at least in part) as traces on PCB900. Multiple instances of PCB900may be connected to one another via a backplane on which the PCBs are mounted.

FIG. 10shows an illustrative (“crossbar”) construction of routing circuitry80. Any two ports P1-P9can be connected to one another via crossbar conductor(s) CB. The switches S between CB and each of the two ports that it is desired to interconnect are closed (by assertion of appropriate control signals C). All other switches S are open.

FIG. 11shows that one crossbar network implementation of routing circuitry80can concurrently and independently make two or more port-to-port connections. Each such port-to-port connection is made using a respective one of CBa, CBb, etc., and the associated switches Sa, Sb, etc.

The present disclosure provides circuitry and methods (or systems) for providing storage management in a distributed flash storage environment like that illustrated byFIGS. 1-11. A storage manager of this disclosure provides memory data block service across distributed storage nodes (e.g., like ICs10and their associated flash120and cache130memory circuits) to still higher-level structure like a file system or database management system. (A “block” may be any convenient amount of data. For example, a block may be the amount of data that fits in the smallest amount of flash memory120that can be separately addressed for data writing or reading. A block will typically be a plurality of data words, but each flash memory120can typically hold many such blocks.)

In accordance with certain possible features of the disclosure, the storage manager may map logical data blocks to physical data blocks of the flash memories120. In accordance with certain other possible features of the disclosure, the storage manager may provide dynamic data block migration across different storage nodes10/120/130to improve data access efficiency. The distributed storage manager is preferably circuitry in and/or software running on each of the ICs10in the distributed storage system. The storage manager system elements in each IC10are preferably tightly coupled to the storage manager system elements in all of the other ICs10in the distributed system. This tight coupling can be via the routing circuitry80of the ICs and the inter-IC connections210between the ICs.

An illustrative embodiment of a distributed flash storage manager1200is shown inFIG. 12. As shown in this FIG., storage manager1200has a layered structure. Flash device driver1230is the lowest layer in this structure. The next higher layer is flash translation layer1220. The upper layer is logic block manager layer1210. Each of these layers may be circuitry (or may include circuitry) on each instance of IC10in the distributed flash storage system. Alternatively, each of these layers may be or may include corresponding firmware and/or software running in circuitry on each instance of IC10in the distributed system. (Again, as noted earlier in this specification, terms like “circuitry” as used herein are generic to circuitry alone and to circuitry with suitable firmware and/or software.)

Considering first flash device driver layer1230, this layer performs hardware-related functions for storage manager1200. For example, layer1230may provide the actual physical device identification (“ID”) for the one of several flash devices120(connected to the IC10including this particular instance of storage manager1200) that is to be accessed in a particular memory transaction (data write or data read). Layer1230may additionally identify the read/write sector in that flash device120that is to be accessed. Layer1230may still further provide the DMA50(FIG. 1) data transfer (e.g., from flash to cache memory or vice versa).

From the foregoing, it will be seen that the outputs of layer1230are specific to particular physical locations in the immediately associated memory elements120/130that are to be used in the particular memory transaction being carried out. Layer1230gets at least the basics of this physical location information from the associated flash translation layer1220. Note, however, that upper layers1210and1220preferably give to the associated layer1230only information for blocks that are in the memory elements120/130that are connected to the IC10that includes this instance of elements1200. Thus one of the functions of upper layers1210and1220is to effectively filter out (and not pass on to the associated layer1230) information for any logical blocks that are not physically “owned by” the elements120/130connected to the IC10including this element1200instance. (“Owned by” means that the block is actually stored in the elements120/130that “own” that block.)

Flash translation layer1220typically provides mapping between each “logical” block of memory data and the physical portion (also sometimes referred to as a block) of the memory resources120/130that actually contains (“owns”) that block of data. A physical block may be identified by a node (IC10) identification (“ID”), a flash120channel number, a flash120device number, a flash120block number, and a flash120sector number. Each logical block may be identified by a node (IC10) ID and a logical block number. Flash translation layer1220may therefore maintain a mapping table whereby each immediately above-mentioned logical block number can be converted to the appropriately corresponding flash channel number, flash device number, flash block number, and flash sector number (all forming parts of a physical portion ID). Again, if (and only if) these last-mentioned physical location numbers are for a block owned by the memory120connected to the IC10having the associated node ID, then layer1220passes these physical location numbers on to the associated layer1230for use in accessing the identified physical portion of the associated memory120.

Each layer1220may also perform related services like block allocation (e.g., when new data is initially written into memory120), garbage collection (e.g., when a portion of memory120no longer contains data that may be needed), and wear leveling (e.g., to avoid excessive over-use of some portions of memory120, while other portions are not being accessed as frequently).

Logic block manager1210provides storage block service to the entire system (i.e., all of the nodes10/120/130in an entire system). Each block has a unique global identification (“ID”), which includes a node (IC10) ID and a logical block number. Any node can request to access any block anywhere in the entire system using the global ID for that block. Based on the node ID portion of the global ID, the request is routed to the correct IC10(the “owner” of the requested block). This routing can be performed via the routing circuitry80and inter-IC connections210needed to get the request from the requesting node to the owner node. When the request reaches the owner node (IC10), the logic block manager1210applies the logical block number part of the request to the flash translation layer1220of that IC10. That layer1220then processes the logical block number information as described earlier in this specification, leading ultimately to accessing the requested block in the flash memory120that is connected to the owner node IC10.

FIGS. 13aand13b(sometimes referred to collectively asFIG. 13) show an illustrative embodiment of how a read command or request may be handled in distributed flash memory systems in accordance with this disclosure. At1310, any node (IC10) may initiate a read command. The node initiating such a read command may be referred to as the “requester node” or “the requester.” The read command may include the global ID of the requested data block.

At1320the read command is routed to the node (IC10) that is the “owner” of the requested data block. This routing can take place through the interconnect networks80/210of the system. As noted earlier, the global ID of each data block includes the node ID of that block. The node ID identifies the node that is the owner of the block, which enables interconnect networks80/210to route the read command to the proper node in the system.

At1330the owner node checks the status of the data block identified in the read command. Two outcomes of such a check are possible. First, it may be found that the data block is “free” (meaning, e.g., that no node is currently writing to that block). Alternatively, it may be found that the data block is “locked” (meaning, e.g., that some node is currently writing to that block). If the node is free, control passes from1330to1340. We will first continue with this branch from1330. Later we will come back to the other branch from1330.

At1340the circuitry of the owner node reads the requested data out of the block identified in the read command. This will typically require processing the logical block number portion of the global ID of the requested block through the storage manager1200(FIG. 12) of the owner node as described earlier in this specification. Also at1340the data block thus read out is routed to the requester via the interconnect networks80/210of the system. The read data may thus get back to the requester via the same route established for the read request, but in the opposite direction. This satisfies the read request, and so the read protocol can end at1350.

Returning now to the other branch from1330, if the data block is locked, control passes from1330to1360. At1360, the owner node sends a data block non-available status packet back to the requester via the interconnect networks80/210. At1370the requester receives this non-available status packet. At1380the requester can try again to satisfy its read request by restarting the protocol at1310.

FIGS. 14a-c(sometimes referred to collectively asFIG. 14) show an illustrative embodiment of how a write command or request may be handled in distributed flash memory systems in accordance with the disclosure. At1410, any node (IC10) can initiate a write command. The node initiating such a write command may be referred to as the “requester node” or “the requester.” The write command may include the global ID of the data block into which it is desired to write data. Any node (IC10) in the system may be the “owner” of this data block, where “owner” has the same meaning as used elsewhere in this specification.

At1420the write command is routed to the owner node. This routing can take place through the interconnect networks80/210of the system. As noted earlier, the global ID of each data block includes the node ID of that block. The node ID identifies the node that is the owner of the block, which enables interconnect networks80/210to route the write command to the proper node in the system.

At1430the owner node checks the status of the data block identified in the write command. If the data block is free (as explained earlier), control passes from1430to1440. If the data block is locked (as also explained earlier), control passes from1430to1460.

At1440the circuitry of the owner node sends a write acknowledge packet back to the requester via interconnect networks80/210. At1452the requester receives the write acknowledge packet. At1454the requester sends the write data packet (i.e., the actual data to be written) to the owner via interconnect networks80/210. At1456the owner writes the write data packet to the data block. At1458the write protocol ends.

Returning to the other branch from1430, at1460the owner sends a data block non-available status packet to the requester via interconnect networks80/210. At1470the requester receives the non-available status packet. At1480the requester can retry the write command by starting again at1410.

FIGS. 15a-c(sometimes collectively referred to asFIG. 15) show an illustrative embodiment of circuitry (or equivalent structure) in accordance with another possible aspect of this disclosure. This is structure for providing dynamic data block migration (e.g., within a distributed flash memory system such as is described elsewhere in this specification). Each IC10in such a system may include an instance of theFIG. 15structure. This structure may be dedicated circuitry on the IC, firmware on the IC, software running on more general-purpose circuitry on the IC, or any combination of the foregoing. To simplify the following discussion, it will be assumed thatFIG. 15shows circuitry (which “circuitry” terminology is again consistent with the generic use of that term herein to refer to circuitry alone or to circuitry with or running software).

TheFIG. 15circuitry includes one counter1510for each data block (e.g., in flash memory120) connected to the node (IC10) that includes those counters1510. Each counter1510counts the number of times that this node accesses the associated data block owned by this node.

TheFIG. 15circuitry also includes M more counters1512for each of the N data blocks owned by this node. M is the number of other nodes (ICs10) in the system. For each data block, each of that data block's M counters1512is associated with a respective one of the M other nodes in the system. Each counter1512counts the number of times that the associated other node accesses the associated data block.

There is one comparator1514associated with each of the counters1512. (It will be understood that the number of comparators1514can be reduced by time-sharing the reduced number of comparators. For example, a single comparator1514can be time-shared by all of counters1512. To simplify the discussion, however, it will be assumed that there is a separate comparator1514for each counter1512.) Each comparator1514compares (1) the output1513of a respective one of counters1512, and (2) the output1511of the counter1510for the same data block that the output1513relates to. If (and only if) output1513is greater than output1511, then the comparator1514applies an enabling signal to a respective one of comparator circuits1518. (Output1511is the count currently registered by the associated counter1510. Output1513is the count currently registered by the associated counter1512.)

There is one comparator1518for each comparator1514. (Again, the number of comparators1518can be reduced by time-sharing as described above in connection with elements1514.)

When enabled, each comparator1518compares the output1513of a respective one of counters1512to a threshold value output by threshold value register1516. For example, any desired threshold value may be programmed into register1516. If (and only if) the output1513exceeds the threshold value, comparator1518produces an output for enabling migration request initiation circuitry1520.

The significance of the foregoing is as follows. Whenever the count of accesses of a data block by a non-owner node exceeds both (1) the number of accesses of the data block by that data block's current owner node and (2) a predetermined threshold number of accesses (from register1516), an attempt will be made to migrate (transfer) that data block from the current owner node to the above-mentioned other node in order to make that other node the new owner of the data block. This tends to give ownership of each data block to the node that is making most frequent use of (i.e., most frequently accessing) that data block. This can greatly increase the access efficiency of the distributed memory system as a whole. The data block migrations needed to produce this result are carried out by elements1520,1530, etc. inFIG. 15b, as will now be described.

When circuitry1520is enabled as mentioned earlier, circuitry1520knows (by knowing which comparator1518enabled it) which data block (“the transferred block”) needs to be migrated, and to which other node (“the destination node”) that data block needs to be migrated. Circuitry1520therefore sends a migration request to the destination node (e.g., via interconnection networks80/210). A migration request (like a read request or a write request) can have the characteristics of a data packet (e.g., as inFIG. 6and described earlier in this specification). Thus, for example, a migration request may have a header including the ID of the destination IC, which enables the interconnect resources80/210of the system to route the migration request to the destination IC. This is similar to what is done for data packets (e.g., as inFIG. 6), read requests, and write requests.

As mentioned earlier, each node (IC10) includes all of the elements shown inFIG. 15. Therefore the illustrated node depicted (in part) inFIG. 15also includes the elements needed to enable that node to be a destination node. The destination node elements can accordingly also be described in connection withFIG. 15(even though in any actual data block migration two different nodes (i.e., a “source node” originating the migration, and the destination node receiving the migration) will be involved. Thus the migration request from the source node is received by migration request acceptance circuitry1530in the destination node. This circuitry1530checks to see whether or not the memory (e.g.,120) connected to that node can receive the data block proposed for transfer (migration). Migration request ACK/NAK (acknowledge/non-acknowledge) circuitry1532of the destination node sends back to the source node either an ACK signal (meaning that the destination node can receive the data block transfer) or a NAK signal (meaning that the destination node cannot receive the data block transfer).

In the source node, migration request ACK/NAK processing circuitry responds to an ACK (and only an ACK) by enabling migration execution circuitry1542to actually send the data block to be migrated to the destination node. (A NAK terminates the attempt to migrate the data block.) When the data block migration has been successfully accomplished, migration report broadcast circuitry1544is enabled to send a broadcast message or report notifying all nodes about the migration of the transferred block. For example, the broadcast migration report allows the circuitry1200(FIG. 12) in all nodes (ICs10) in the system to update the records the nodes maintain as to the locations of all data blocks in the system. This is shown in more detail inFIG. 15c, which is discussed in the next paragraph. Upper layer system components (e.g., file system, database management system, etc., components (not shown)) may also be notified about the migration of the block (e.g., via an external link140(FIG. 1)). AlthoughFIG. 15bshows elements1542and1544operating as part of source node operations, they may alternatively operate as part of destination node operations.

As shown inFIG. 15c, each IC10further includes storage management update circuitry1550for receiving and processing a migration report that has been broadcast as discussed in connection with element1544inFIG. 15b. When such a migration report is received, circuitry1550causes the logic block manager1210in the IC10that includes that circuitry1550to change in that block manager's records (mapping information) the owner node ID of the transferred block from the source node ID to the destination node ID. Similarly, circuitry1550in the source node causes the associated source node flash translation layer1220to delete from that translation layer's records (mapping information) the logical block number of the transferred block, while the circuitry1550in the destination node causes that circuitry's associated destination node flash translation layer1220to add to its records (mapping information) the logical block number of the transferred block. (As an alternative to making these changes to the translation layer1220records in response to the broadcast migration report, these changes could instead be made as part of the data migration operation itself, because these changes only affect the translation layers in the source and destination nodes involved in the migration.) Flash device driver1230inFIG. 15chas already been fully described in connection withFIG. 12.

FIGS. 16a-c(sometimes referred to collectively asFIG. 16) show illustrative embodiments of dynamic data block migration methods that can be performed, e.g., by circuitry of the type shown inFIG. 15in accordance with this disclosure. Each node (IC10) in a distributed memory system may perform theFIG. 16method.

At1610each access of each data block by the owner node of that data block is counted.

At1620each access of each data block by each other node is separately counted.

At1630each count from1620is compared to (1) the count (from1610) of accesses of the same data block by the node that currently owns that data block, and (2) a threshold value. For any data block whose count (from1620) for some non-owner node exceeds both the owner node count (from1610) and the threshold, control passes from1630to1640. The last-mentioned data block may be referred to as the transferred block, and the last-mentioned non-owner node may be referred to as the destination node. (If there is no “yes” outcome from1630, control passes from1630back to1610.)

At1640the current owner block (“the source block”) sends a request to transfer the transferred block to the destination node.

At1650the destination node determines whether or not it can accept the proposed transfer. If not, control passes back to1610and the proposed transfer does not take place. If the destination block can accept the proposed transfer, control passes to1660.

At1660the source node transfers the transferred block to the destination node. At1670a message or report is broadcast to all nodes (ICs10) notifying them about the transfer of the transferred block. At1680upper layer elements such as file system elements, database management system elements, etc., are notified about the migration of the transferred block.

FIG. 16cshows in more detail operations that may be performed in ICs10in response to a message broadcast as discussed above in connection with element1670inFIG. 16b. TheFIG. 16coperations are performed to update the records (mapping information) in elements1210and1220(e.g.,FIGS. 12 and 15c) in view of the data block migration (transfer) that has taken place. At1672the record of the owner node ID of the transferred block is changed (from the source node ID to the destination node ID) in all logic block manager circuits1210throughout the system. At1674the flash translation layer1220in the source node has that translation layer's records updated by deleting the logical block number of the transferred block. At1676the flash translation layer1220in the destination node has that translation layer's records updated by adding the logical block number of the transferred block. (Again, a possible alternative is to perform operations1674and1676in connection with the actual data migration, rather than in response to a broadcast migration report.)

Throughout this disclosure, references to “data,” “information,” or the like refer to physical embodiments of such data, information, or the like (e.g., as electrical signals, stored electrical charge, particular magnetic states of magnetizable media, etc.). Also throughout this disclosure (as has already been said), terms like “circuit,” “circuitry,” “integrated circuit,” “IC,” and the like can refer to combinations of hardware and software.

It will be understood that the foregoing is only illustrative of the principles of the disclosure, and that various modifications can be made by those skilled in the art without departing from the scope and spirit of the disclosure. For example, systems can be constructed with any number of nodes (ICs10) to provide distributed flash memory systems of any desired size. As another example of modifications within the scope of this disclosure, elements and/or functions that are shown herein as separate may be combined into single elements and/or functions; and elements and/or functions that are shown herein as integral or unitary may be subdivided into two or more separate sub-elements or sub-functions.