Patent Description:
Emerging fabric standards such as Compute Express Link (CXL) <NUM>, Gen-Z, or Slingshot exemplify an approach to datacenter disaggregation in which a central processing unit (CPU) host is able to access Fabric-Attached Memory (FAM) modules. Such modules contain memory attached to the datacenter fabric with no or few compute capabilities associated with it. With FAM, hosts are not constrained with the memory capacity limitations of their local servers. Instead, hosts gain access to vast pools of memory which need not be attached to any particular host. The FAM is partitioned among the hosts, and partitions may be dedicated to a host or shared among multiple hosts.

Memory consistency is an important consideration in the process of developing software applications for use with FAM systems. Consistency defines how the memory instructions (to different memory locations) in a multi-processor or multi-threaded system will be ordered, and is implemented by reordering independent memory operations according to a consistency model.

Various consistency models have developed that impose various ordering constraints on independent memory operations in a single processor's instruction stream where high level dependence is involved. In a simple consistency model, known as Sequential Consistency, the processor is not allowed to reorder reads and writes. Another model, known as "Total Store Order" (TSO), allows store buffering. In this scheme, a store buffer holds store operations that need to be sent to memory until designated conditions are met and a group of operations can be sent to memory. Loads are allowed to pass stores, but the stores are sent to memory in program order. The address of a load operation is checked against addresses in the store buffer, and the store buffer is used to satisfy the load operation if there is an address match. Publication <CIT>discloses a data processing system including an interconnect fabric, a system memory coupled to the interconnect fabric and including a virtual barrier synchronization region allocated to storage of virtual barrier synchronization registers (VBSRs), and a plurality of processing units coupled to the interconnect fabric and operable to access the virtual barrier synchronization region.

Other consistency models known as relaxed or weak consistency models rely on some version of a fence (or barrier) operation that demarcates regions within which reordering of operations is permissible. Release consistency is one example of weak consistency model, where synchronization accesses are divided into "Acquire", in which operations like lock must complete before all following memory accesses, and "Release", in which operations like unlock must complete with all memory operations before release are complete.

In the following description, the use of the same reference numerals in different drawings indicates similar or identical items. Unless otherwise noted, the word "coupled" and its associated verb forms include both direct connection and indirect electrical connection by means known in the art, and unless otherwise noted any description of direct connection implies alternate embodiments using suitable forms of indirect electrical connection as well.

A method is for use with a fabric-attached memory system including a fabric-attached memory and a plurality of requestors coupled to the fabric-attached memory through a fabric. Notifications are requested from a fabric manager regarding changes in requestors authorized to access a fabric-attached memory region. In response to a notification from the fabric manager indicating that more than one requestor is authorized to access the fabric-attached memory region, for each requestor so authorized, fences are activated for selected memory access instructions in a local application concerning the fabric-attached memory region.

A data processor includes a processing core, a fabric-attached memory interface, and a requestor-side adaptive consistency controller. The processing core executes an application. The fabric-attached memory interface is coupled to the processor core and adapted to connect to a data fabric and fulfill memory access instructions from the processing core to a fabric-attached memory. The requestor-side adaptive consistency controller is coupled to the processing core and the fabric-attached memory interface and requests notifications from a fabric manager for the fabric-attached memory regarding changes in requestors authorized to access a fabric-attached memory region which the data processor is authorized to access. Responsive to a notification from the fabric manager indicating that more than one requestor is authorized to access the fabric-attached memory region, the requestor-side adaptive consistency controller causes fences to be activated for selected memory access instructions in a local application.

A fabric-attached memory system includes a fabric-attached memory, a data fabric, a fabric manager, and a plurality of data processors. The data fabric is connected to the fabric-attached memory. The fabric manager is connected to the data fabric and operable to authorize and de authorize requestors to access memory regions of the fabric-attached memory. The plurality of data processors are connected to the data fabric and each including a processing core executing an application, a fabric-attached memory interface, and a requestor-side adaptive consistency controller coupled to the processing core and the fabric-attached memory interface. The requestor-side adaptive consistency controller request notifications from the fabric manager regarding changes in requestors authorized to access a fabric-attached memory region which the data processor is authorized to access. Responsive to a notification from the fabric manager indicating that more than one requestor is authorized to access the fabric-attached memory region, the requestor-side adaptive consistency controller causes fences to be activated for selected memory access instructions in a local application.

<FIG> illustrates in block diagram form a fabric-attached memory (FAM) system <NUM> according to the prior art. The depicted FAM system <NUM> is merely one example of a data fabric topology among many topologies that are often employed for disaggregated data centers. FAM system <NUM> generally includes a data center fabric <NUM>, and a number of device groups <NUM> referred to as pods.

Each pod <NUM> contains multiple compute nodes "C", multiple memory nodes "M", and an interconnect network "ICN". Compute nodes C are connected to the ICN through routers "R". Compute nodes C contain multiple CPUs (multiple cores each) or multiple accelerated processing units (APUs) that are part of the same consistency domain. Each compute node C contains a fabric bridge such as a network interface card (NIC), CXL interface, or other suitable fabric interface that is a gateway into datacenter fabric <NUM> for the compute note C. Memory nodes M are connected to the ICN through routers R. Each memory node M includes a similar fabric interface and a media controller that satisfies the requests to FAM. The ICN includes switches for interconnecting the various compute nodes C with memory nodes M, and may include routers in some topologies.

The depicted topology includes a local datacenter fabric formed by routers R and the ICN, and a global data center fabric labeled data center fabric <NUM>. In this embodiment, the local data center fabric is within a rack, and the global data center fabric includes multiple racks. However, various fabric topologies may be implemented within a rack or within a datacenter, and may include compute nodes accessing the datacenter remotely through a network. It is noted that many topologies have compute nodes C that also include memory which is part of the FAM pool. Such memory may be mapped as fabric-attached memory and made available for use by other compute nodes according to a resource allocation process referred to as "composability".

Data center fabric <NUM> provides data interconnect between pods <NUM>, including switches and routers coupling data traffic in a protocol such as CXL, Gen-Z, or other suitable memory fabric protocol. It is noted that multiple protocols may be employed together in a data center fabric. In this exemplary embodiment, CXL is employed to interconnect devices within a rack, while Gen-Z is employed to interconnect various racks within the data center.

<FIG> illustrates in block diagram form an APU <NUM> which is suitable for use as a compute unit C in a FAM system such as FAM system <NUM> of <FIG>. APU <NUM> is an integrated circuit suitable for use as a processor in a host data processing system, and includes generally a central processing unit (CPU) core complex <NUM>, a graphics core <NUM>, a set of display engines <NUM>, a data fabric <NUM>, a memory management hub <NUM>, a set of peripheral controllers <NUM>, a set of peripheral bus controllers <NUM>, and a system management unit (SMU) <NUM>, and a group of memory interfaces <NUM>.

CPU core complex <NUM> includes a CPU core <NUM> and a CPU core <NUM>. In this example, CPU core complex <NUM> includes two CPU cores, but in other embodiments CPU core complex <NUM> can include an arbitrary number of CPU cores. Each of CPU cores <NUM> and <NUM> is bidirectionally connected to a system management network (SMN), which forms a control fabric, and to local data fabric <NUM>, and is capable of providing memory access requests to data fabric <NUM>. Each of CPU cores <NUM> and <NUM> may be unitary cores, or may further be a core complex with two or more unitary cores sharing certain resources such as caches. Each of CPU cores <NUM> and <NUM> includes µCode <NUM> which runs to execute certain instructions on the CPU, including performing certain functions for memory consistency on a data center fabric as further described below.

Graphics core <NUM> is a high performance graphics processing unit (GPU) capable of performing graphics operations such as vertex processing, fragment processing, shading, texture blending, and the like in a highly integrated and parallel fashion. Graphics core <NUM> is bidirectionally connected to the SMN and to data fabric <NUM>, and is capable of providing memory access requests to data fabric <NUM>. In this regard, APU <NUM> may either support a unified memory architecture in which CPU core complex <NUM> and graphics core <NUM> share the same memory space, or a memory architecture in which CPU core complex <NUM> and graphics core <NUM> share a portion of the memory space, while graphics core <NUM> also uses a private graphics memory not accessible by CPU core complex <NUM>. Memory regions may be assigned from local memory or a data center fabric.

Display engines <NUM> render and rasterize objects generated by graphics core <NUM> for display on a monitor. Graphics core <NUM> and display engines <NUM> are bidirectionally connected to common memory management hub <NUM> for uniform translation into appropriate addresses in system memory.

Local data fabric <NUM> includes a crossbar switch for routing memory access requests and memory responses between any memory accessing agent and memory management hub <NUM>. It also includes a system memory map, defined by basic input/output system (BIOS), for determining destinations of memory accesses based on the system configuration, as well as buffers for each virtual connection.

Peripheral controllers <NUM> include a universal serial bus (USB) controller <NUM> and a Serial Advanced Technology Attachment (SATA) interface controller <NUM>, each of which is bidirectionally connected to a system hub <NUM> and to the SMN bus. These two controllers are merely exemplary of peripheral controllers that may be used in APU <NUM>.

Peripheral bus controllers <NUM> include a system controller or "Southbridge" (SB) <NUM> and a Peripheral Component Interconnect Express (PCIe) controller <NUM>, each of which is bidirectionally connected to an input/output (I/O) hub <NUM> and to the SMN bus. I/O hub <NUM> is also bidirectionally connected to system hub <NUM> and to data fabric <NUM>. Thus for example a CPU core can program registers in USB controller <NUM>, SATA interface controller <NUM>, SB <NUM>, or PCIe controller <NUM> through accesses that data fabric <NUM> routes through I/O hub <NUM>. Software and firmware for APU <NUM> are stored in a system data drive or system BIOS memory (not shown) which can be any of a variety of non-volatile memory types, such as read-only memory (ROM), flash electrically erasable programmable ROM (EEPROM), and the like. Typically, the BIOS memory is accessed through the PCIe bus, and the system data drive through the SATA interface.

SMU <NUM> is a local controller that controls the operation of the resources on APU <NUM> and synchronizes communication among them. SMU <NUM> manages power-up sequencing of the various processors on APU <NUM> and controls multiple off-chip devices via reset, enable and other signals. SMU <NUM> includes one or more clock sources (not shown), such as a phase locked loop (PLL), to provide clock signals for each of the components of APU <NUM>. SMU <NUM> also manages power for the various processors and other functional blocks, and may receive measured power consumption values from CPU cores <NUM> and <NUM> and graphics core <NUM> to determine appropriate power states.

Memory management hub <NUM> is connected to local data fabric <NUM>, graphics core <NUM> and display engines <NUM> for providing direct memory access capability to graphics core <NUM> and display engines <NUM>.

Memory interfaces <NUM> include two memory controllers <NUM> and <NUM>, DRAM media <NUM> and <NUM>, and a FAM memory interface <NUM>. Each of memory controllers <NUM> and <NUM> are connected to local data fabric <NUM> and connected to a respective one of DRAM media <NUM> and <NUM> through a physical layer (PHY) interface. In this embodiment, DRAM media <NUM> and <NUM> include memory modules based on based on DDR memories such as DDR version five (DDR5). In other embodiments, other types of DRAM memory are used, such as low power DDR4 (LPDDR4), graphics DDR version five (GDDR5), and high bandwidth memory (HBM).

FAM memory interface <NUM> includes a fabric bridge <NUM>, an adaptive consistency controller (ACC) <NUM>, and a fabric PHY <NUM>. Fabric bridge <NUM> is a fabric-attached memory interface connected to local data fabric <NUM> for receiving and fulfilling memory requests to a FAM system such as FAM system <NUM>. Such memory requests may come from CPU core complex <NUM> or may be direct memory access (DMA) requests from other system components such as graphics core <NUM>. Fabric bridge <NUM> is also bidirectionally connected to fabric PHY <NUM> to provide the connection of APU <NUM> to the data center fabric. An adaptive consistency controller (ACC) <NUM> is bidirectionally connected to fabric bridge <NUM> for providing memory consistency control inputs to fabric bridge <NUM> and CPU core complex <NUM>, as further described below. In operation, ACC <NUM> communicates with CPU cores in CPU core complex <NUM> to receive notifications that designated memory access instructions have been recognized by µCode <NUM> running on CPU cores <NUM> and <NUM>, as further described below. ACC <NUM> also provides configuration inputs to CPU core complex <NUM> for configuring memory consistency models.

<FIG> illustrates in block diagram form certain elements of a FAM system <NUM> according to some embodiments. FAM system <NUM> generally includes a compute node <NUM>, data center fabric <NUM>, FAM memory node <NUM>, and fabric manager <NUM>.

Compute node <NUM> is one of many requestor compute nodes connected to data center fabric <NUM> and generally is implemented with an APU such as APU <NUM>. Compute node <NUM> can implement an internet server, an application server, a supercomputing node, or another suitable computing node that benefits from accessing a FAM. Only the FAM interface components of compute node <NUM> are depicted in order to focus on the relevant portions of the system. Compute node <NUM> includes fabric bridge <NUM>, fabric PHY <NUM>, and ACC <NUM>.

Fabric bridge <NUM> is connected to the local data fabric as described above with respect to <FIG>, and to fabric PHY <NUM> and ACC <NUM>. ACC <NUM> in this version includes a microcontroller (µC) <NUM>. The µC <NUM> performs memory consistency control functions as described below, and is typically also connected to a tangible non-transitory memory for holding firmware to initialize and configure µC <NUM> to perform its functionality.

The µC <NUM> performs memory consistency control functions as described below, and is typically also connected to a tangible non-transitory memory for holding firmware to initialize and configure µC <NUM> to perform its functionality.

Fabric manager <NUM> is a controller connected to data center fabric <NUM> for managing configuration and access to FAM system <NUM>. Fabric manager <NUM> executes a data fabric management application for the particular standard employed on data center fabric <NUM>, such as CXL or Gen-Z. The data fabric management application manages and configures data center fabric functions such as authorizing compute nodes, allocating memory regions, and managing composability by identifying and configuring memory resources among the various nodes in FAM system <NUM>. It is noted that while one FAM memory node <NUM> and one compute node <NUM> are shown, the system includes multiple such nodes which can appear in many configurations such as the example configuration depicted in <FIG>. In some embodiments, fabric manager <NUM> has an adaptive consistency controller (ACC) module <NUM> installed for accessing the data fabric management application and reporting data to each ACC <NUM> on respective compute nodes throughout FAM system <NUM>.

FAM memory node <NUM> includes a media controller <NUM> and a memory <NUM>. Media controller <NUM> typically includes a memory controller suitable for whatever type of memory is selected for use in memory <NUM>. For example, if memory <NUM> is a DRAM memory, a DRAM memory controller is used. Memory <NUM> may also include persistent memory modules and mixed. In some embodiments, ACC <NUM> maintains data in requestor table <NUM> at FAM memory node <NUM> concerning other requestors authorized to access a FAM memory region allocated to compute node <NUM>. Requestor table <NUM> tracks updates to the compute nodes authorized to access the same memory region, and includes fields for a "Timestamp" reflecting the time of the update, a "Region ID" reflecting an identifier for the FAM memory region allocated to compute node <NUM>, and a "# Requestors" reflecting the number of requestors on FAM system <NUM> that were authorized to access the memory region as of each update. Requestor table <NUM> is updated based on reporting from fabric manager <NUM> as further described below. In this embodiment, FAM memory node <NUM> includes a buffer accessable by media controller <NUM> holding requestor table <NUM>.

<FIG> shows a diagram <NUM> illustrating a process of operating an ACC <NUM> according to some embodiments. Diagram <NUM> depicts activity at a requestor compute node at the data fabric, and shows CPU core complex <NUM>, an operating system <NUM>, ACC <NUM>, and fabric bridge <NUM> at the requestor compute node. Diagram <NUM> also shows fabric manager <NUM> and a media controller <NUM>, which is one of many media controllers on the data fabric.

When the requestor is assigned a particular FAM region to use for system memory by fabric manager <NUM>, ACC <NUM> makes a callback request to fabric manager <NUM> to request notifications when changes are made to the number of requestors which authorized to use the same memory region as the requestor compute node, as shown by the outgoing request labeled "CALLBACK". Fabric manager <NUM> provides a notification back to ACC <NUM> each time the number of requestors authorized to use the memory region changes, as indicated by the "# USERS" response on diagram <NUM>. In this embodiment, the requestor table <NUM> (<FIG>) at FAM memory node <NUM> is updated each time the number of users changes, as indicated on diagram <NUM> by the "# Users" arrow going to media controller <NUM>. In some embodiments, ACC <NUM> maintains a requestor table in a buffer local to ACC <NUM>. In some embodiments, ACC module <NUM> (<FIG>) running at fabric manager <NUM> manages the process of monitoring requestors authorized to access the memory region and sending notifications to ACC <NUM>. Based on the # USERS update notifications, ACC <NUM> sets a consistency model for use with the assigned FAM memory region. Generally this process includes, when the current requestor is the only requestor authorized to access the memory region, setting a the consistency model to a first consistency model, and when more than one requestor is authorized to access the memory region, setting the consistency model to a second consistency model. Setting the consistency model is shown by the command "SET MODEL (FAMi) = (WB, FENCED)", where "FAMi" identifies the fabric-attached memory region concerned, and "WB, FENCED" indicates which consistency model is to be activated. In some embodiments, the first consistency mode is characterized as being relaxed with respect to the second consistency model. An example of this process is further described below with respect to <FIG>.

In diagram <NUM>, µCode <NUM> (<FIG>) running at CPU core complex <NUM> helps to implement the second consistency model when it is active. Specifically, µCode <NUM> recognizes designated memory access instructions in an application executing at CPU core complex <NUM> which indicate data fencing is needed for designated memory instructions concerning the fabric-attached memory region assigned to the requestor node. The µCode <NUM> has a variety of ways to recognize the designated instructions as further described below with respect to <FIG>. When such an instruction is recognized, µCode <NUM> communicates with ACC <NUM> to send notifications identifying the selected instructions which will be sent to the data fabric through fabric bridge <NUM>. ACC <NUM> then adds fence commands to the command stream going to media controller <NUM> as indicated by the outgoing arrow labeled "FENCE". When the first consistency model is active, µCode <NUM> will not make such notifications and instead will allow the selected commands to execute normally with memory consistency handled by the first consistency model set with operating system <NUM>. While in this embodiment µCode <NUM> recognizes the designated instructions, in other embodiments this function is performed by CPU firmware or a combination of CPU firmware and µCode.

While the diagram shows fence commands going to media controller <NUM>, in topologies including a local datacenter fabric and a global datacenter fabric, ACC <NUM> will causes fence commands to be sent to media controllers on both the local datacenter fabric and the global datacenter fabric in scenarios for which the memory region includes accessing both levels of the fabric topology.

<FIG> shows a flowchart <NUM> of a process for managing memory consistency models at an adaptive consistency controller according to an exemplary embodiment. The process begins at block <NUM> where a requestor node on a data fabric is authorized to access a designated FAM memory region. Such authorization is typically provided by the fabric manager but in some embodiments may be configured by other system components. Based on the authorization the memory region is established in the addressable system memory space of the requestor node. The memory region is typically used for an application running at the requestor node which may have dependencies with other requestor nodes.

At block <NUM>, ACC <NUM> at the requestor node requests notifications from the fabric manager regarding any changes in the number of requestors authorized to access the particular memory region. In one embodiment, this request has the form of a callback request to the fabric manager to track the number of compute requestors that are accessing the FAM region. ACC module <NUM> (<FIG>) may be employed to receive such requests and either implement the request at the fabric manager or configure the fabric manager to implement the request. In other embodiments, the fabric manager may have such capability as part of the fabric manager application and not require an additional module.

At block <NUM>, ACC <NUM> at the requestor node receives a notification from the fabric manager in response to the request. Based on this notification, ACC <NUM> determines the number of requestors currently authorized to use the FAM region and updates requestor table <NUM> (<FIG>). In some embodiments, the notification includes the data fields used in requestor table <NUM>, including Timestamp, Region ID, and # Requestors. In other embodiments, the fabric manager may not provide all the data instead only provide data indicating that a requestor authorization was added or removed for the FAM region. In such case, ACC <NUM> will update the data in requestor table <NUM> based on the current notification data and the previous update to requestor table <NUM>.

At block <NUM>, if more than one requestor is authorized for the memory region, the process goes to block <NUM>. If not, the process goes to block <NUM>. At block <NUM>, the process causes fences to be activated for selected memory access instructions in the local application concerning the FAM region. If at block <NUM> a transition is made from having only one requestor authorized to more than one, the process includes deactivating the first memory consistency model and activating the second memory consistency model at the requestor node, as described above with respect to <FIG>. If more than one requestor was already authorized at block <NUM>, then the second memory consistency model is already active and does not need to be changed.

At block <NUM>, the update notification received at block <NUM> has resulted in a state in which only one requestor is authorized for the FAM region, and so the process activates the first memory consistency model. In some embodiments, the first memory consistency model includes mapping the FAM region as write-back memory for the single requestor. Mapping the FAM region as write-back memory for the local compute node is preferably accomplished by ACC <NUM> sending appropriate messaging to the operating system running on CPU core complex <NUM>, which then marks the memory page or pages that correspond to the FAM region as write-back in the requestor's page tables. Mapping the FAM region as write-back memory ensures that the normal, local consistency scheme (such as an x86 total-store-order (TSO) consistent scheme) will be applied by the compute node for the FAM region. When only one requestor is authorized for the FAM region, no changes are required to the application's functionality to use FAM rather than local memory. The process of blocks <NUM>, <NUM>, <NUM>, and <NUM> is repeated each time the fabric manager sends an update notification for the FAM memory region concerned.

<FIG> shows a flowchart <NUM> of a process for implementing a memory consistency model according to some embodiments. Generally, applications which are not originally written for use with a FAM system, if they have any dependencies between compute nodes, require modifications in order for the FAM system to properly account for dependencies between compute nodes. Such modifications can be made when compiling the code, or can be applied after-the-fact by adding instructions to selected memory commands in the application. The process of <FIG> is performed for applications in which compile-time instructions have been added in order to adapt the application for use with a FAM system. Such applications include applications that are written and compiled with tools for handling consistency in a FAM system, and applications that have been modified with a suitable tool to be adapted for use with a FAM system.

At block <NUM> the process begins activating fences for an application with compile-time instructions included in the application. Block <NUM> occurs when ACC <NUM> changes the consistency model for a compute node to activate the second, more relaxed, consistency model. Generally, data fabric protocols like CXL <NUM> and Gen-Z employ a relaxed-ordering consistency model implemented via fences. When the relaxed consistency model is active, ACC <NUM> seeks to insert datacenter fabric fences such as CXL/Gen-Z fences in the critical locations transparently to the application code. This transparency means that activity performed by ACC <NUM> should not require any adjustments by the application concerned.

At block <NUM>, the process recognizes compile-time fabric interface instructions for selected memory access instructions. Generally, the selected memory access instructions are parts of the code such as flags, locks, semaphores, control variables, etc. that perform parallel synchronization in between compute threads, so the memory accesses need to be ordered. The locations of the selected memory access instructions are identified with hints in the application that can be provided by the software developer to the adaptive controller via several mechanisms.

Compiler hints like C++<NUM> atomic_load/atomic_store consistency constructs or primitives or runtime hints like OpenMP's flush construct can be integrated with a development tool such as the "CodeAnalyst" tool (by Advanced Micro Devices of Santa Clara, California) to make the consistency hints easier to share for the developer. The compiler inserts requestor-side fabric interface markers such as a special "FABRIC_ACQUIRE_FENCE" instruction before the marked control variable and a special "FABRIC_RELEASE_FENCE" instruction after the control variable. These special instructions or markers are converted to no-operation (NOP) by the CPU's µCode <NUM> on a non-FAM system or a FAM system with only one requestor node accessing the relevant FAM region, but are recognized at block <NUM> by µCode <NUM> when more than one application is authorized to access the relevant FAM region.

Responsive to recognizing such instructions, at block <NUM> the CPU µCode <NUM> notifies ACC <NUM> that a fabric fence is needed for the instruction, as shown by the SELECTED INSTRUCTIONS arrow in <FIG>. In some embodiments, such notification occurs with a message over the local data fabric from the CPU to ACC <NUM>. The notification includes identifying information for ACC <NUM> to identify the instruction when it is received over the local data fabric, such as the memory address of the variable concerned. In other embodiments, the notification may be implemented by adding a predetermined flag or marker to the memory access instruction to be modified when the instruction is sent by the CPU over the local data fabric to the fabric bridge <NUM>. In some embodiments, such a marker includes information necessary to determine whether an acquire fence instruction is needed, or a release fence instruction.

At block <NUM>, in response to receiving each notification, ACC <NUM> will issue acquire/release datacenter fabric fences. These fences are commands inserted into the command stream that goes to the data fabric and on to the media controller for the relevant FAM region, as shown by the FENCE arrow in <FIG>. The media controller then implements the commands to provide fences for the variables concerned.

While the depicted process occurs after the application is already modified to include FABRIC_ACQUIRE_FENCE and FABRIC_RELEASE_FENCE instructions, in some embodiments the process also includes inserting such hints or markers into the application such that µCode <NUM> can recognize the selected memory instructions.

<FIG> shows a flowchart <NUM> of another process for implementing a memory consistency model according to some additional embodiments. The depicted process is performed for applications that are not modified with compile-time instructions for implementing fabric fences. This process has the advantage that applications for which the developers have not provided a version configured for use with FAM system can still be used with a FAM system without causing consistency problems that would ordinarily occur with such use. Another advantage is for deployments in which a separate or more expensive software license is required to obtain an application version for use with a FAM system, the process can enable using the non-FAM version with a FAM system.

At block <NUM>, the process begins activating FAM fences for such an application. Recognizing the selected memory access instructions that need a fence command is different in this process than the process of <FIG> because no specific instructions are included in the application for fabric fences. At block <NUM>, the CPU µCode <NUM> recognizes the memory access instructions in the application for which fences are needed based recognizing instructions in a predetermined list of instruction types and instruction prefixes associated with synchronization of data between threads. In some embodiments, the predetermined list is provided for µCode <NUM> to cover all types of applications for which it is able to recognize the selected memory access instructions. The list includes instructions and instruction prefixes which are associated with dependencies between applications. For example, this list can include x86's LOCK instruction prefix, X86's xacquire/xrelease, and PowerPC's SYNC instruction.

At block <NUM>, whenever any instruction from this predefined list is invoked, the CPU µCode <NUM> will notify ACC <NUM>, which will then issue the fabric fence as shown at block <NUM>.

FAM memory interface <NUM>, or any portions thereof, such as ACC <NUM> or fabric bridge <NUM>, may be described or represented by a computer accessible data structure in the form of a database or other data structure which can be read by a program and used, directly or indirectly, to fabricate integrated circuits. For example, this data structure may be a behavioral-level description or register-transfer level (RTL) description of the hardware functionality in a high level design language (HDL) such as Verilog or VHDL. The description may be read by a synthesis tool which may synthesize the description to produce a netlist including a list of gates from a synthesis library. The netlist includes a set of gates that also represent the functionality of the hardware including integrated circuits. The netlist may then be placed and routed to produce a data set describing geometric shapes to be applied to masks. The masks may then be used in various semiconductor fabrication steps to produce the integrated circuits. Alternatively, the database on the computer accessible storage medium may be the netlist (with or without the synthesis library) or the data set, as desired, or Graphic Data System (GDS) II data.

Claim 1:
A method for use with a fabric-attached memory system (<NUM>) including a fabric-attached memory (<NUM>) and a plurality of requestors (<NUM>) coupled to the fabric-attached memory through a fabric (<NUM>), comprising:
requesting notifications from a fabric manager (<NUM>) regarding changes in requestors authorized to access a fabric-attached memory region; and
in response to a notification from the fabric manager indicating that more than one requestor is authorized to access the fabric-attached memory region, causing fences to be activated for selected memory access instructions.