Patent ID: 12223355

DETAILED DESCRIPTION

This disclosure relates to multiprocessing and, more particularly, to synchronizing system resources within a multi-socket (MS) data processing system. The inventive arrangements also may be applied to symmetric multiprocessing (SMP) data processing systems. An MS data processing system may be implemented in which each socket includes a System-on-Chip (SOC). Each SOC may include a plurality of processor cores.

In an MS-SMP data processing system, certain resources must be synchronized across the sockets once the boot process completes to ensure proper and error free operation. Resources that are architecturally defined within the MS-SMP data processing system as one per SMP (e.g., one per SOC) require this synchronization. Examples of these resources include, but are not limited to, system counters, debug trace timestamp generators, and the like.

For purposes of illustration, the system counter associated with each socket is free running. Despite each system counter being configured to count at the same frequency, there is no guarantee that each system counter starts at the same time. If the system counters of the respective sockets have different start times, the values of the system counters will not be equal or in synchronization. With the system counter for each socket being out of synchronization, the operating system of the MS-SMP data processing system may exhibit unpredictable behavior when managing and scheduling processes on processor cores in different sockets.

Debug trace timestamps are another example of a system resources that requires synchronization. When debug trace timestamps are not synchronized, the chronology of debug and/or trace events generated and stored in the system may be incorrect. This makes establishing the true chronology of such events prohibitively difficult. Debugging activities may be hindered by the inaccurate event chronology. Synchronizing debug trace timestamps improves the debugging capabilities of the MS-SMP system by improving accuracy of the event chronology, which facilitates debugging efforts.

The inventive arrangements described within this disclosure provide methods, systems, and computer-program products capable of ensuring that certain system resources of an MS data processing system and/or MS-SMP data processing system are synchronized. Further aspects of the inventive arrangements are described below with reference to the figures.

FIG.1illustrates an example system100. System100is an example of an MS data processing system. Further, system100may be an MS-SMP data processing system. In the example ofFIG.1, system100includes a plurality of SOCS102. System100may include fewer or more SOCS102than shown. In the example ofFIG.1, each SOC102is disposed in its own socket. In the example ofFIG.1, each SOC102includes a plurality of processor cores112. The processor cores112may be hardwired processor cores (e.g., hardened circuits). Each processor core may be configured to execute program code. Further, each SOC102may be identical.

In another aspect, each SOC102may include one or more “soft processors” formed from programmable logic. In still another aspect, each SOC102may include a combination of hardwired and soft-processors. In the case where one or more SOCS102implement a soft-processor, those SOCS102that do implement soft-processor(s) should be implemented such that the soft-processor(s) have access to the synchronized system resource.

In one aspect, each of SOCS102may be disposed on a same circuit board or card. In another aspect, each of SOCS102may be disposed on a different circuit board or card. In other aspects, two or more circuit boards or cards may be used where SOCS102are distributed across such circuit boards/cards in varying combinations of one or more SOCS per circuit board. SOCS102may be used as the processors of a data processing system or may be accelerators included in a larger data processing system that includes a host processor or Central Processing Unit (CPU).

Within system100, each socket, and as such each SOC102, may be identified with a socket identifier (ID) that uniquely identifies socket and, as such, the SOC102disposed in the socket. Though the operating system of an MS-SMP data processing system may view each SOC equally, one SOC may be designated as a primary SOC. Thus, the SOC102disposed in the primary socket may be designated as the primary SOC. The SOC disposed in the primary socket is generally responsible for handling operations relating to configuration and booting of system100, including the other sockets. For example, the primary SOC may perform functions such as boot, power management, error, and health management of system100. In the example ofFIG.1, SOC102-1is the primary SOC102.

SOCS102are coupled to a communication bus104. In one aspect, communication bus104may be implemented as a Peripheral Component Interconnect Express (PCIe) bus. As discussed, a coherent communication link may run over bus104. SOCS102are also communicatively linked to a global synchronization circuit (GSC)106. In the example ofFIG.1, GSC106is coupled to each of SOCS102via one or more sideband channels. For example, each SOC102may be connected to GSC106via an outgoing sideband channel108(e.g.,108-1,108-2,108-3, and108-N). Each SOC102is also coupled to GSC106via an incoming sideband channel110over which events and/or interrupts may be broadcast from GSC106. The sideband channels are different physical signal paths than are implemented with respect to communication bus104. That is, sideband channels may be distinct from communication bus104.

In one or more example implementations, sideband channel108-1may be included while sideband channels108-2,108-3, and108-N may be omitted.

In one or more example implementations, where SOCs102are disposed on a same circuit board, GSC106may be disposed on the same circuit board as SOCs102. In another example implementation, GSC106may be disposed on a circuit board that is separate and distinct from the circuit boards on which SOCs102are disposed. In still another example implementation, where SOCs102are disposed on a plurality of different circuit boards, GSC106may be disposed on one of the circuit boards that includes one or more of the SOCs102. GSC106may be coupled via the sideband channels to SOCs102(e.g., to sockets for the SOCs102) via wires/traces in the circuit boards and/or by dedicated cables.

In one or more example implementations, each SOC102may include a particular processor core that is designated for performing the synchronization related functions described herein. In some cases, the designated processor core is referred to as the System Control Processor (SCP). An SCP typically handles boot operations, power management, error management, and health management of the system.

In other cases, the designated processor core is referred to as a System Resource Reset Processor (SRRP). It should be appreciated that in cases where the SOCS102are identical, the terms SCP and SRRP, to the extent such terms are used to refer to particular processor cores in the primary SOC and/or in non-primary SOCS, are used only to refer to the roles of the respective designated processor cores in the respective SOCS102. The SCP and SRRPS may be the same processor core from an architectural perspective.

In one aspect, the SCP/SRRPS are specialized processor cores in each SOC102tasked with performing the operations described within this disclosure. In one or more other example implementations, the SCP/SRRPS of each SOC102may be the same as the other processor cores in the respective SOC102, but designated or selected to perform the functions described herein. That is, the processor cores may be identical with the SCP/SRRP being one of the plurality of identical processor cores designated as such in the respective SOCS102.

Each SOC102further includes one or more synchronized system resources (SSRS)114. Within this disclosure, system counters are used as example SSRS114. A system counter is a circuit that monotonically increases and does not roll-over during the life of the boot session of the system. Processor cores of an SOC102may subscribe to the system counter for that SOC and use the system counter to generate timer events on a per processor core basis, which may be used by the operating system and/or applications to obtain a notion of time, for scheduling, and the like.

Other examples of SSRS114may include debug trace timestamp generators. It should be appreciated, however, that SSRS114are not intended to be limited to the examples described. In general, SMP resources that are architecturally defined as one per SMP, e.g., one per SOC102, may require synchronization and may be considered SSRS114.

FIG.2illustrates an example method200depicting certain operative features of system100ofFIG.1. For purposes of illustration, SOC102-1is designated as the “primary” SOC in system100.FIG.3is a signal flow diagram illustrating example communications among SOCS102of system100ofFIG.1.FIGS.2-3illustrate example operations for synchronizing SSRS114within system100and may be performed as part of a boot process of system100. For purposes of illustration, SSRS114are considered to be system counters. Initially, as noted, SSRS114are free running and have different values as SSRS114as the system resources did not have synchronized start times. Rather, one or more or all of the SSRS114may have started operating at different times. The different values of SSRS114are reflected in the values stored in each shown as X1for SSRS114-1, X2for SSRS114-2, and XNfor SSRS114-N.

Referring toFIGS.2-3, in block202, SCP302-1, the SCP of the primary SOC102-1, provides a trigger event304to GSC106. In one aspect, trigger event304may be a pulse. In the example, SCP302-1is configured to provide trigger event304over a first sideband channel108-1to GSC106. In one or more example implementations, only SCP302-1, e.g., the SCP of the primary SOC, is configured to generate trigger event304and forward the trigger event to GSC106via a sideband channel.

In block204, in response to trigger event304, GSC106is capable of broadcasting a synchronization event306over sideband channel110to each of the plurality of SOCS102. In one aspect, synchronization event306is an interrupt. Each SOC102, in response to synchronization event306, is capable of initiating an interrupt service routine. In block206, SCP302-1, SRRP302-2,302-N in each SOC102is interrupted in response to receiving synchronization event306. An “interrupt service routine,” also referred to as an interrupt handler, is a software process invoked by an interrupt request that may be generated from a hardware device, a software interrupt instruction, or a software exception. The interrupt service routine handles the request and sends the request to a particular program or portion of code that, in executing, interrupts the active process executed by the processor/processor core prior to the occurrence of the interrupt. When the interrupt service routine is complete, the interrupted process is resumed.

In block208, a designated processor core in each SOC102programs308the respective SSR114located in the same SOC. The designated processor core may be SCP302-1in the primary SOC102-1and SRRP302-2,302-N in the non-primary SOCS102-2,102-N. The SSR114of each SOC102is synchronized in consequence of the programming. For example, the programming may write the same configuration data or value to each SSR114. In the example, in response to receiving synchronization event306, SCP302-1, SRRP302-2,302-N is capable of executing an interrupt service routine that causes the respective processor core to program the respective SSR114located in the same SOC. Since each SOC102receives the synchronization event306broadcast over the sideband channel, each receives the synchronization event306at the same time. Referring toFIG.3, each designated processor core (e.g., SCP302-1, SRRP302-2,302-N) writes a value of 0 to the corresponding SSR114at the same time. In consequence, SSRS114are synchronized. As illustrated inFIG.3, each SSRS114continues to operate from the initialized value, e.g., 0 in this case, and has a same value of at the Xsyncthereafter.

FIG.4illustrates another example implementation of system100. In the example ofFIG.4, for ease of description and clarity, only SOCS102-1and102-2are shown. In the example, each SOC102includes a plurality of processor cores402,404,406, and302, a system counter408(e.g., an SSR114), system counter control registers410, a phase locked loop (PLL)412, an output pin414, and an input pin416.

In the example ofFIG.4, SCP302-1is capable of generating trigger event304that is conveyed out of output pin414-1to GSC106via sideband channel108-1. In the example, SRRP302-2may be coupled to output pin414-2. SRRP302-2, however, not being part of primary SOC102-1, does not generate or send trigger events.

In response to receiving trigger event304, GSC106broadcasts synchronization event306over sideband channel110. Each SOC102receives synchronization event306, e.g., the interrupt, via the corresponding input pin416. Synchronization event306is provided to the designated processor core in each SOC102. The designated processor core of the primary SOC102is SCP302-1, while the designated processor core in each non-primary SOLS102is SRRP302-2.

SCP302-1, in response to receiving synchronization event306, executes an interrupt service routine that causes SCP302-1to write configuration data to system control registers410-1to reset system counter408-1. SRRP302-2, in response to receiving synchronization event306, executes the interrupt service routine that causes SRRP302-2to write configuration data to system control registers410-2to reset system counter408-2. Because synchronization event306is broadcast to each SOC102, each SOC102receives synchronization event306at the same time. As such, resetting of system counters408is performed in synchronization.

Though system counters408are free running, each runs off of a common reference clock that flows through PLL412. Once synchronized, each system counter408is capable of counting in a synchronized manner with each other system counter408as each is driven by a respective PLL412that receives a common reference clock. System counters408continue to operate in synchronization with one another once synchronized. Accordingly, once synchronized, processor cores402,404, and406(and SCP302-1and SRRP302-2) in each SOC102each see a same value whether in system counter408-1or408-2at any given time. That is, system counter408-1is synchronized with system counter408-2.

As noted, counters408may be monotonically increasing counters that do not “roll over” during the life of the boot session of system100. Processor cores of each SOC102subscribe to the system counter located in the same SOC in order to generate timer events (e.g., on a per processor core basis) that may be used by the operating system and/or applications to determine or obtain a notion of time and for scheduling.

FIGS.5-7describe an alternative implementation for synchronizing SSRS114that utilizes dedicated sideband channels while avoiding the need for a dedicated SRRP as illustrated inFIGS.2-4.

FIG.5illustrates an example method500depicting certain operative features of system100ofFIG.1. For purposes of illustration, SOC102-1is designated as the “primary” SOC in system100.FIG.6is a signal flow diagram illustrating example communications among SOCS102of system100ofFIG.1.FIGS.5-6illustrate example operations for synchronizing SSRS114within system100and may be performed as part of a boot process of system100. For purposes of illustration, SSRS114are considered to be system counters. Initially, as noted, SSRS114are free running and have different values as SSRS114did not have synchronized start times. The different values of SSRS114are reflected in the values stored in each shown as X1for SSRS114-1, X2for SSRS114-2, and XNfor SSRS114-N.

Referring toFIGS.5-6, in block502, SCP302-1, the SCP of the primary SOC102-1, provides a trigger event304to GSC106. In one aspect, trigger event304may be a pulse. In the example, SCP302-1is configured to provide trigger event304over a first sideband channel108-1to GSC106. In one or more example implementations, only SCP302-1, e.g., the SCP of the primary SOC, is configured to generate trigger event304and forward the trigger event to GSC106.

In block504, in response to trigger event304, GSC106is capable of broadcasting synchronization event306over sideband channel110to each of the plurality of SOCS102. In one aspect, synchronization event306may be a pulse. In the examples ofFIGS.3-7, synchronization event306does not trigger execution of an interrupt service routine. Rather, in the examples ofFIGS.3-7, synchronization event306is provided directly to the control logic (e.g., the control logic for the respective SSR114in each SOC102). In block506, the control logic in each SOC102receives synchronization event306. In block508, in response to receiving synchronization event306, the control logic in each SOC102resets the SSR114therein.

In consequence of block508, the SSR114of each SOC102is synchronized as a result of the programming. For example, the programming may write the same configuration data or value to each SSR114. Since each SOC102receives synchronization event306broadcast over the sideband channel, each receives synchronization event306at the same time. In the example, in response to receiving synchronization event306, the control logic of each SOC102resets the SSR114therein to 0 at the same time as illustrated inFIG.6with the “0” value in the row corresponding to block506. In consequence, SSRS114are synchronized. As illustrated inFIG.6, each of SSRS114continues to operate from the initialized value, e.g., 0 in this case, and has a same value of at the Xsyncthereafter.

FIG.7illustrates another example implementation of system100ofFIG.1. In the example ofFIG.7, for ease of description and clarity, only SOCS102-1and102-2are shown. In the example, each SOC102includes a plurality of processor cores402,404,406, and302, a system counter408(e.g., an SSR114), control logic702, a Phase Locked Loop (PLL)412, an output pin414, and an input pin416.

In the example ofFIG.7, SCP302-1is capable of generating trigger event304that is conveyed out of output pin414-1to GSC106via sideband channel108-1. In the example, SCP302-2may be coupled to output pin414-2. SCP302-2, however, not being part of the primary SOC102-1, does not generate or send trigger events.

In response to receiving trigger event304, GSC106broadcasts synchronization event306over sideband channel110. Each SOC102receives synchronization event306via the corresponding input pin416. Synchronization event306is provided directly to control logic702in each respective SOC102. Control logic702in each SOC102, in response to receiving synchronization event306, resets the corresponding system counter408to 0. Since counter control logic702in each SOC102receives synchronization event306at the same time, resetting of system counters408is performed in synchronization.

Though system counters408are free running, each runs off of a common reference clock that flows through PLL412. Once synchronized, each system counter408is capable of counting in a synchronized manner with each other system counter408as each is driven by a PLL412that receives a common reference clock. System counters408continue to operate in synchronization with one another once synchronized. Accordingly, once synchronized, processor cores402,404, and406(and SCP302-1and SCP302-2) in each SOC102each see a same value whether in system counter408-1or408-2at any given time. That is, system counter408-1is synchronized with system counter408-2.

In the example ofFIG.7, counters408may be monotonically increasing counters that do not “roll over” during the life of the boot session of system100. Processor cores of each SOC102subscribe to the system counter located in the same SOC in order to generate timer events (e.g., on a per processor core basis) that may be used by the operating system and/or applications to determine or obtain a notion of time and for scheduling.

FIGS.8-10describe another alternative implementation for synchronizing SSRS114that utilizes virtualized synchronization without dedicated sideband channels. In the examples ofFIGS.8-10, rather than using a sideband channels, system software is used to perform the synchronization as a part of a boot process of an MS-SMP data processing system. Because dedicated sideband channels are not used, the example implementations ofFIGS.8-10do not require specialized hardware or hardware updates to an existing system to be employed. The lack of sideband channels, however, leads to inter-socket communication latencies that are larger than the examples that utilize dedicated sideband channels.

FIG.8illustrates an example system800. System800may be implemented substantially as described in connection withFIG.1. In the example ofFIG.8, system800does not include any sideband channels or GSC106. Each SOC102may be considered to have a multi-processor core architecture. For purposes of illustration, each SOC102may have an architecture as illustrated in the example ofFIG.7, though the particular architecture implemented in each SOC102is not intended to be limiting.

In the examples ofFIGS.8-10, the SCP of the primary SOC102accesses the necessary resources in each individual socket of the MS-SMP data processing system. For the SCP of the primary SOC102to be able to access the system counter control registers and/or control logic for each socket of the MS-SMP data processing system, the system address map of the MS-SMP data processing system must expose the control and status registers of the MS-SMP data processing system to the SCP of the primary SOC102. The system address map is a data structure that specifies the address ranges of various memory mapped targets, e.g., components, circuit blocks, peripherals, etc., in the system. Further, during the boot sequence, the synchronization operation relies on certain hardware components being brought out of a reset state to allow the SCP of the primary SOC102to access such hardware components.

FIG.9illustrates an example method900depicting certain operative features of system800ofFIG.8, e.g., an MP-SMP data processing system that does not include sideband channels or a GSC. For purposes of illustration, SOC102-1is designated as the “primary” SOC in system800.FIG.10is a signal flow diagram illustrating example communications among SOCS102of system800ofFIG.8.FIGS.9-10illustrate example operations for synchronizing SSRS114within system800and may be performed as part of a boot process of system800. For purposes of illustration, SSRS114are considered to be system counters. Initially, as noted, SSRS114are free running and have different values as SSRS114did not have synchronized start times. The different values of SSRS114are reflected in the values stored in each shown as Y1for SSRS114-1, Y2for SSRS114-2, Y3for SSRS114-3, and YNfor SSRS114-N. In the examples ofFIGS.8-10, the events and/or signals conveyed are conveyed over bus104.

Referring toFIGS.9-10, in block902, the SCP of the primary SOC102(e.g., SCP302-1) issues a halt event to each SSR114. In the example ofFIG.10, SCP302-1issues a halt event1002to the SSR114in the same SOC, e.g., SSR114-1, and to the SSR114in each other SOC102. As part of block902, the SCP further reads the value of each SSR114once halted. In general, the values of the respective SSRS114may be determined with reference to SSR114-1in that each other SSR will have a value that is some amount of delay (D) after the value of SSR114-1. SCP302-1is capable of determining the delay for each SSR114in reference to SSR114-1.

In block904, the SCP of the primary SOC102updates the SSR in each other SOC102. The SCP does not update the SSR in the same SOC. Referring toFIG.10, SCP302-1leaves SSR114-1with the same value as initially read in block902. SCP302-1updates the value in each other SSR114-2,114-3,114-N by writing1004the value of SSR114-1plus the respective delay for the particular SSR114being updated.FIG.10illustrates the updated values written to each of SSRS114-2,114-3, and114-N by SCP302-1during block904.

In block906, the SCP of the primary SOC102sends an event signaling each of SSRS114to resume operation. In the example ofFIG.10, SCP302-1sends event1006to each SSR to resume operation. In signaling each SSR114to resume operation, the signal provided from SCP302-1to each respective SSR will incur the same delay as measured in block902.FIG.10illustrates that in block908, each SSR114is synchronized and free running. Each SSR114, as discussed, may be clocked using a common reference clock to remain synchronized with the other SSRS114.

In one or more example implementations, events such as halts, updates, and the like, as described herein and/or in connection withFIG.9, may be implemented as register writes, e.g., to a control register, associated with the resource being synchronized and conveyed over bus104. Register writes directed to the synchronized resources of the various sockets may be performed back-to-back in quick succession. In other example implementations, a dedicated broadcast functionality to convey an event to multiple sockets may be included such that individual writes to the synchronized resource in the various sockets need not be performed. Whether back-to-back writes or broadcast functionality is performed, delays in communicating with other sockets may be accounted for using the example techniques described herein.

Table 1 illustrates example states of the SSRS114, e.g., the system counters, at various points in time described inFIGS.9-10.

TABLE 1Socket 1Socket 2Socket 3(SOC(SOC(SOCSocket N102-1)102-2)102-3)(SOC 102-SystemSystemSystemN) SystemEventCounterCounterCounter. . .CounterOut of Reset & FreeX1X2X3. . .XNRunningSCP Issues HaltY1Y2y3. . .YNcommand to all systemcounters

In the example of Table 1, socket 1 corresponds to SOC102-1, e.g., the primary SOC. Initially, each system counter may be out of reset (e.g., corresponding to a boot process) and free running. Operation of the system counter is not synchronized as each has a different value. The SCP of the primary SOC issues the halt event. Once the system counters have halted, the SCP reads the value of each system counter and computes the difference between the values of the respective system counters and the value of the system counter in the primary socket. For a given counter in socket “N,” this difference is DN, where DN=Y1−YN. The value DNalso indicates the delay between halting the system counter of socket “N” and halting the system counter of socket 1.

The delay can be approximated as the latency incurred by the SCP in the primary socket to access the control registers and/or control logic of a particular SSR in a particular socket, e.g., a system counter in this case. Therefore, each system counter value can be re-written by the SCP such that the updated counter values compensate for this latency when the SCP unhalts each system counter. The SCP can re-write the system counter of socket “N” to Y1+DN. This ensures that when the SCP unhalts the system counters, the delay incurred in resuming the system counters will be negated by the adjustment made in the updated system counter values using the delay DN.

Table 2 illustrates example states of the SSRS114, e.g., the system counters, as the SCP of the primary socket overwrites the values of the system counters and issues the resume event (e.g., event1006).

TABLE 2Socket 1Socket 2Socket 3(SOC(SOC(SOCSocket N102-1)102-2)102-3)(SOC 102-SystemSystemSystemN) SystemEventCounterCounterCounter. . .CounterSCP re-programs systemY1Y1+ D2Y1+ D3. . .Y1+ DNcounters with updatedvaluesSCP configures systemY1Yet toYet to. . .Yet tocounters to resumeresumeresumeresumeAfter time D2, Socket 2Y1+ D2Y1+ D2Yet toYet tosystem counter resumesresumeresumeAfter time D3, Socket 3Y1+D3Y1+ D3Y1+ D3. . .Yet tosystem counter resumesresumeEventually, after time DN,Y1+ DNY1+ DNY1+ DN. . .Y1+ DNSocket N system counteris made to resume. Allcounters aresynchronized

Though the example implementations described within this disclosure are described in the context of MS-SMP data processing systems, it should be appreciated that the various synchronization mechanisms and techniques described herein may be extended to synchronize system resources of other data processing systems having multiple sockets that are not SMP data processing systems.

FIG.11illustrates an example method1100of synchronizing system resources in an MS data processing system (system). The system may be an SMP data processing system.

In block1102, a primary SOC of the system is capable of providing a trigger event to a GSC106. The primary SOC is one of a plurality of SOCS and the trigger event is provided over a first sideband channel108-1. In block1104, in response to the trigger event, the GSC106is capable of broadcasting a synchronization event to the plurality of SOCS over a second sideband channel110. In block1106, in response to the synchronization event, the system resource in each SOC of the plurality of SOCS of the system is programmed with a common value. The programming synchronizes the system resources of the plurality of SOCS.

The foregoing and other implementations can each optionally include one or more of the following features, alone or in combination. Some example implementations include all the following features in combination.

In one aspect, the operations described, e.g., the providing, the broadcasting, and the programming, are performed as part of a boot process for an MS data processing system including the plurality of SOCS. The operations may be performed in response to the system resources (e.g., SSRS114) coming out of a reset state.

In another aspect, the plurality of SOCS are identical.

In another aspect, each SOC of the plurality of SOCS includes a plurality of processor cores. A selected processor core of the plurality of processor cores of a selected SOC of the plurality of SOCS provides the trigger event. For example, an SCP as described herein may provide the trigger event.

In another aspect, each SOC of the plurality of SOCS includes a plurality of processor cores. The synchronization event may be an interrupt. A selected processor core of each SOC of the plurality of SOCS executes an interrupt service routine in response to the interrupt to program the system resource located in the same SOC. The selected processor may be the SCP in the primary SOC and a SRRP in the non-primary SOCS.

In another aspect, the selected processor core within each SOC of the plurality of SOCS, in executing the interrupt service routine, writes to a control register and/or control logic of the system resource within the same SOC.

In another aspect, within each SOC of the plurality of SOCS, the synchronization event is received by control logic of the system resource. In response to the synchronization event, the control logic of the system resource in each respective SOC resets the system resource within the same SOC.

FIG.12illustrates an example method1200of synchronizing system resources in an MS data processing system (system). The system may be an SMP data processing system. In the example ofFIG.12, the system does not include sideband channels and does not include a GSC106.

In block1202, the primary SOC is capable of halting each of a plurality of system resources in the system. Each system resource is located in a different SOC of a plurality of SOCS of a multi-SOC system. The primary SOC is one of the plurality of SOCS. In block1204, the primary SOC is capable of writing an updated value to the system resource of each other SOC of the plurality of SOCS while halted. In block1206, the primary SOC is capable of initiating (unhalting) operation of the system resource in each SOC of the of the plurality of SOCS subsequent to the writing.

The foregoing and other implementations can each optionally include one or more of the following features, alone or in combination. Some example implementations include all the following features in combination.

In one aspect, subsequent to the halting and prior to the writing, the primary SOC, e.g., the SCP of the primary SOC, is capable of reading a value from each of the plurality of system resources. For each SOC of the plurality of SOCS other than the primary SOC, the SCP is capable of determining a difference between a value read from the system resource of the primary SOC and the value read from the system resource of the other SOC. The writing includes the primary SOC, e.g., the SCP of the primary SOC, writing the updated value to each of the other SOCS. The updated value written to each of the other SOCS depends on the difference between the value read from the system resource of the primary SOC and the value read from the system resource of the other SOC.

In another aspect, the writing the updated value to each system resource of the other SOCS and the initiating operation accounts for latency in the primary SOC communicating with the respective ones of the plurality of system resources in the respective other SOCS.

In another aspect, the plurality of SOCS are identical.

In another aspect, the halting, writing, and initiating are conveyed as part of a boot process. The operations may be performed in response to the system resources (e.g., SSRS114) coming out of a reset state.

In another aspect, communication between the primary SOC and the other SOCS is conveyed over a communication bus communicatively linking the plurality of SOCS.

FIG.13illustrates an example implementation of an MS data processing system1300. Data processing system1300further may be an SMP type of system. The components of data processing system1300can include, but are not limited to, a plurality of SOCS1302, a memory1304, and a bus1306that couples various system components including memory1304to SOCS1302. SOCS1302may include processor cores having any of a variety of different architectures including, but not limited to, x86 type of architecture (IA-32, IA-64, etc.), a Power Architecture, ARM processors, and the like. As noted, in some cases, SOCS1302may include programmable logic that may be used to implement one or more of the plurality of processor cores of the SOCS.

As discussed, in some cases, MS data processing system may include a central processing unit (CPU) and include the plurality of SOCS1302as accelerators. The accelerators may be disposed on one or more different circuit boards, cards, chassis structures, or implemented in other available form factors that communicatively link with data processing system1300, for example, via a communication bus such as bus1306. In one or more example implementations, the accelerators may be implemented in accordance with any of the various standards and/or specifications set forth as part of the Open Compute Project (OCP) and/or OCP Accelerator Module (OAM) specifications. Further, it should be appreciated that any such accelerators may include any of a variety of different connectors and/or combinations of connectors for coupling to different systems and/or devices using one or more different communications protocols.

Bus1306represents one or more of any of a variety of communication bus structures. By way of example, and not limitation, bus1306may be implemented as a Peripheral Component Interconnect Express (PCIe) bus. Other communication busses may be used in lieu of or in addition to PCIe. Data processing system1300typically includes a variety of computer system readable media. Such media may include computer-readable volatile and non-volatile media and computer-readable removable and non-removable media.

Memory1304can include computer-readable media in the form of volatile memory, such as random-access memory (RAM)1308and/or cache memory1310. Data processing system1300also can include other removable/non-removable, volatile/non-volatile computer storage media. By way of example, storage system1312can be provided for reading from and writing to a non-removable, non-volatile magnetic and/or solid-state media (not shown and typically called a “hard drive”). Although not shown, a magnetic disk drive for reading from and writing to a removable, non-volatile magnetic disk (e.g., a “floppy disk”), and an optical disk drive for reading from or writing to a removable, non-volatile optical disk such as a CD-ROM, DVD-ROM or other optical media can be provided. In such instances, each can be connected to bus1306by one or more data media interfaces. Memory1304is an example of at least one computer program product.

Program/utility1314, having a set (at least one) of program modules1316, may be stored in memory1304. Program/utility1314is executable by processor cores of SOCS1302. By way of example, program modules1316may represent an operating system, one or more application programs, other program modules, and program data. Program modules1316, upon execution, cause data processing system1300, e.g., one or more CPUs (not shown) and/or SOCS1302, to carry out the functions and/or methodologies of the example implementations described within this disclosure. Program/utility1314and any data items used, generated, and/or operated upon by data processing system1300are functional data structures that impart functionality when employed by data processing system1300.

For example, in one or more example implementations, SOCS1302may be implemented as described in connection withFIG.1and include a GSC (not shown). In one or more other example implementations, SOCS1302may be implemented as described in connection withFIG.8.

Data processing system1300may include one or more Input/Output (I/O) interfaces1318communicatively linked to bus1306. I/O interface(s)1318allow data processing system1300to communicate with one or more external devices1320and/or communicate over one or more networks such as a local area network (LAN), a wide area network (WAN), and/or a public network (e.g., the Internet). Examples of I/O interfaces1318may include, but are not limited to, network cards, modems, network adapters, hardware controllers, etc. Examples of external devices also may include devices that allow a user to interact with data processing system1300(e.g., a display, a keyboard, and/or a pointing device) and/or other devices such as accelerator card.

Data processing system1300is only one example implementation. Data processing system1300can be practiced as a standalone device (e.g., as a user computing device or a server, as a bare metal server), in a cluster (e.g., two or more interconnected computers), or in a distributed cloud computing environment (e.g., as a cloud computing node) where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media including memory storage devices. The example ofFIG.13is not intended to suggest any limitation as to the scope of use or functionality of example implementations described herein. A data processing system is an example of computer hardware that is capable of performing the various operations described within this disclosure.

Data processing system1300may include fewer components than shown or additional components not illustrated inFIG.13depending upon the particular type of device and/or system that is implemented. The particular operating system and/or application(s) included may vary according to device and/or system type as may the types of I/O devices included. Further, one or more of the illustrative components may be incorporated into, or otherwise form a portion of, another component. For example, a processor may include at least some memory.

Data processing system1300may be operational with numerous other general-purpose or special-purpose computing system environments or configurations. Examples of computing systems, environments, and/or configurations that may be suitable for use with data processing system1300include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices, and the like.

Some computing environments, e.g., cloud computing environments and/or edge computing environments using data processing system1300or other suitable data processing system, generally support the FPGA-as-a-Service (FaaS) model. In the FaaS model, user functions are hardware accelerated as circuit designs implemented within programmable ICs operating under control of the (host) data processing system. Other examples of cloud computing models are described in the National Institute of Standards and Technology (NIST) and, more particularly, the Information Technology Laboratory of NIST.

While the disclosure concludes with claims defining novel features, it is believed that the various features described within this disclosure will be better understood from a consideration of the description in conjunction with the drawings. The process(es), machine(s), manufacture(s) and any variations thereof described herein are provided for purposes of illustration. Specific structural and functional details described within this disclosure are not to be interpreted as limiting, but merely as a basis for the claims and as a representative basis for teaching one skilled in the art to variously employ the features described in virtually any appropriately detailed structure. Further, the terms and phrases used within this disclosure are not intended to be limiting, but rather to provide an understandable description of the features described.

For purposes of simplicity and clarity of illustration, elements shown in the figures have not necessarily been drawn to scale. For example, the dimensions of some of the elements may be exaggerated relative to other elements for clarity. Further, where considered appropriate, reference numbers are repeated among the figures to indicate corresponding, analogous, or like features.

As defined herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise.

As defined herein, the term “approximately” means nearly correct or exact, close in value or amount but not precise. For example, the term “approximately” may mean that the recited characteristic, parameter, or value is within a predetermined amount of the exact characteristic, parameter, or value.

As defined herein, the terms “at least one,” “one or more,” and “and/or,” are open-ended expressions that are both conjunctive and disjunctive in operation unless explicitly stated otherwise. For example, each of the expressions “at least one of A, B, and C,” “at least one of A, B, or C,” “one or more of A, B, and C,” “one or more of A, B, or C,” and “A, B, and/or C” means A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B and C together.

As defined herein, the term “automatically” means without human intervention. As defined herein, the term “user” means a human being.

As used herein, the term “cloud computing” refers to a computing model that facilitates convenient, on-demand network access to a shared pool of configurable computing resources such as networks, servers, storage, applications, ICs (e.g., programmable ICs) and/or services. These computing resources may be rapidly provisioned and released with minimal management effort or service provider interaction. Cloud computing promotes availability and may be characterized by on-demand self-service, broad network access, resource pooling, rapid elasticity, and measured service.

As defined herein, the term “computer readable storage medium” means a storage medium that contains or stores program code for use by or in connection with an instruction execution system, apparatus, or device. As defined herein, a “computer readable storage medium” is not a transitory, propagating signal per se. A computer readable storage medium may be, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. The various forms of memory, as described herein, are examples of computer readable storage media. A non-exhaustive list of more specific examples of a computer readable storage medium may include: a portable computer diskette, a hard disk, a RAM, a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an electronically erasable programmable read-only memory (EEPROM), a static random-access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, or the like.

As defined within this disclosure, the term “data structure” means a physical implementation of a data model's organization of data within a physical memory. As such, a data structure is formed of specific electrical or magnetic structural elements in a memory. A data structure imposes physical organization on the data stored in the memory as used by an application program executed using a processor.

As defined herein, the term “if” means “when” or “upon” or “in response to” or “responsive to,” depending upon the context. Thus, the phrase “if it is determined” or “if [a stated condition or event] is detected” may be construed to mean “upon determining” or “in response to determining” or “upon detecting [the stated condition or event]” or “in response to detecting [the stated condition or event]” or “responsive to detecting [the stated condition or event]” depending on the context.

As defined herein, the term “responsive to” and similar language as described above, e.g., “if,” “when,” or “upon,” means responding or reacting readily to an action or event. The response or reaction is performed automatically. Thus, if a second action is performed “responsive to” a first action, there is a causal relationship between an occurrence of the first action and an occurrence of the second action. The term “responsive to” indicates the causal relationship.

As defined herein, “data processing system” means one or more hardware systems configured to process data, each hardware system including at least one processor programmed to initiate operations and memory.

As defined herein, the term “processor” or “processor core” means at least one circuit capable of carrying out instructions contained in program code. The circuit may be an integrated circuit or embedded in an integrated circuit. An SOC, as described herein, includes a plurality of processor cores and, in some cases, may be referred to as a “processor.”

As defined herein, the term “soft” in reference to a circuit means that the circuit is implemented in programmable logic or programmable circuitry. Thus, a “soft processor” means at least one circuit implemented in programmable circuitry that is capable of carrying out instructions contained in program code.

As defined herein, the term “output” means storing in physical memory elements, e.g., devices, writing to display or other peripheral output device, sending or transmitting to another system, exporting, or the like.

As defined herein, the term “substantially” means that the recited characteristic, parameter, or value need not be achieved exactly, but that deviations or variations, including for example, tolerances, measurement error, measurement accuracy limitations, and other factors known to those of skill in the art, may occur in amounts that do not preclude the effect the characteristic was intended to provide.

The terms first, second, etc., may be used herein to describe various elements. These elements should not be limited by these terms, as these terms are only used to distinguish one element from another unless stated otherwise or the context clearly indicates otherwise.

A computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the inventive arrangements described herein. Within this disclosure, the term “program code” is used interchangeably with the term “computer readable program instructions.” Computer readable program instructions described herein may be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a LAN, a WAN and/or a wireless network. The network may include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge devices including edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.

Computer readable program instructions for carrying out operations for the inventive arrangements described herein may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, or either source code or object code written in any combination of one or more programming languages, including an object-oriented programming language and/or procedural programming languages. Computer readable program instructions may include state-setting data. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a LAN or a WAN, or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some cases, electronic circuitry including, for example, programmable logic circuitry, an FPGA, or a PLA may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the inventive arrangements described herein.

Certain aspects of the inventive arrangements are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, may be implemented by computer readable program instructions, e.g., program code.

These computer readable program instructions may be provided to a processor of a computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the operations specified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operations to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various aspects of the inventive arrangements. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified operations.

In some alternative implementations, the operations noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. In other examples, blocks may be performed generally in increasing numeric order while in still other examples, one or more blocks may be performed in varying order with the results being stored and utilized in subsequent or other blocks that do not immediately follow. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, may be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.