System interconnect dynamic scaling by predicting I/O requirements

Interface management techniques provide reduced power consumption along with reducing heat and EMI generation in a computer system having multiple interconnected processing units. Physical link layers of external interfaces that interconnect the processing units of have dynamically adjustable bandwidth and the bandwidths are dynamically adjusted by predicting interface bandwidth requirements. An interface controller detects events other than I/O requests that occur in a processing unit that are indicators of potential future transactions on one of the external interfaces connected to the processing unit. The interface controller predicts, from the detected events, that future transactions will likely occur on the interface, and in response, controls the dynamically adjustable bandwidth of physical link layer of the interface to accommodate the future transactions.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates generally to interconnected processing systems, and more particularly, to processing systems that dynamically control I/O interface performance based on a prediction of I/O requirements.

2. Description of Related Art

Interfaces within and between present-day integrated circuits have increased in operating frequency and width. In particular, in multiprocessing systems, both wide and fast connections are provided between many processing units. Data width directly affects the speed of data transmission between systems components, as does the data rate, which is limited by the maximum frequency that can be supported by an interface. However, such fast and wide interconnects are significant power consumers in a computer system formed from interconnected processing units.

The processing units in a multi-processing system, particularly a symmetric multi-processing system (SMP) may need to communicate at any time, since, for example, when close affinity is provided between processors, a processor might access memory that is located on a remote node, but that is otherwise available in the processor's memory space. Therefore, for the above and other reasons, present-day multi-processing systems typically keep the physical layer of the interfaces operational and cycle idle data patters on the interconnects in order to maintain calibration of the links when transmissions are not being made over the interface physical layer. However, such operation wastes power, generates heat, and raises background noise levels (electromagnetic emissions) in the system. The alternative of placing the interface physical layers in a power-managed state would lead to unacceptable latency for transactions.

It is therefore desirable to provide an interface and computer system that more effectively manage the state of interface physical link layers in a multi-processing system to reduce power consumption and background noise levels.

BRIEF SUMMARY OF THE INVENTION

The above-mentioned objective of providing improved performance and/or power efficiency of a system interconnect physical layer between processing units is provided in a a computer system, a computer program product and an interface controller.

The computer system, computer program product and interface controller manage the state of a physical link layer of external interfaces that interconnect processing units of the computer system. The physical link layers have dynamically adjustable bandwidth. The method detects events other than I/O requests that occur in a processing unit that are indicators of potential future transactions on one of the external interfaces connected to the processing unit. The method predicts, from the detected events, that future transactions will likely occur on the interface, and in response, controls the dynamically adjustable bandwidth of physical link layer of the interface to accommodate the future transactions by increasing the dynamically adjustable bandwidth of the first physical link layer interface. After the future transactions have occurred, the dynamically adjustable bandwidth of first physical link layer is restored to a lower value.

DETAILED DESCRIPTION OF THE INVENTION

The present invention encompasses techniques for controlling the bandwidth, including the width and/or frequency of links, such as parallel busses or serial connections, that interconnect processing units in a processing system. Non I/O (input/output) transaction events occurring within the processing units are used to predict when I/O transactions are likely to occur over the links and the prediction is used to control the bandwidth of the links to accommodate the predicted transactions. The techniques thus can reduce power consumption and radiated emissions by maintaining the links in a lower power or inactive state between use.

With reference now to the figures, and in particular with reference toFIG. 1a distributed computer system in accordance with an embodiment of the present invention is shown. A first processing unit10A includes a processor core12coupled to a memory14that stores program instructions for execution by processor12. The program instructions may include program instructions forming computer program products that perform portions of the techniques disclosed herein within processing units10A-10D. Processing unit10A also includes a network interface (NWI)16that couples processing unit10A to interface links11, which are wired or wireless links to other processing units10B,10C, and provide for access between processing unit10A and resources such as remote memory14A within processing unit10B. Links11have dynamically adjustable bandwidth/power consumption, which is controlled as disclosed below. Other processing units10B-10D are of identical construction in the exemplary embodiment, but embodiments of the invention may be practiced in asymmetric distributed systems having processing units with differing features. The distributed computer system ofFIG. 1also includes other resources such as I/O devices19, including graphical display devices, printers, scanners, keyboards, mice, which may be coupled to the links11or one of nodes10A-10D. Processing units10A-10D are also coupled to storage devices18, for storing and retrieving data and program instructions, such as storing computer program products in accordance with an embodiment of the invention.

Referring now toFIG. 2, details within a processing unit10that can be used to implement processing units10A-10D are shown. Within processing unit, controllers30A,30B are shown to illustrate two possible locations of a controller that manages the bandwidth of a physical link layer24of interface11according to one or more control signals bw. Within one or more of core12, memory14and network interface16, logic, control logic detects events that are indicative of future external bus transactions that are likely to be issued over interface11. For example, a controller30A within core12might detect that certain instructions are being executed, or memory ranges are being read or written, that correspond to operations that will generate I/O transactions over interface11. For example, controller30A may detect that a direct-memory access (DMA) buffer is being allocated, or a DMA channel being initialized in bus I/O unit20or elsewhere within processing unit10for transfer to buffers21that supply data to, or receive data from, a logical link layer22of network interface16. Controller30A may be coupled to one or more trace array units13within core12to capture state information that is indicative of the events, and use the state information contained in the trace array to provide detected events as input for predicting a required bandwidth of interface11in the near future. System level events such as a hypervisor executing within processing unit10starting a thread with an association to remote memory, or the association of remote memory to a running thread can be used to predict and trigger an increase in link bandwidth between the core on which the thread is running and the location of the remote memory, so that when the inevitable memory accesses by the thread occur, the link is operating at sufficient bandwidth. Similarly, a controller30B within arbiter26of logical link layer22may detect that the logical link layer22, and thus interface11is being arbitrated for and therefore physical link layer24will soon need to be active for a number of transactions. In another example, controller30B may count idle cycles of logical link layer22to determine a required bandwidth for physical link layer24. Alternatively, or in combination, controller30B within network interface16(whether or not within arbiter26) might also be connected to detect activity in buffers21with write operations anticipating upcoming output operations, or initialization of the buffer indicating a future read transaction that will be commanded by core12or another actor within processing unit10.

Processing unit10ofFIG. 2is used to illustrate control of one of links11between two of processing units10A-10D, but the techniques of the present invention extend to connection of memories, peripherals and other functional units within a computer system or other electronic device and are not to be construed as limiting as to the particular system in which they are implemented. Links11between processing units10A-10D are, in the example, made by a uni-directional physical layer interconnect of wired signals connected between processing units10A-10D, however, the techniques of the present invention extend to non-physically connected (wireless) interfaces having multiple datapaths and to bi-directional interfaces, as well. In order to support the adjustable bandwidth of links11, processing units10A-10D may include elastic interface (EI) units with adjustable operating frequency and/or selectable width as described in detail in U.S. Pat. No. 8,050,174 entitled “SELF HEALING CHIP-TO-CHIP INTERFACE”, U.S. Pat. No. 7,117,126 entitled “DATA PROCESSING SYSTEM AND METHOD WITH DYNAMIC IDLE FOR TUNABLE INTERFACE CALIBRATION” and in U.S. Pat. No. 7,080,288 entitled “METHOD AND APPARATUS FOR INTERFACE FAILURE SURVIVABILITY USING ERROR CORRECTION.” The disclosures of the above-referenced U.S. Patents are incorporated herein by reference.

Referring now toFIG. 3, details of a controller30that may be used to detect events and predict future transactions on a physical layer of interface11is shown. Controller30may, for example, implement controller30A within core12as shown inFIG. 2. Controller30is also provided only as one example of an architecture that may be implemented in discrete logic, for example as a state machine, or may be implemented in firmware or software as program instructions executed by core12or another processor within processing unit10, such as a core within logical link layer22or a service processor coupled to core12. As an example of a mechanism for detecting events, a bus snooper31observes transactions on an internal or external bus of core12, such as a bus that couples core12to memory14. In another example a hypervisor34reports thread state change or remote memory association events, such as the above-described connection between a thread executing within processing unit10and a remote memory. When an event detector32A detects that a combination of events indicates a likelihood that a number of transactions will soon occur over interface11, a counter35A in prediction unit34is incremented. Similarly, another event detector32B receives indications of activity at logical link layer22and determines whether to increment another counter35B based on whether the activity indicates that a number of transactions will occur over interface11. A bandwidth profile calculator33determines from the values of counters35A and35B, which may be periodically reset, or reset according to another mechanism, the bandwidth that is likely needed over interface11. Bandwidth profile calculator33provides a control signal to a physical link layer bandwidth control circuit36that sets the operating frequency and/or width of the physical link layer of interface11appropriately to balance power consumption (or generated noise, etc., depending on the particular system criteria) with the bandwidth supplied over interface11for the transactions. A timer37is provided to restore the bandwidth to an initial value after a predetermined or programmable interval. In one exemplary implementation, timer37controls a time between intervals of full-bandwidth or partial-bandwidth operation as commanded by bandwidth profile calculator33and a low-power shutdown state. The width of the intervals can also be set by bandwidth profile calculator, so that interface11is cycled between the low-power state and the full-bandwidth or partial-bandwidth state in order to complete transactions that are allowed to accumulate in buffers21between the intervals of full-bandwidth or partial-bandwidth operation. In all of the cases above, the actual demand generated by I/O requests is generally combined with the predicted demand to determine an appropriate link bandwidth.

Referring now toFIG. 4, a method of operating a processing system is illustrated in a flowchart. First, interface links between processing units are initialized and calibrated at a nominal interface width and frequency (step50). During operation, events are detected that indicate I/O is likely to occur over one or more of the links (step51). The events are logically combined and counted to generate predictors that indicate a bandwidth that will be needed for the one or more links (step52). Once the predictor is over a threshold value (decision53) or the link utilization is over a threshold value (decision54), the bandwidth of the physical layer (PHY) is raised for a predetermined time period (step55). After the predetermined time period has elapsed (decision56) the bandwidth of the physical layer is lowered to the previous bandwidth (step57). Until the scheme is ended or the system is shut down (decision58), steps51-57are repeated.

As noted above, portions of the present invention may be embodied in a computer program product, e.g., a program executed processors having program instructions that direct the operations outlined inFIG. 4, by controlling the interfaces ofFIG. 2andFIG. 3. The computer program product may include firmware, an image in system memory or another memory/cache, or stored on a fixed or re-writable media such as an optical disc having computer-readable code stored thereon. Any combination of one or more computer readable medium(s) may store a program in accordance with an embodiment of the invention. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

In the context of the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely-propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses propagating through a fiber-optic cable), or electrical signals transmitted through a wire. A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable signal medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.