Managing latencies in accessing memory of computer systems

The present invention, in various embodiments, provides techniques for managing latencies in accessing memory of computer systems. In one embodiment, upon accessing the memory system for a piece of data used by a first process, a latency manager determines the access time to acquire the piece of data in the memory system. The latency manager then compares the determined access time to a threshold. If the determined access time is greater than the threshold, the latency manager triggers an interrupt for the operating system to switch threads or processes so that execution of the first process is postponed and execution of a second process starts. Various embodiments include the latency manager is polled for the access time when the processor is stalled, the latency manager triggers a process switch when a particular memory subsystem is accessed, etc.

FIELD OF THE INVENTION

The present invention relates generally to managing computer memory systems and, more specifically, to managing latencies in memory accesses.

BACKGROUND OF THE INVENTION

Currently, in various situations, a processor or operating system accessing memory does not know the access time, e.g., the time it takes to acquire the data from the memory system. Processor time and other resources can be wasted due to this memory access time because during this time the processor is idled waiting for the access data. Further, the access time can be very long such as in case of a memory page miss in which a slow device like a hard disc is accessed. In some approaches, a process seeking the access data keeps waiting for the data until the allocated wait time runs out, at that time the process is put in the background.

Based on the foregoing, it is clearly desirable that mechanisms be provided to solve the above deficiencies and related problems.

SUMMARY OF THE INVENTION

The present invention, in various embodiments, provides techniques for managing latencies in accessing memory of computer systems. In one embodiment, upon accessing the memory system for a piece of data used by a first process, a latency manager determines the access time to acquire the piece of data in the memory system. The latency manager then compares the determined access time to a threshold. If the determined access time is greater than the threshold, the latency manager notifies the operating system to switch threads or processes so that execution of the first process is postponed and execution of a second process starts. Various embodiments include the latency manager is polled for the access time when the processor is stalled, the latency manager triggers a process switch when a particular memory subsystem is accessed, etc.

DETAILED DESCRIPTION OF THE VARIOUS EMBODIMENTS

In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid obscuring the invention.

Hardware Overview

FIG. 1shows a processor system100upon which techniques of the invention may be implemented. System100includes, in relevant part, a central processing unit (CPU)102, a memory system104, and a hard disc130. CPU102in turns includes a processor105and cache memory108, while memory system104includes a memory controller110, level-1 memory120, and level-2 memory125. Memory system104is commonly referred to as main memory from which program instructions are executed and program data are manipulated. Memory controller110includes latency manager112. Level-2 memory125is shown outside of system104to illustrate that accessing level-2 memory125takes relatively longer than accessing level-1 memory120. System100normally runs by an operating system170resided in level-1 memory120. Processor105, cache memory108, memory controller110, level-1 memory120, level-2 memory125, hard disc130, and operating system170are common computer components. Each of cache108, level-1 memory120, level-2 memory125, and hard disc130may be referred to as a memory subsystem since each stores data for system100.

In one embodiment, data in system100is accessed in a specific order, such as, from a fast memory subsystem (e.g., cache) to a slow memory subsystem (e.g., hard disc, personal computer memory card international association (PCMCIA) card, etc). However, techniques of the invention are not limited to that order, but are applicable in any other order such as a random order, an order independent from the access time of the memory subsystems, a non-sequential order, e.g., an order in which one subsystem is not necessarily always followed by the same subsystem, etc. For illustration purposes, upon a memory access for a piece of data, the data is accessed (searched) in the order of cache108, level-1 memory120, level-2 memory125, and hard disc130. If the data is missed (e.g., not found) in cache108, then it is searched in level-1 memory120. If it is missed in level-1 memory120, then it is searched in level-2 memory125. If it is missed in level-2 memory125, then it is searched in hard disc130, etc. Generally, a time to access each memory subsystem ranges from a minimum time tminto a maximum time tmax, and any time between this tminto tmaxrange, including an average time tave, can be used as an access time for that subsystem. Selecting a time, e.g., tmin, tave, or tmaxas an access time for a memory subsystem varies depending on various factors including the goals and priorities of system designers designing the system. If the time to access cache108, level-1 memory120, level-2 memory125, and hard disc130is designated as times t1, t2, t3, and t4, then, in one embodiment, times t1, t2, t3, and t4increase in that order, i.e., t4>t3>t2>t1.

In this document, the configuration of system100inFIG. 1is used only as an example; the techniques disclosed herein may be implemented in other configurations of a processing system. For example, cache108can be part of processor105,CPU102, memory system104, etc; there may be more than one processor105in CPU102and/or more than one CPU102; there may be various levels of cache, memory, hard disc, and other storage devices that constitute memory system104; memory latency manager112may be in CPU102's instruction fetch unit, load/store unit, bus interface, main memory controller, or other locations for latency manager112to acquire or estimate enough information to determine the access time for a piece of data.

Latency Manager

In theFIG. 1example, latency manager112resides in memory controller110. However, in general, latency manger112is placed in the data path of the access data, e.g., between processor105and the memory subsystems that store the data, including, for example, cache108, level-1 memory120, level-2 memory, hard disc130, memory in PCMCIA cards, etc. Placing latency manager112in the data path is beneficial because latency manager112may efficiently acquires the data access time. In embodiments where latency manger112is not in the access data path, additional communications, such as messages or signals, are usually implemented to communicate the latencies with latency manager112.

In one embodiment, latency manager112stores a latency threshold tthand, upon a memory access for a piece of data of a first process, latency manager112determines the access time to acquire that particular piece of data. Latency manager112then compares the determined access time to threshold tth. If the determined access time is greater than threshold tth, then latency manager112provides that information to an appropriate intelligence to take appropriate actions. The intelligence could be any intelligent logic including hardware, software, firmware, such as CPU102, processor105, operating system107, software running on the processor, hardware or software managing the memory system, etc. In one embodiment, latency manager112provides the information to processor105and/or operating system170for them to take actions, such as to cause a performance monitor of memory subsystem104or of system100as a whole, to postpone execution of the first process/thread, to cause process switches, etc. Latency manager112may also directly cause such actions to be performed. In one embodiment and for illustration purposes, latency manager112's action triggers a process switch so that execution of the first process is postponed and execution of a second process may start. In one embodiment, latency manager112puts the first process in a sleeping queue or schedules that process out until it is ready to be executed again. In an alternative embodiment, latency manager112notifies or triggers an interrupt to operating system170. As soon as operating system170recognizes the reason for interrupt, operating system170responds by switching out the currently executing process to postpone execution of this process and allows execution of another process. However, if the determined access time is less than or equal to threshold tth, then latency manager112takes no special actions, e.g., allows system100to function as usual. In the above situations, postponing executing the first process prevents wasting resources due to waiting for the access data. As an example, if t4>t3>tth>t2>t1, then in response to a memory access in cache108or in level-1 memory120, latency manager112does not take special actions. However, in response to a memory access in level-2 memory125or in hard disc130, latency manager112's actions initiate a process switch. Those skilled in the art will recognize that a process is an executing program and may be used loosely as a “task.” Further, a thread is a part of a program that can execute independent of other parts. For illustration purposes, the term “process” used in this document refers to a process, a program, a thread, or their equivalences.

In one embodiment, latency manager112's action causes a process switch when the access data is missed in a predetermined memory subsystem. This is conveniently implemented when the data is accessed in an order from a faster subsystem to a slower subsystem in which as the data is missed in one subsystem, the data is searched in a next slower subsystem up to a point where accessing a too slow subsystem would waste too much processor idle time. For example, let the search be in the order of cache108, level-1 memory120, level-2 memory125, and hard disc130, and it has been determined that accessing level-2 memory125takes too long for processor105to wait, then level-1 memory120is “earmarked,” such that when the access data is missed in level-1 memory subsystem120, a process switch is triggered. In this document, a “slower” subsystem has a longer access time while a “faster” subsystem has a shorter access time.

In one embodiment, upon a memory access, processor105continues performing its functions until it is stalled, such as when processor105completes its instruction queue and has no other instructions to execute. Processor105then polls latency manager112or other appropriate intelligence to determine the time it takes to complete the memory access. If the time taken to complete the memory access is greater than a predetermined time, e.g., the threshold tth, then a process switch is triggered. In this embodiment, processor time to poll latency manager112does not add costs to system100because processor105, being stalled and thus idled, would not otherwise execute any beneficial instructions.

In one embodiment, a counter is used to determine whether to switch processes. In general, latency manager112knows whether a data access is about to occur, and, as soon as the data access starts, latency manager112enables the counter to count the time elapsed from the time the data access starts. This counted time thus keeps increasing as the data is being accessed in memory system104. When the counted time increases greater than threshold tth, a switch process is triggered. For illustrative purposes, the access time for cache108, level-1 memory120, and level-2 memory125is 10, 100, and 300 time units, respectively. Further, it is determined that a data access to level-2 memory125will cause a process switch. As soon as the counter counts past 100, which is the maximum access time for level-1 memory120, and which is also the latency threshold tth, latency manager112triggers a process switch. In one embodiment, latency manager112is placed in processor105's outstanding memory access buffer (not shown), and each buffer includes a counter to keep track of how long a memory access has been outstanding. If one or more counters exceed latency threshold tth, then a process switch is triggered.

In various embodiments, a memory access may result in searching for the same data in multiple places, which is commonly referred to as parallel access since the same access is sent to different memory subsystems, e.g., to both a faster subsystem and a slower subsystem. Parallel memory access does not add cost to the system, but saves time in accessing the slower subsystem when the data is missed in the faster subsystem because the data is searched in the slower subsystem in parallel with searching in the faster subsystem. In one embodiment, the initial access time is that of the faster subsystem, and, when the access misses in the faster subsystem, the access time is that of the slower subsystem.

A multiple memory access occurs in case of accessing multiple pieces of data. Normally, while the first access is in progress, the second access starts. In one embodiment, upon a multiple memory access, latency manger112uses the longest access time to determine a process switch. In such situations, for comparison to threshold tth, latency manager112is updated with a new access latency if this access latency is greater than the last latency stored in latency manager112.

Alternatively, latency manager112is loaded with a new access latency upon each miss in a memory subsystem. For example, before the first memory access, latency manager112is loaded with t1, e.g., the access time for cache108. Latency manager112may also be initialized with a value 0 because, in one embodiment, as long as the predicted access time is less than tth, latency manager112does not take special actions. When the memory access misses in cache108, latency manager112is loaded with access time t2of level-1 memory120. When the memory access misses in level-1 memory120, latency manager112is loaded with access time t3of level-2 memory125, and when the memory access misses in level-2 memory125, latency manager112is loaded with access time t4of hard disc130, etc.

Variations

FIG. 2shows a system200upon which techniques of the invention may be implemented. System200is described in details in copending application Ser. No. 09/896043, of which this application is a continuation-in-part (above). In this system200, cache208, physical memory220, swap memory228, hard disc230, and their equivalences are considered memory subsystems. Each memory subsystem corresponds to an access time, e.g., time tt1, tt2, tt3, tt4for cache208, physical memory220, swap memory228, and hard disc230, respectively.

A latency manager, e.g., latency manager212(not shown), which is comparable to latency manager112in system100, may be implemented in system200. Latency manager212works by itself or with memory manager265and/or memory table268to perform latency manager212's functions consistent with the techniques disclosed herein. In one embodiment, latency manager212is advantageously resided in memory manager265or memory table268because both of them are usually in the data path and contain information related to the access data, the access times to the memory subsystems, etc.

In one embodiment, it is predetermined that accessing some particular subsystems that have a long access time, e.g., swap memory228, hard disc230, etc., would cause a process switch. In this embodiment, latency manager212includes information, such as logical bits, to determine whether such a process switch is desirable. For example, each memory subsystem corresponds to a bit, and the bits corresponding to cache208and physical memory220are set to a logical zero to indicate that accessing data in these memory subsystems do not cause a process switch. Similarly, the bits corresponding to swap memory228and hard disc230are set to a logical one to indicate that accessing data in these memory subsystems do cause a process switch. When it is determined that a subsystem of memory system204is about to be accessed, latency manager212reviews the bit corresponding to that subsystem to determine a process switch. For example, if the bit is at a logical high, then latency manager212sends a signal that can trigger a process switch; otherwise, no special action is desirable.

Techniques of the invention are advantageously used in system200because memory system204, with the implementation of memory manger265, in various embodiments manages its memory subsystems independent of processor205and operating system270. As a result, access times to memory subsystems of memory system204, e.g., tt1, tt2, tt3, tt4, etc., are further hidden from processor205and operating system270. This can cause long idle processor time. Latency manager212, working with memory manger265and memory table268can provide relevant access times to processor205and operating system270. Together, they make appropriate decisions, e.g., switching processes to prevent wasting idle processor time.

Other Considerations

Causes for process switches and trigger threshold tthvary depending on various factors. For example, different types of memory subsystems or different types of instructions may have different trigger thresholds. In one embodiment, threshold tthis greater than the time to access level-1 memory and cache subsystems. This threshold tthis determined based on various factors such as what is a realistic time for a memory access, the cost of switching the processes, the cost of wasting idle processor time, etc. A particular instruction may or may not trigger a process switch. In one embodiment, a store instruction, e.g., writing data to memory, does not cause a process switch because the processor does not wait for the results of such an instruction. Consequently, threshold tthis set to a very high value so that no process switch will occur. Conversely, a load instruction, e.g., getting data from memory, can cause a process switch, and thus the trigger threshold tth, can be set accordingly, e.g., greater than t1and t2and lesser than t3and t4as in the above example.

Access times t1, t2, t3, t4, tt1, tt2, tt3, tt4, and threshold tthmay be measured in absolute time values such as nano seconds, micro seconds, etc., or in terms of system100's cycles or frequencies.

Generally, mechanisms are provided to prevent unwanted process switches in situations such as initiating a second switch (e.g., interrupt) due to data remained from the first switch, initiating a second counter for a second process switch while a first process switch is in progress, etc. In one embodiment, the memory access latency stored in latency manager112is cleared when the latency manager interrupt is triggered, threshold tthis updated, a process is switched, etc. In an alternative embodiment, processor105ignores a second switch while the first switch is in progress.

Computer System Overview

FIG. 3is a block diagram showing a computer system300upon which embodiments of the invention may be implemented. For example, computer system300may be implemented to include system100, system200, latency managers112,212, etc. In one embodiment, computer system300includes a processor304, random access memories (RAMs)308, read-only memories (ROMs)312, a storage device316, and a communication interface320, all of which are connected to a bus324.

Processor304controls logic, processes information, and coordinates activities within computer system300. In one embodiment, processor304executes instructions stored in RAMs308and ROMs312, by, for example, coordinating the movement of data from input device328to display device332.

RAMs308, usually being referred to as main memory, temporarily store information and instructions to be executed by processor304. Information in RAMs308may be obtained from input device328or generated by processor304as part of the algorithmic processes required by the instructions that are executed by processor304.

ROMs312store information and instructions that, once written in a ROM chip, are read-only and are not modified or removed. In one embodiment, ROMs312store commands for configurations and initial operations of computer system300.

Storage device316, such as floppy disks, disk drives, or tape drives, durably stores information for used by computer system300.

Communication interface320enables computer system300to interface with other computers or devices. Communication interface320may be, for example, a modem, an integrated services digital network (ISDN) card, a local area network (LAN) port, etc. Those skilled in the art will recognize that modems or ISDN cards provide data communications via telephone lines while a LAN port provides data communications via a LAN. Communication interface320may also allow wireless communications.

Bus324can be any communication mechanism for communicating information for use by computer system300. In the example ofFIG. 3, bus324is a media for transferring data among processor304, RAMs308, ROMs312, storage device316, communication interface320, etc.

Computer system300is typically coupled to an input device328, a display device332, and a cursor control336. Input device328, such as a keyboard including alphanumeric and other keys, communicates information and commands to processor304. Display device332, such as a cathode ray tube (CRT), displays information to users of computer system300. Cursor control336, such as a mouse, a trackball, or cursor direction keys, communicates direction information and commands to processor304and controls cursor movement on display device332.

Computer system300may communicate with other computers or devices through one or more networks. For example, computer system300, using communication interface320, may communicate through a network340to another computer344connected to a printer348, or through the world wide web352to a web server356. The world wide web352is commonly referred to as the “Internet.” Alternatively, computer system300may access the Internet352via network340.

Computer system300may be used to implement the techniques described above. In various embodiments, processor304performs the steps of the techniques by executing instructions brought to RAMs308. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions to implement the described techniques. Consequently, embodiments of the invention are not limited to any one or a combination of software, hardware, or circuitry.

Instructions executed by processor304may be stored in and carried through one or more computer-readable media, which refer to any medium from which a computer reads information. Computer-readable media may be, for example, a floppy disk, a hard disk, a zip-drive cartridge, a magnetic tape, or any other magnetic medium, a CD-ROM, or any other optical medium, paper-tape, punch-cards, or any other physical medium having patterns of holes, a RAM, a ROM, an EPROM, or any other memory chip or cartridge. Computer-readable media may also be coaxial cables, copper wire, fiber optics, acoustic, or light waves, etc. For example, the instructions to be executed by processor304are in the form of one or more software programs and are initially stored in a CD-ROM being interfaced with computer system300via bus324. Computer system300loads these instructions in RAMs308, executes some instructions, and sends some instructions via communication interface320, a modem, and a telephone line to a network (e.g.340, the Internet352, etc). A remote computer, receiving data through a network cable, executes the received instructions and send the data to computer system300to be stored in storage device316.

In the foregoing specification, the invention has been described with reference to various embodiments thereof. However, it will be evident that modifications and changes may be made thereto without departing from the broader spirit and scope of the invention. For example, to trigger a process switch, it is not necessary that an access time of a memory subsystem is greater than the threshold; the access time can be close to or equal to the threshold, etc. The techniques disclosed herein may be implemented as a method, an apparatus, a system, a device, or their equivalences, a computer-readable medium, etc. Accordingly, the specification and drawings are to be regarded as illustrative rather than as restrictive.