Apparatus and method for executing processes in a multiprocessor system

Abstract: Process migration is controlled in a multiple processor system (10) by a circuit providing a rapid lookup to see if an available processor has any affinity for the top N entries on the system run queue. If the currently available processor has an affinity for, i.e. has one or more lines of operands and/or instructions stored in its local cache related to the process, then the processor selects, using a process affinity cache (54), the process it has affinity for from the top N processes on the run queue. In this manner, unnecessary replacements of cache lines with data from main memory is reduced.

Description: The present invention relates to computer systems having multiple processors. In a multiple processor computer system that shares information stored in a common system memory, a given process may become active and execute at various times on various processors. This process migration from one processor to another may occur for numerous reasons. The process may be suspended from running on one of the multiple processors while the process is waiting for an input/output operation to complete. When the input/output operation is completed, the waiting process is marked as ready to run and placed on the run queue. From the run queue, the process may be selected by the next available processor of the multiple processors, which is not necessarily the processor that the process was previously executed by before the suspension. The process may also be suspended from running on a processor because the time slice allocated thereto has expired. Once a process time slice has expired, the process is typically assigned a lower priority and placed on the run queue, where it may be selected by another processor for execution, i.e. it may migrate. Migration of a process among multiple processors is a fairly recent problem, because system architectures having multiple processor with each processor having a large capacity cache memory associated therewith are fairly new. This system architecture is, however, a common architecture for modern computer systems. Cache memories are included in order reduce system bus data bandwidth and memory data bandwidth requirements. Cache memories reduce data bandwidth requirements by reducing the number of accesses made to the common system memory to fetch instructions or to read or write operands. Cache memories also reduce the cycle time of an average operation because a data transfer between a processor and its local cache memory is faster than a transfer between the same processor and the more distant common system memory, as is well known. Process migration, however, typically results in a cache miss, instead of a cache match, and a cache miss results in a lengthy load operation from system memory. Each time a process migrates to a new processor, the migration starts with a cache miss that is followed by a subsequent loading of all or part of the process from system memory. Thus, frequent cache misses followed by lengthy loads from system memory have a negative effect on system bus and memory data bandwidths. This is exactly the opposite result desired by implementing a system architecture having multiple processors with large local caches. Besides the processor time lost with the cache loading, each time a process migrates, the instructions and operands of that process are duplicated and stored in another cache memory. Needlessly duplicated copies of the same process waste limited cache memory space. Further, migration causes unnecessary cache line replacements of soon-to-be-used processes. In Operating Systems Review (SIGOPS), vol 25, no 5, 1991, New York US, pages 26-40, R. Vaswami et al., "The Implications of Cache Affinity on Processor Scheduling for Multiprogrammed Shared-Memory Multiprocessors" work on quantifying the effect of processor reallocation on the performance of various parallel applications multiprogrammed on a shared memory multiprocessor is discussed. It is the object of the present invention to provide a computer system and a method for operating a computer system, wherein process migration is reduced. Therefore, according to one aspect of the present invention, there is provided a computer system having a plurality of processors each having a local external cache memory means, characterized by affinity cache means adapted to store in a plurality of cache data lines identification codes that identify the most recently executed user processes, each of said cache data lines containing data which indicates the number of data lines occupied in the local external cache memory means by a respective user process; and means for comparing an identification code of a ready-to-run process listed in a run queue of the computer system to each of said identification codes in said affinity cache means to determine if a user process has an affinity for an available processor, based on the number of cache lines associated therewith. According to another aspect of the present invention, there is provided a method of operating a computer system having multiple processors with each processor having cache memory means, including the steps of selecting a process from a run queue for execution, and assigning an identification code to each active process, characterized by the steps of: storing the identification codes of N most recent processes executed for each respective processor; and associating for each of the N most recent processes of each processor a count of a number of cache lines occupied in the corresponding cache memory means with the identification code assigned thereto and storing each of said counts with its associated identification code, each processor selecting from the run queue a next process with an identification code that corresponds to one of the N most recently executed processes if such process is on the run queue, otherwise selecting the highest priority process on the run queue. It will be appreciated that in a computer system and method according to the present invention process migration is reduced by providing a rapid look-up ability to ascertain if an available processor has an affinity for a user process to be executed by the system, thereby reducing unnecessary replacements of cache lines with data from main memory. One embodiment of the present invention will now be described by way of example, with reference to the accompanying drawings, in which:- Fig. 1 is a simplified block diagram of a computer system having P processors. Fig. 2 is a block diagram of one of the P processors with its respective cache memory and its respective affinity scheduling circuits. Fig. 3 is an illustration of the arrangement of data fields in the cache memory and the affinity scheduling circuits shown in Fig. 2. Fig. 4 is a block diagram of an affinity scheduling circuit in accordance with the invention; Fig. 5 is a block diagram of control register zero according to the invention. Fig. 6 is a block diagram of a process affinity cache read input register according to the invention; Fig. 7 is a block diagram of a control register one according to the invention. Fig. 8 is a block diagram of a result register according to the invention. Fig. 9 is a flow diagram showing one method of operating an affinity scheduling apparatus in accordance with the invention. Fig. 10 is a flow diagram showing another method of operating an affinity scheduling apparatus in accordance with the invention. Referring now to Fig. 1, a computer system 10 having multiple processors P 1 -P P is shown. Each of the processors P 1 -P P has a respective cache memory C 1 -C P attached thereto. Each of the processors P 1 -P P is connect via a respective processor bus PB 1 -PB P to a system interconnect network 12, for example a parallel system bus. The interconnect network 12 is also connected to a common system memory 16 via a memory bus 14. The common memory 16 may be accessed by any of the processors P 1 -P P via the interconnect network 12. The system 10, unless preventive measures are taken, will be prone to random process migration. Referring now to Fig. 2, a detailed description of processor P 1 of the P multiple processors of the computer system is given. Since all of the processors are substantially the same, those skilled in the art will understand the structure and operation of the remaining processors and their cache arrangements from the description of P 1 and its cache arrangement. Thus for simplicity, only P1 and its cache arrangement will be described. Processor P 1 is connected to a cache interface 20, which allows the cache memory system C 1 to interact with the processor P 1 and the common system memory (shown in Fig. 1) using any one of a number of standard cache protocols. The cache interface 20 connects the processor P 1 to the cache memory system C 1 and to the processor bus PB 1 . The processor bus PB 1 has a 32 bit parallel address bus, a 64 bit parallel data bus, and an n bit parallel control bus. Processor bus PB 1 further connects to the system interconnect 12 via a system interconnect interface 22 1 . The cache memory system includes a cache controller and cache tag memory (cache controller) 30 1 and a cache memory store 32 1 . The cache memory store 32 1 has a capacity of 8K lines, with each line having 256 bits. The cache controller 30 1 , may be a well known type such as a type 82485 by Intel Corporation, which operates with one or more standard static RAM arrays as the cache memory store 32 1 . Those skilled in the art will recognize that other size cache stores and lines sizes are possible modifications and such modifications are deemed to be within the scope of the present invention. The affinity scheduling system 40 1 is connected to the processor bus PB 1 and to the cache controller 30 1 . The affinity scheduling system 40 1 includes a cache status RAM 42 1 , which in the embodiment of Fig. 2 is an 8K by 10 bit RAM. The address input leads for the cache status RAM 42 1 are the same leads, A16-A5, as the address input leads to the cache memory store 32 1 . By this arrangement, each entry in the cache status RAM 42 1 is associated with a 256 bit line in the cache memory store 32 1 and is a logical extension of each 256 bit line by 10 additional bits. Referring to Figs. 2 and 3, the arrangement and use of cache status RAM 42 1 will be explained. As mentioned above, cache memory store 32 1 and cache status RAM 42 1 are both connected to address lines A16-A5. This connection allows access to 4K addresses. The cache controller 30 1 is connected to address lines A31-A17 of PB 1 in order to transfer cache tag addresses, to address lines A16-A5 thereof in order to transfer cache index addresses, and to n lines of the control bus of PB 1 in order to transfer control signals with the processor P 1 . The cache controller 30 1 uses information from address lines A31-A17 to select between the two 4k arrays of cache memory store 32 1 and also to select a set of addresses from the 8k array of the cache status RAM 42 1 . Each of the addresses from cache status RAM 42 1 is associated with a respective address of the cache memory store 32 1 . Line 36 connects the cache controller 30 1 to the cache memory store 32 1 . An array selection signal is provided via line 36. Line 38 connects the cache controller 30 1 to the cache status RAM 42 1 to provide a set selection signal. Thus the cache controller 30 1 not only is used to operate the cache memory store 32 1 according to standard cache protocols, but also is used to coordinate the association of each of the 256 bit cache data lines in the cache memory store 32 1 with each 10 bit line of the cache status RAM 42 1 . The cache memory store 32 1 has a 64 bit wide data input connected via data bus 47 1 to lines D31-D0 of processor bus PB 1 . Over data bus 47 1 , 64 bits of a 256 bit cache data line are transferred under control of the cache controller 30 1 and address lines A31-A5 in a manner known in the art. The data bits stored in the 10 bit lines of cache status RAM 42 1 are controlled by P 1 and the affinity control 46 1 , as explained below. The affinity control 46 1 is connected via a cache control bus 48 1 to control lines of the control bus of processor bus PB 1 . This cache control bus 48 1 provides the affinity control 46 1 with control signals from processor P 1 and cache controller 30 1 in order to store and retrieve affinity information in association with each of the 256 bit cache lines in cache memory store 32 1 . Storage and retrieval of affinity information is accomplished under control of the operating system of processor P 1 in a manner similar to the operation of the cache system C 1 . In addition to the control bus 48 1 , the affinity control 46 1 is connected by bus 50 1 to address lines A6-A3 of the address bus of PB 1 and by data bus 52 1 to lines D31-D0 of the data bus portion of PB 1 . The processor P 1 in conjunction with the other processors execute the operating system software and assigns a process identification number (PID) to each process as it becomes active. The embodiment shown in Fig. 4 has 8 bits, so PIDs between 0 and 255 are assigned to the first 256 active processes as aliases of unique 32 bit process identification numbers that are pre-assigned to all of the processes that run on the computer system 10 (shown in Fig. 1). For an eight processor system, a total of 256 identification codes corresponds to 32 currently active processes per processor which is considered more than enough for system usage. If the number of active processes exceeds 256, a replacement process, similar to the least recently used cache replacement strategy can be implemented to re-assigned aliases. For larger systems with more than 256 active processes, the number of bits in the PID could be extended as required. In one embodiment of the invention, each processor P 1 -P P has an on-chip 8k byte directly mapped cache memory store. To make the affinity scheduling system operate best, data inclusion between the on-chip cache memory store and the off-chip cache memory store 32 1 is provided by the protocol of the cache controller 30 1 such that all information in the on-chip cache memory store is also in the cache memory store 32 1 . Further, in such an embodiment the operating system assigns the 32 bit process identification number of the process to the operating system kernel that is servicing the process. To account for this, a ninth bit termed a kernel/user bit is added to each PID so the affinity scheduling system may distinguish between the user process and the operating system kernel that is servicing the user process. The former is included in the affinity scheduling system and the latter is not. The tenth bit shown in each 10 bit line of the cache status RAM 42 1 is a valid/invalid bit. When a line in the cache status RAM 42 1 is initially stored, it is marked valid, but if the PID is written to again in an operation with the local processor or another processor then the status of the cache line may be invalid and if so it is marked invalid by changing this bit. Referring now to Figs. 4 and 5, the affinity scheduling system will be described in greater detail. The data bus 52 1 is connected to a PID register 60 1 and the PAC 54 1 , as well as the affinity control 46 1 . A nine bit output from the PID register 60 1 is connected by bus 62 1 to MUX 58 1 and by bus 64 1 to a controlled bus driver 66 1 . The output of the controlled bus driver 66 1 is connected to the cache status RAM 42 1 and to another input of the MUX 58 1 by bus 68 1 . The affinity control 46 1 is connected to the bus driver 66 1 via line 70 1 and controls transmission of data signals over bus 68 1 by a signal transmitted therethrough. The MUX 58 1 has a third input connected to an eight bit output from the affinity control 46 1 via bus 72 1 . An eight bit output of the MUX 58 1 is connected via bus 74 1 to PAC 54 1 . The affinity scheduling system 40 1 is assigned its own address space. The affinity control 46 1 uses this address space as an Input/Output address space. A first control register (CRG0) is located at address offset zero in this Input/Output address space. In the preferred embodiment shown in Fig. 5, this register is 32 bits wide, although twenty of the bits are not used. Operation of the affinity scheduling system 40 1 is enabled by setting bit 31 of CRG0 to a logic 1. Once enabled, control register CRG0 can be written to by processor P 1 in order to load a PID or a kernel/user bit into the affinity control 46 1 . With the PID bits or the kernel/user bit the processor P 1 also writes a two binary bit operation code that instructs the affinity control 46 1 what action is to be taken between the CRG0 and a process affinity cache (PAC) 54 1 . The four operations that involve the control register zero CRG0 and the PAC 54 1 are shown in Table 1. D30 D29 OPERATION 0 0 Toggle kernel/user bit in the PAC, no PID change. 0 1 Loading new PID, allocate to PAC if 1 0 Loading new PID but re-using PID from a different process; allocate to PAC if needed and clear count if in PAC. 1 1 Invalidate all PAC entries and clear every count in the PAC. The PAC 54 1 is a fully associative cache. In the preferred embodiment, the PAC 54 1 is an 8 by 32 bit cache. The format of each line in the PAC 54 1 is shown in Fig. 4. The eight RPID fields are locations for eight most recently used USER PIDs that are used by processor P 1 . Each of the eight RPID fields has associated with it a four bit PAC address-and-enable signal (ADE 3:0), and a sixteen bit count field. The count field represents the number of cache lines in the cache status RAM 42 1 that are occupied by the process corresponding to each RPID in PAC 54 1 . Either of the loading operations shown in Table 1 (01) or (10) will allocate one or more lines for a USER PID, i.e. a PID with its kernel/user bit=0, a location in the PAC 54 1 if that PID is not already stored in the PAC. For the loading and re-using of a PID from a different process operation (10), if the PID is found in the PAC 54 1 , the cache line count field for that PID will be cleared. During either type of loading operation (01 or 10), the MUX 58 1 switches the PID of the current process from the PID register 60 1 to the PAC 54 1 via bus 74 1 . Since the PAC 54 1 is a fully associative cache memory, all of the RPIDs are compared to the current process PID in parallel, except that any RPID that is marked as invalid by valid/invalid bit is omitted from this comparison. If the current process PID matches one of the RPIDs, a match signal, which includes an enable and a three bit address, is sent on a four conductor bus 76 1 is sent from the PAC 54 1 to the affinity control 46 1 . The three bit address on bus 76 1 is the logical line address of the matching RPID. The processor P 1 then completes the loading process with the affinity data into PAC 54 1 . If a type (10) loading operation is initiated, the affinity control 46 1 logically provides a reset signal that resets the count field of a matching RPID. If a matching RPID is not found, a miss signal is sent from the PAC 54 1 to the affinity control 46 1 via bus 76 1 . In the event of a miss, the PAC 54 1 also sends via bus 76 1 the logical address of the RPID with the lowest count field value to the affinity control 46 1 . The affinity control 46 1 subsequently writes the eight bit PID that missed into the cache line in the PAC 54 1 that corresponds to the logical address sent with the miss signal, sets the valid bit of that line, and resets the count field of that line to zero. Alternatively, another embodiment provides that upon such a miss, the PAC 54 1 sends the logical address of the first cache line that has a its valid/invalid bit marked as invalid as the replacement criterion. Referring back to Fig. 2 in operation, the processor P 1 writes instructions and data to CRG0 to perform one of the four operations in Table 1. When the current operation is completed, the affinity control 46 1 sends a control signal to the processor P 1 to indicate that the affinity scheduling operation is completed and the processor may continue program execution. After this complete signal is sent, MUX 58 1 switches a ten bit output from cache status RAM 42 1 onto the input of the PAC 54 1 via bus 74 1 . If the result of an access is a cache miss, as detected by monitoring the match/miss control line in bus 48 1 , the ten bit output of the cache status RAM 42 1 is tested by the affinity control 46 1 via bus 82 1 . If the valid/invalid bit is asserted and the kernel/user bit is reset, a lookup is performed in the PAC 54 1 . If a matching RPID entry is found in the PAC 54 1 , the associated count is decremented by the affinity control 46 1 , since a cache line belonging to that process is being replaced in the cache memory store C 1 . This operation takes place while the cache controller 30 1 is requesting and obtaining the system interconnect 12 to perform a cache line fill from common memory 16 (shown in Fig. 1). Next, the MUX 58 1 switches the eight bit PID from the PID register 60 1 to the input of the PAC 54 1 via buses 62 1 and 74 1 . At the same time, affinity control 46 1 controls bus driver 66 1 to transmit the kernel/user bit and the eight bit PID from PID register 60 1 to the input/output port of the cache status RAM 42 1 via buses 62 1 and 68 1 . While a cache line is being accessed from common memory, a lookup is performed in the PAC 54 1 , except if the kernel/user bit is reset since no kernel RPIDs are stored into the PAC 54 1 . As the cache line is stored in the cache memory store 32 1 , the kernel/user bit and the eight bit current PID is written from the PID register 60 1 to the cache status RAM 42 1 and the valid/invalid bit is set. If the current PID from the PID register 60 1 has a kernel/user bit that is reset to logic zero, the matching entry in the PAC 54 1 will have its count field incremented since the associated user process now has an additional line in the cache memory store 32 1 and/or the on-chip cache of the processor P 1 (not shown). In addition to cache actions caused by processor P 1 , the affinity control 46 1 also monitors the control bus 48 1 to determine if an invalidating snoop cycle initiated by another processor of the computer system 10 (shown in Fig. 1) is affecting the local cache 32 1 . If a snoop cycle that invalidates at least one cache line of cache memory store 32 1 is detected by the affinity control 46 1 , the eight bit PID of the invalidated cache line will be entered into the PID register 60 1 and this eight bit PID is transmitted to the PAC 54 1 test for an RPID match. If a match occurs, the count field in the cache line of the matching RPID is decremented. Thus, the affinity scheduling system 40 1 remains coherent when another processor in the system 10 writes to or reads with intent to write to, i.e. read for ownership, an address contained in a line stored in a local cache of the processor P 1 . In this manner, each affinity scheduling system maintains a count of the total number of cache lines associated with up to eight of the most recently executed user processes that are presently in its respective cache memory store of its respective processor. If a process is suspended, for example for system I/O or time slice expiration, the local processor will re-enter the kernel of the operating system and will set the kernel/user bit associated with the PID of that process. Further, the processor should write a toggle operation (00) shown in Table 1 to its control register CRG0 of its affinity control 46 1 . The kernel/user bit being set will indicate to the affinity scheduling system 40 1 that this process is not ready to run and it will not be contained in the PAC 54 1 as one of the eight most recently run programs. Thus, the PAC 54 1 only contains RPIDs of ready to run, high affinity processes. This provides the best criterion for affinity scheduling. As a processor P 1 -P P becomes available for a next process, it can read the up to eight entries in its PAC 1 -PAC P by reading 32 bit words from an output port of the PAC. The data format of the 32 bit word is shown in Fig. 6. After reading the eight line entries in its respective PAC, the available processor has data that describe the eight processes that have the greatest affinity for the processor. If one or more of these eight processes are in the run queue as ready-to-run, the processor can schedule the process that occupies the greatest number of lines in its respective cache memory system C 1 -C P . Since this process on the run queue must be executed sooner or later, it will be executed more efficiently by the processor that has to load fewest lines into its cache in order to run it. Methods for selecting ready-to-run user processes from the system run queue according to the affinity of the process for the available processor are shown in Figs. 9 and 10. Referring now to Figs. 7 and 8, an alternate process for affinity scheduling is to include a probe capability within the affinity scheduling system 40 1 . Such a process uses a second control register CRG1 that has a data format as shown in Fig. 7 and a result register having a data format as shown in Fig. 8. The process provides that a PID is extracted from a ready-to-run process identification number of a process in the run queue. This is accomplished by writing an eight bit probe PID into CRG1. This write triggers the affinity control 46 1 to compare the probe PID to the RPID entries in the PAC 54 1 . If no matching entry is found, the result register is loaded with the probe PID, a count field of all zeroes, and a not found flag is asserted. If a matching RPID entry is found, the result register is loaded with the probe PID, the count field associated with the matching RPID entry, and a found flag is asserted. The processor can read the result register and then run the found process. Modifications of the described embodiment are possible. For example the PID numbers could be longer, to make a greater number of them available. Also, the process affinity cache could be exapanded beyond eight entries to encompass more of the most recently executed processes.

Claims: A computer system having a plurality of processors (P) each having a local external cache memory means (32), characterized by affinity cache means (54) adapted to store in a plurality of cache data lines identification codes that identify the most recently executed user processes, each of said cache data lines containing data which indicates the number of data lines occupied in the local external cache memory means (32) by a respective user process; and means for comparing an identification code of a ready-to-run process listed in a run queue of the computer system to each of said identification codes in said affinity cache means (54) to determine if a user process has an affinity for an available processor, based on the number of cache lines associated therewith. A computer system according to claim 1, characterised in that said affinity cache means (54) is a fully associative cache memory. A computer system according to claim 1, or claim 2, characterized by means for associating and storing a respective kernel/user bit with each of the plurality of identification codes in said affinity cache means (54). A computer system according to any one of the preceding claims, characterized by means for storing a respective valid/invalid bit that is associated with each of the plurality of identification codes in said affinity cache means (54). A computer system according to any one of the preceding claims, characterized in that the entries in said affinity cache means (54) are maintained according to a lowest affinity replaced first policy, where the process in said cache memory means (32) that occupies the fewest cache lines has the lowest affinity to a non-matching process. A computer system according to any one of claims 1 to 4, characterized in that the entries in said cache memory means (32) are maintained according to a least recently used replacement process. A method of operating a computer system having multiple processors with each processor having cache memory means (32), including the steps of selecting a process from a run queue for execution, and assigning an identification code to each active process, characterized by the steps of: storing the identification codes of N most recent processes executed for each respective processor; and associating for each of the N most recent processes of each processor a count of a number of cache lines occupied in the corresponding cache memory means (32) with the identification code assigned thereto and storing each of said counts with its associated identification code, each processor selecting from the run queue a next process with an identification code that corresponds to one of the N most recently executed processes if such process is on the run queue, otherwise selecting the highest priority process on the run queue. A method according to claim 7, characterized in that said step of selecting involves selecting the one of the N most recently executed processes which has the highest count of cache lines associated therewith.

Inventors: COCHCROFT JR ARTHUR F, PIKE JIMMY D, COCHCROFT JR.,ARTHUR F., PIKE, JIMMY D.
Applicants: NCR INT INC, NCR INTERNATIONAL, INC.
Application year: 1993
Main classification: G06F9
IPC codes: G06F9, G06F12