Patent Publication Number: US-8112611-B2

Title: Allocating resources to partitions in a partitionable computer

Description:
This application is a continuation under 35 U.S.C. §120 of U.S. patent application Ser. No. 10/898,590, filed Jul. 23, 2004 now U.S. Pat. No. 7,606,995, and entitled “Allocating Resources To Partitions In A Partitionable Computer,” the entirety of which is hereby incorporated herein by reference. 
    
    
     BACKGROUND 
     1. Field of the Invention 
     The present invention relates to partitionable computers and, more particularly, to techniques for allocating resources to partitions in partitionable computers. 
     2. Related Art 
     Computer system owners and operators are continually seeking to improve computer operating efficiencies and hence to reduce the cost of providing computing services. For example, servers of various kinds—such as database servers, web servers, email servers, and file servers—have proliferated within enterprises in recent years. A single enterprise may own or otherwise employ the services of large numbers of each of these kinds of servers. The cost of purchasing (or leasing) and maintaining such servers can be substantial. It would be advantageous, therefore, to reduce the number of servers that must be used by an enterprise without decreasing system performance. 
     One way to reduce the number of servers is through the process of “server consolidation,” in which multiple independent servers are replaced by a single server, referred to herein as a “consolidation server.” A consolidation server typically is a powerful computer system having significant computing resources (such as multiple processors and large amounts of memory). The consolidation server may be logically subdivided into multiple “partitions,” each of which is allocated a portion of the server&#39;s resources. A multi-partition consolidation server is an example of a “partitionable computer.” Each partition may execute its own operating system and software applications, and otherwise act similarly to an independent physical computer. 
     Unlike a collection of independent servers, typically it is possible to dynamically adjust the resources available to each partition/application in a consolidation server. Many applications experience variation in workload demand, which is frequently dependent on time of day, day of month, etc. Periods of high workload demand are frequently not coincident. Applying available resources to current high-demand workloads achieves improved resource utilization, decreased overall resource requirements, and therefore reduced overall cost. 
     As partitionable computers become more powerful, the trend is for them to include a greater and greater number of processors. In particular, a single partitionable computer typically includes several (e.g., 4) “cell boards,” each of which includes several (e.g., 2, 4, 8, or 16) processors. The cell boards are interconnected through a switching fabric and collectively provide an effective processing power that approaches the aggregate processing power of the individual processors they contain. Each successive generation of cell boards tends to include a greater number of processors than the previous generation. 
     Early processors, like many existing processors, included only a single processor core. A “multi-core” processor, in contrast, may include one or more processor cores on a single chip. A multi-core processor behaves as if it were multiple processors. Each of the multiple processor cores may essentially operate independently, while sharing certain common resources, such as a cache. Multi-core processors therefore provide additional opportunities for increased processing efficiency. 
     As the size, power, and complexity of partitionable computer hardware continues to increase, it is becoming increasingly desirable to provide flexibility in the allocation of computer resources (such as processors and I/O devices) among partitions. Insufficient flexibility in resource allocation may, for example, lead to underutilization of resources allocated to a first partition, while a second partition lacking sufficient resources operates at maximum utilization. What is needed, therefore, are improved techniques for allocating computer resources to partitions in partitionable computer systems. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a block diagram of a multiprocessor computer system according to one embodiment of the present invention; 
         FIG. 2  is a block diagram of one of the CPUs of the computer system of  FIG. 1  according to one embodiment of the present invention; 
         FIG. 3  is a flowchart of a method that is performed by a bit substitution circuit of  FIG. 2  according to one embodiment of the present invention; 
         FIG. 4A  is a flowchart of a method that is performed by the cache of  FIG. 2  according to one embodiment of the present invention; 
         FIG. 4B  is a flowchart of a method that is performed by the address mapper of  FIG. 2  according to one embodiment of the present invention; 
         FIG. 5  is a diagram of a mapping between processor cores and hardware partitions in a partitionable computer system according to one embodiment of the present invention; 
         FIGS. 6A-6B  illustrate an I/O controller according to one embodiment of the present invention; 
         FIG. 7  is a diagram of a mapping between I/O ports and partitions in a partitionable computer system according to one embodiment of the present invention; 
         FIG. 8  is a flowchart of a method performed by the destination decoder of  FIGS. 6A-6B  to decode a physical address in an incoming transaction according to one embodiment of the present invention; 
         FIG. 9  is a flowchart of a method that is performed by the bit substitution circuit of  FIGS. 6A-6B  according to one embodiment of the present invention; 
         FIG. 10  is a flowchart of a method that is performed by the cache of  FIGS. 6A-6B  according to one embodiment of the present invention; 
         FIG. 11  is a flowchart of a method that is performed by the address mapper of  FIGS. 6A-6B  according to one embodiment of the present invention; 
         FIG. 12A  is a diagram of a partition-identifying address according to one embodiment of the present invention; and 
         FIG. 12B  is a diagram of a partition-identifying address according to another embodiment of the present invention. 
     
    
    
     DETAILED DESCRIPTION 
     Before describing embodiments of the present invention, general features of multiprocessor computer architectures will be described. Although there are a variety of multiprocessor computer architectures, the symmetric multiprocessing (SMP) architecture is one of the most widely used architectures. Referring to  FIG. 1 , a computer system  100  having an SMP architecture is shown in block diagram form. The computer system  100  includes a plurality of cell boards  102   a - d  interconnected using a switching fabric  116 , also referred to as a “system fabric” or simply a “fabric.” Each of the cell boards  102   a - d  includes a plurality of CPUs, a system bus, and main memory. 
     For ease of illustration and explanation, the cell board  102   a  is shown in more detail in  FIG. 1  and will now be described in more detail. The other cell boards  102   b - d , however, may include components and a structure that are the same as or similar to that of cell board  102   a . The cell board  102   a  includes a plurality of CPUs  104   a - n , where n is a number such as 2, 4, 8, or 16. The CPUs  104   a - n  include on-board caches  106   a - n , respectively. The cell board  102   a  also includes a system bus  108 , main memory  112   a , and memory controller  110   a . The CPUs  102   a - n  are coupled directly to the system bus  108 , while main memory  112   a  is coupled to the system bus  108  through memory controller  110   a . CPUs  104   a - n  may communicate with each other over the system bus  108  and may access the memory  112   a  over the system bus  108  through the memory controller  110   a , as is well-known to those of ordinary skill in the art. 
     Although cell boards  102   a - d  include their own local system memories  112   a - d  coupled to corresponding memory controllers  110   a - d , the memories  112   a - d  may be addressed by the CPUs in the cell boards  102   a - d  using a single combined physical address space. The fabric  116  provides a mechanism for communication among the cell boards  102   a - d  to perform such shared memory access and other inter-cell board communication. 
     The fabric  116  may, for example, include one or more crossbar switches. A crossbar switch is a device that has a number of input/output ports to which devices may be connected. A pair of devices connected to a pair of input/output ports of a crossbar switch may communicate with each other over a path formed within the switch connecting the pair of input/output ports. The paths set up between devices can be fixed for some duration or changed when desired. Multiple paths may be active simultaneously within the crossbar switch, thereby allowing multiple pairs of devices to communicate with each other through the crossbar switch simultaneously and without interfering with each other. 
     The fabric  116  may be implemented using components other than crossbar switches. For example, the fabric  116  may be implemented using one or more buses. 
     Cell board  102   a  also includes a fabric agent chip  114   a  that is coupled to the fabric  116  and which acts as an interface between the cell board  102   a  and the other cell boards  102   b - d  in the system  100 . The other cell boards  102   b - d  similarly include their own fabric agent chips  114   b - d , respectively. Although the fabric agent chips  114   a - d  are illustrated as distinct components in  FIG. 1 , fabric agent chips  114   a - d  may be considered to be part of the system fabric  116 . 
     As described above, the local memories  112   a - d  in the cell boards  102   a - d  may be accessed using a single physical address space. In an SMP such as the system  100  shown in  FIG. 1 , this is made possible by the fabric agent chips  114   a - d  For example, consider a case in which CPU  104   a  issues a memory access request to memory controller  110   a  that addresses a memory location (or range of memory locations) in the shared physical address space. If the memory controller  110   a  cannot satisfy the memory access request from the local memory  112   a , the memory controller  110   a  forwards the request to the fabric agent chip  114   a . The fabric agent chip  114   a  translates the physical address in the request into a new memory address (referred to as a “fabric address”) that specifies the location of the requested memory, and transmits a new memory access request using the new fabric address to the fabric  116 . The fabric  116  forwards the memory access request to the fabric agent chip in the appropriate cell board. 
     The requested memory access is performed using the local memory of the receiving cell board, if possible, and the results are transmitted back over the fabric  116  to the fabric agent chip  114   a  and back through the memory controller  110   a  to the CPU  104   a . The CPUs in cell boards  102   a - d  may thereby access the main memory in any of the other cell boards  102   a - d  over the fabric  116  using the fabric agent chips  114   a - d  in the cell boards  102   a - d . One goal of such a system is to make the implementation of memory access transparent to the CPUs  104   a - d , in the sense that the CPUs  104   a - d  may transmit and receive responses to memory access requests in the same way regardless of whether such requests are satisfied from onboard memory or offboard memory. 
     In one embodiment of the present invention, techniques are provided for allocating multiple physical resources on a single chip to a plurality of partitions in a partitionable computer system. In this embodiment, when one of the resources generates a transaction containing a physical address, a partition identification value (identifying the partition to which the resource is allocated) is stored in the physical address to create a partition-identifying address. The transaction, including the partition-identifying address, is transmitted over the fabric  116  and thereby routed to the appropriate destination. 
     This embodiment will be explain using an example in which multiple microprocessor cores in a single microprocessor are allocated to a plurality of partitions. For example, referring to  FIG. 2 , a functional block diagram is shown of the CPU  104   a  according to one embodiment of the present invention. In the embodiment illustrated in  FIG. 2 , the CPU  104   a  is a multi-core processor. In particular, the CPU  104   a  includes a plurality of processor cores  204   a - n  on a single chip, where n may any number, such as 2, 4, 8, or 16. The cores  204   a - n  may, for example, be conventional processor cores such as those found in conventional multi-core processors. In the embodiment illustrated in  FIG. 1 , all of the cores  204   a - n  share a single cache  208 . The cores  204   a - n  need not, however, share a single cache. Rather, for example, each core may have its own cache, or groups of cores may share different caches. 
     In a conventional partitionable computer system, all of the cores in a multi-core processor are required to be allocated to a single partition. Furthermore, if the CPU  104   a  were a conventional multi-core processor, the cores  204   a - n  would communicate directly with the cache  208 . For example, the core  204   a  would transmit a memory write request, including the address of the memory address to be written, directly to the cache  208 , which would satisfy the request locally if possible or by performing an off-board write to main memory otherwise. 
     The multi-core processor  104   a  illustrated in  FIG. 2 , in contrast, enables the cores  204   a - n  to be allocated to a plurality of partitions. For example, referring to  FIG. 5 , a diagram is shown of a mapping  502  between processor cores  506   a - h  and partitions  504   a - d  in the partitionable computer system  100  according to one embodiment of the present invention. Cores  506   a - h  in  FIG. 5  represent cores  204   a - n  in  FIG. 2  in the case where n=8. For example, core  506   a  represents core  204   a  and core  506   h  represents core  204   n  when n=8. 
     Note that each of the partitions  504   a - d  is not itself a physical component of the computer system  100 . Rather, each of the partitions  504   a - d  is a logical construct that is defined by the resources (e.g., processor cores) that are allocated to it. The resources allocated to a particular partition may change over time. 
     In the example shown in  FIG. 5 , core  506   b  is allocated to partition  504   a  (indicated by mapping  502   b ), cores  506   a  and  506   b  are allocated to partition  504   b  (indicated by mappings  502   a  and  502   b , respectively), cores  506   c ,  506   e , and  506   f  are allocated to partition  504  (indicated by mappings  502   c ,  502   e , and  502   f , respectively), and cores  506   g - h  are allocated to partition  504   d  (indicated by mappings  502   g - h , respectively). 
     The particular mapping  502  illustrated in  FIG. 5  is shown merely for purposes of example and does not constitute a limitation of the present invention. There may be any number of partitions, and cores may be allocated to partitions in any arrangement. 
     To enable the cores  204   a - n  to be allocated to multiple partitions, the CPU  104   a  includes a plurality of partition ID registers  210   a - n  associated with the plurality of cores  204   a - n  respectively. For example, partition ID register  210   a  is associated with core  206   a  and stores a value that represents mapping  502   a  ( FIG. 5 ). Similarly, partition ID register  210   n  is associated with core  204   n  and stores a value that represents mapping  502   h . Each of the partition ID registers  210   a - n  includes at least enough bits to represent the number of partitions in the computer system  100 . In particular, if P is the number of partitions in the computer system  100 , each of the partition ID registers  210   a - n  includes at least log 2  P bits. For example, if there are four partitions (as in the example illustrated in  FIG. 5 ), each of the partition ID registers  210   a - n  includes at least 2 (log 2  4) bits. 
     Each of the partition ID registers  210   a - n  stores a unique partition ID value that uniquely identifies the partition to which the corresponding one of the cores  204   a - n  is allocated. For example, let PIR i  be the partition ID register at index i, and let C i  be the corresponding processor core at index i, where i ranges from 0 to n−1. If core C i  is allocated to partition j, then the value j may be stored in the partition ID value in partition ID register PIR i . In this way, a unique value identifies each of the partitions in the system  100 . The values stored in the partition ID registers  210   a - n  may, for example, be set by configuration software executing in the computer system  100 . 
     For example, referring again to the example illustrated in  FIG. 5 , the value 1 (binary 01) may be stored in partition ID register  210   a , thereby indicating that core  204   a  (represented by core  506   a  in  FIG. 5 ) is allocated to partition  1  ( 504   b ). Similarly, the value 3 (binary 11) may be stored in partition ID register  210   n , thereby indicating that core  204   n  (represented by core  506   h  in  FIG. 5 ) is allocated to partition  3  ( 504   d ). 
     The CPU  104   a  may be configured so that the partition ID values stored in the partition ID registers  210   a - n  cannot be changed by the operating system executing on the computer system  100 . This fixedness of the partition ID values may be enforced, for example, by any of a variety of hardware security mechanisms, or simply by agreement between the configuration software and the operating system. 
     To implement the allocation of the cores  204   a - n  to the multiple partitions  504   a - d , the main memory  112   a - d  of the computer system  100  is allocated among the partitions  504   a - d , so that each partition is allocated a portion of the main memory  112   a - d . The main memory  112   a - d  may be allocated to the partitions  504   a - d  in blocks of any size. For example, the main memory  112   a - d  may be allocated to partitions  504   a - d  on a per-address, per-page, or per-controller basis. 
     In one embodiment of the present invention, a core that transmits a memory access request need not specify the partition to which the requested memory addresses are allocated. Rather, the core need only specify the requested memory address using a memory address (referred to as a “physical address”) within an address space (referred to as a “physical address space”) associated with the partition to which the core is allocated. Typically the main memory  112   a - d  is logically divided into a plurality of physical address spaces. Each of the physical address spaces typically is zero-based, which means that the addresses in each physical address space typically is numbered beginning with address zero. 
     To accomplish this result, mechanisms are provided for distinguishing a particular address in one partition from the same address in other partitions. In particular, the CPU  104   a  includes bit substitution circuits  212   a - n , which are coupled between cores  204   a - n  and partition ID registers  210   a - n , respectively. 
     To appreciate the function performed by the bit substitution circuits  212   a - n , consider a case in which core  204   a  transmits a write command  230   a  on lines  214   a  to bit substitution circuit  212   a . The write command  230   a  includes a physical address of the memory location to be written and a value to write into that location. The physical address is illustrated in  FIG. 2  as “a[54:0]” to indicate that bits  0 - 54  of the address contain useful (address-identifying) information. 
     The term “system space” refers herein to an address space that contains unique addresses for each memory location in the entire main memory  112   a - d . Assume, for purposes of example, that the system address space is 4 GB (0x100000000) and that there are four equally-sized (1 GB) partitions. The physical memory space of each of the partitions in such a case would have an address range of 0-1 GB (0x00000000-0x40000000). The first partition might be allocated (mapped) to the first gigabyte of the system address space, the second partition might be allocated to the second gigabyte of the system address space, and so on. When a core allocated to a particular partition generates a physical memory address as part of a memory access request, it is necessary to translate the physical memory address into a system memory address. Examples of techniques for performing this translation according to one embodiment of the present invention will now be described. 
     For purposes of example, assume that the physical address in the write command  230   a  transmitted on lines  214   a  is a 64-bit value but that only the 55 least significant bits are needed to fully address the physical address space allocated to a single partition. In such a case, the 9 uppermost address bits are not needed to specify physical addresses. Upon startup of a multi-partition computer system, the operating system executing in each partition is informed of the size of the physical address space that is allocated to it. As a result, a well-behaved operating system will not generate addresses that use more bits than necessary (e.g., 55) to address its allocated memory partition. As described in more detail below, however, even if the operating system in a particular partition is not well-behaved and generates addresses outside of its allocated address range, the techniques disclosed herein prevent such an operating system from accessing such prohibited addresses, thereby enforcing inter-partition security. 
     Referring to  FIG. 3 , a flowchart is shown of a method  300  that is performed by the bit substitution circuit  212   a  according to one embodiment of the present invention when write command  230   a  is transmitted by core  204   a  on lines  214   a . The bit substitution circuit  212   a  receives the write command  230   a  (or other memory access request, such as a read command) (step  302 ). In response to receiving write command  230   a , the bit substitution circuit  212   a  reads the partition ID value from the partition ID register  210   a  on lines  216   a  (step  304 ). The bit substitution circuit  212   a  writes the partition ID value into the physical address, thereby producing a partition-identifying address that includes both the original physical address and the partition ID value (step  306 ). 
     Referring to  FIG. 12A , a diagram is shown of an example of a partition-identifying address  1200  produced in step  306  according to one embodiment of the present invention. The example partition-identifying address  1200  illustrated in  FIG. 12A  is 64 bits wide. Portion  1202  (bits  0 - 52 ) of partition-identifying address  1200  contains bits  0 - 52  of the physical address contained in the original write command  230   a . In one embodiment of the present invention, bit substitution circuit  212   a  writes the partition ID value obtained from write command  230   a  into portion  1204  (bits  53 - 54 ) of the partition-identifying address  1200  (step  306 ), thereby overwriting the original values stored in portion  1204 . Portion  1208 , which includes both portions  1202  and  1204 , therefore unambiguously identifies the system memory address indicated by the original write command  230   a . Portion  1206  (bits  55 - 63 ) of the partition-identifying address  1200  are unused. Portion  1208  therefore represents the “used portion” of address  1200  because the combination of the partition ID portion  1204  and the physical address portion  1202  are used to specify a unique address in the system  100 . 
     Recall that a well-behaved operating system will not attempt to access memory locations having addresses outside of the address space that has been allocated to it, and will therefore not set any of the bits in portions  1204  or  1206 . If, however, an operating system does set any bits in portion  1204 , such bits will be overwritten by the bit substitution circuit  212   a  in step  306 . The bit substitution circuit  212   a  may further be configured to overwrite portion  1206  with zeros or some other value. The bit substitution circuit  212   a  may thereby prevent the operating system from accessing addresses outside of its partition and thereby enforce inter-partition security. 
     The particular layout of the partition-identifying address  1200  in  FIG. 12A  is shown merely for purposes of example and does not constitute a limitation of the present invention. Rather, partition-identifying addresses of any size and having any layout may be used in conjunction with embodiments of the present invention. For example, the layout of partition-identifying addresses may vary from partition to partition. For example, one partition may be allocated twice as much address space as another, in which case addresses in the larger partition will include one less bit of partition ID (portion  1204 ) and one more bit of physical address (portion  1202 ) than addresses in the smaller partition. The bit substitution circuits  212   a - n , therefore, may be individually programmable to insert partition IDs of varying sizes into the addresses generated by the cores  204   a - n.    
     The bit substitution circuit  212   a  generates a first modified write command  232   a  (or other memory access request) containing the partition-identifying address generated in step  306  (step  308 ). The bit substitution circuit  212   a  transmits the first modified write command  232   a  (or other memory access request) to the cache  208  on lines  218   a  (step  310 ). 
     The combination of a core, partition ID register, and bit substitution circuit in the manner described and illustrated above with respect to  FIG. 2  is referred to herein as an “extended core.” For example, CPU  104   a  includes extended cores  206   a - n . Extended core  206   a  includes core  204   a , partition ID register  210   a , and bit substitution circuit  212   a , while extended core  206   n  includes core  204   n , partition ID register  210   n , and bit substitution circuit  212   n . Although core  204   a , bit substitution circuit  212   a , and partition ID register  210   a  are illustrated as distinct components in  FIG. 2 , the functions performed by the bit substitution circuit  212   a  and/or partition ID register  206   a  may be integrated into the core  204   a , so that the core  204   a  may communicate directly with the cache  208 . 
     Referring to  FIG. 4A , a flowchart is shown of a method  400  that is performed by the cache  208  in response to receipt of the first modified write command  232   a  according to one embodiment of the present invention. The cache  208  receives the first modified write command  232   a  from the bit substitution circuit  212   a  (step  402 ). The cache  208  determines whether the write request can be satisfied locally, i.e., whether there is a cache hit in cache lines  234  based on the partition-identifying address contained in the first modified write command  232   a  (step  404 ). In other words, the cache  208  determines whether the value of the memory location addressed by the partition-identifying address contained in the first modified write command  232   a  is stored in cache lines  234 . The cache  208  may perform step  404  by using the partition-identifying address contained in the first modified write command  232   a  as an index and tag and then using any of a variety of well-known techniques to determine whether there is a cache hit based on that index and tag. 
     Note that the address bits in which the partition ID value is stored may occupy either the index or tag field of the cache  208 . If the partition ID value is stored in the index field of the cache  208 , then the partitions  504   a - d  are allocated fixed and distinct (non-overlapping) portions of the cache  208 . If, however, the partition ID value is stored in the tag field of the cache  208 , then the entire cache  208  is shared by the partitions  504   a - d , and the particular cache locations used by any partition is dynamic and depends on the workload of the cores  204   a - n  at any particular point in time. 
     If there is a cache hit, the cache  208  performs the write locally (i.e., within the cache lines  234 ) (step  406 ) and the method  400  terminates. The cache  208  may transmit an acknowledgment to the core  204   a  on lines  224   a . If the core  204   a  transmits a read command to the cache  208 , the cache  208  may transmit the read values to the core  204   a  on lines  224   a.    
     If there is a cache miss, the cache  208  transmits a second modified write command  236  to an address mapper  222  (step  408 ). In one embodiment of the present invention, the second modified write command  236  contains: (1) a source terminus ID (e.g., the terminus ID of the memory controller  110   a  that services the CPU  104   a ), labeled “S” in  FIG. 2 ; (2) a transaction ID (a unique transaction identifier), labeled “I” in  FIG. 2 ; (3) a request type (e.g., memory read or write), labeled “R” in  FIG. 2 ; and (4) the partition-identifying address  1200  extracted from the first modified write command  232   a , labeled “P 1 , a[n:0]” in  FIG. 2 . 
     Although particular transactions are described above with respect to core  206   a  for purposes of example, the other cores  206   b - n  may perform transactions in the same manner. For example, core  204   n  may generate a write command  230   n  on lines  214   n , in response to which bit substitution circuit  212   n  may read the value of partition ID register  210   n  on lines  216   n . The bit substitution circuit  212   n  may transmit a first modified write command  232   n  on lines  218   n , which may be processed by the cache  208  in the manner described above. The cache  208  may communicate with the core  204   n  directly over lines  224   n.    
     In one embodiment of the present invention the partition-identifying address contained in the second modified write command  236  is translated into a system address. Referring to  FIG. 4B , a flowchart is shown of a method  420  that is performed in one embodiment of the invention to perform such a translation. The method  420  may, for example, be performed after step  408  of method  400  ( FIG. 4A ). 
     The CPU  104   a  includes an address mapper  222 , which is coupled to the cache  208  over lines  220  and which therefore receives the second modified write command  236  (step  422 ). The address mapper  222  maps the partition-identifying address  1200  contained in the second modified write command  230  to: (1) a destination terminus ID (e.g., a terminus ID of the memory controller that controls access to the requested memory addresses), and (2) a transaction type (step  424 ). The transaction type serves a purpose similar to that of the original request type (e.g., memory read or write), except that the request type is used for transactions over the fabric  116 . Techniques for translating request types into transaction types are well-known to those of ordinary skill in the art. 
     In one embodiment of the present invention, each of the CPUs in the system  100  (e.g., CPUs  104   a - n ) and each of the memory controllers  110   a - d  in the system  100  has a unique terminus identifier (terminus ID). In such an embodiment, a particular physical address in a particular partition may be uniquely addressed by a combination of the physical address, the partition ID of the partition, and the terminus ID of the memory controller that controls the memory in which that physical address is stored. Note further that because the address transmitted over the fabric  116  is a partition-identifying address (i.e., an address which includes both a physical address and a partition ID), the target memory controller may distinguish among the same physical address in different partitions. In the embodiment illustrated in  FIG. 2 , therefore, a single memory controller may control memory allocated to any number of partitions. 
     It should be appreciated, however, that this particular scheme is merely an example and does not constitute a limitation of the present invention. Other addressing schemes may be used in conjunction with the techniques disclosed herein, in which case different combinations of terminus identifiers, physical addresses, system addresses, partition identifiers, or other data may be required to uniquely address particular memory locations. 
     The address mapper  222  may, for example, maintain an address mapping  238  that maps partition-identifying addresses to destination terminus IDs and transaction types. The address mapper  222  may use the mapping  238  (which may, for example, be implemented as a lookup table) to perform the translation in step  424 . The address mapping  238  need not contain an entry for every partition-identifying address. Rather, the address mapping  238  may, for example, map ranges of partition-identifying addresses (identified by their most significant bits) to pages of memory or to memory controllers. The address mapper  222  may ensure that a processor core allocated to one partition cannot access memory locations in another partition by mapping such requests to a null entry, thereby causing the address mapper  222  to generate a mapping fault. 
     The address mapper  222  generates and transmits a third modified write command  240  to the system fabric  116  (step  426 ). The third modified write command  240  includes: (1) the source terminus ID (S), transaction ID (I), request type (R), and partition-identifying address (P 1 , a[n:0]) from the second modified write command  236 ; and (2) the destination terminus ID (D) and transaction type (T) identified in step  424 . The system fabric  116  includes a router  228  that uses techniques that are well-known to those of ordinary skill in the art to transmit the third modified write command  240  to the memory controller having the specified destination terminus ID. The router  228  may, for example, maintain a mapping  244  that maps pairs of input ports and destination terminus IDs to output ports. 
     When the router  228  receives the third modified write command  240  on a particular input port, the router uses the identity of the input port and the destination terminus ID contained in the third modified write command  240  to identify the output port that is coupled to the memory controller that controls access to the requested memory address(es). The router  228  transmits the third modified write command  240  (or a variation thereof) to the identified memory controller on lines  242 . The third modified write command  240  may then be satisfied by the destination memory controller using techniques that are well-known to those of ordinary skill in the art. 
     When the router  228  receives an inbound transaction on lines  246 , the router  228  may route the transaction to the cache on lines  226  using techniques that are well-known to those of ordinary skill in the art. The incoming transaction may then be processed by the cache  208  and, if necessary, by one or more of the cores  206   a - n , using conventional techniques. 
     In another embodiment of the present invention, techniques are provided for allocating a plurality of hardware resources to a plurality of partitions in a partitionable computer system. This embodiment will be explained using an example in which a plurality of resources in a single I/O controller are allocated to a plurality of partitions. For example, referring to  FIGS. 6A-6B , a functional block diagram is shown of an I/O controller  602  according to one embodiment of the present invention. The I/O controller  602  serves two I/O devices  604   a - b  coupled to the I/O controller  602  through I/O ports  630   a - b , respectively. Examples of techniques will now be described for allocating the first I/O port  628   a  to a first partition and the second I/O port  628   b  to a second partition, and thereby for allocating the first I/O device  604   a  to the first partition and the second I/O device  604   b  to the second partition. 
     For example, referring to  FIG. 7 , a diagram is shown of a mapping  702  between I/O ports  628   a - b  and hardware partitions  704   a - d  in the partitionable computer system  100  according to one embodiment of the present invention. The mapping  702  includes mappings  702   a - b  between I/O ports  628   a - b  and partitions  704   a - b , respectively. Note that there are two partitions  704   c - d  to which neither of the I/O ports  628   a - b  is mapped. Other I/O ports in other I/O controllers (not shown), however, may be mapped to partitions  704   c - d . Although in the particular example illustrated in  FIG. 7  there are two I/O ports  628   a - b  allocated to two partitions  704   a - b , there may be any number of I/O ports and any number of partitions mapped to each other in any arrangement. 
     The I/O controller  602  includes a destination decoder  608 , which verifies that incoming transactions (on lines  610 ) are addressed to one of the I/O devices  604   a - b  controlled by the I/O controller  602 . If an incoming transaction is not addressed to one of the I/O devices  604   a - b , the destination decoder  608  does not transmit the transaction further within the I/O controller  602 . 
     Referring to  FIG. 8 , a flowchart is shown of a method  800  that is performed by the destination decoder  608  when an incoming transaction  612  is received on lines  610  in one embodiment of the present invention. The destination decoder  608  receives the incoming transaction  612  (step  802 ). In one embodiment of the present invention, the transaction  612  includes (1) a source terminus identifier (e.g., the terminus ID of the device that originated the transaction  612 ), represented as “S” in  FIG. 6A ; (2) the physical address to access, represented as “a” in  FIG. 6A ; (3) the transaction type (e.g., read or write), represented as “T” in  FIG. 6A ; and (4) data associated with the transaction (e.g., data to write if the transaction  612  is a write command), represented as “d” in  FIG. 6A . 
     The destination decoder  608  examines the source terminus ID in transaction  612  to determine whether the device that transmitted the transaction  612  is allocated to any of the partitions to which the I/O ports  628   a - b  are allocated (step  804 ). If the transaction  612  was not transmitted by such a device, the transaction is not authorized to access the devices  604   a - b , and the destination decoder  608  does not transmit the transaction  608  to the I/O devices  604   a - b  (step  806 ). 
     More specifically, the destination decoder  608  may maintain a list  614  of valid source terminus IDs. The list  614  may contain the source terminus IDs of those devices in the system  100  that are allocated to any of the partitions  704   a - b  to which the I/O ports  628   a - b  are allocated. The destination decoder  608  may perform step  804  by determining whether the source terminus ID in transaction  612  is in the list  614  and by then determining that the transaction  612  is not from an appropriate partition if the source terminus ID is not in the list  614 . 
     If the destination decoder  608  determines in step  804  that the transaction  612  is from an appropriate device, the destination decoder  608  maps the source terminus ID to the partition ID value of the one of the I/O ports  628   a - b  that is in the same partition as the device that transmitted the transaction  612  (step  808 ). The destination decoder  608  may maintain a table  616  or other mapping of source terminus identifiers to partition ID register values. The destination decoder  608  may therefore perform step  808  by using the source terminus ID in transaction  612  as an index into the table  616  and thereby identifying the corresponding partition ID register value. 
     The destination decoder  608  generates a first modified transaction  620  that contains: (1) the partition ID register value (p) identified in step  808 ; (2) the physical address (a) contained in the transaction  612 ; and (3) the data (d) contained in the transaction  612 . The destination decoder  608  transmits the first modified transaction  620  to a transaction router  622  on lines  618  (step  810 ). 
     The transaction router  622  routes the transaction  620  to the one of the I/O ports  628   a - b  that is allocated to the partition identified in the first modified transaction  620  (step  812 ). More specifically, the transaction router  622  identifies the one of the I/O ports  628   a - b  that is allocated to the partition ID contained in the first modified transaction  620  (step  814 ). The transaction router  622  may, for example, contain a lookup table that maps partition IDs to I/O ports  628   a - b , and may use that lookup table to perform step  814 . The transaction router  622  may generate a second modified transaction by stripping the partition ID from the first modified transaction  620  and then transmit the second modified transaction to the device identified in step  814  (step  816 ). 
     In one embodiment of the present invention, I/O ports  628   a - b  may either: (1) both be allocated to partition  704   a ; or (2) be separately allocated to partitions  704   a - b  in the manner illustrated in  FIG. 7 . To enable the I/O controller  602  to implement either such partitioning of the I/O ports  628   a - b , I/O controller  602  includes switch  632 . I/O device  604   a  is coupled to switch  632  over lines  630   a  and I/O device  604   b  is coupled to switch  632  over lines  630   b . Switch  632  is in turn coupled to I/O port  628   a  over lines  630   c . In one embodiment of the present invention, switch  632  creates a permanent pass-through connection between I/O device  604   a  and I/O port  628   a . As a result, I/O device  604   a  communicates with I/O controller  602  through I/O port  628   a . Transaction router  622  may be configured to route transactions associated with partition  704   a  to I/O port  628   a  and thereby to implement the allocation of I/O device  604   a  to partition  704   a.    
     If both I/O ports  628   a - b  are allocated to partition  704   a , I/O port  628   b  may be disabled and the switch  632  may be set to a first setting which routes all communications to and from I/O device  604   b  through I/O port  628   a . If I/O port  628   a  is allocated to partition  704   a  and I/O port  628   b  is allocated to partition  704   b  (as shown in  FIG. 7 ), then I/O port  628   a  may be enabled and the switch  632  may be set to a second setting which routes all communications to and from I/O device  604   b  through I/O port  628   b . Note that use of the switch  632  in the manner described above is merely one example of a way in which a transaction may be decoded and routed to a specific port, and does not constitute a limitation of the present invention. 
     Returning to step  812  of method  800 , the transaction router  622  may maintain a mapping of partition ID values and associated I/O ports. For example, consider the case in which I/O device  604   a  is mapped to partition  704   a  and in which I/O device  604   b  is mapped to partition  704   b  (as shown in  FIG. 7 ). In such a case, if the partition ID in the first modified transaction  620  identifies partition  704   a , the transaction router  622  may generate and transmit a second modified transaction  626   a  to I/O port  628   a  on lines  624   a , through which the second modified transaction  628   a  may be forwarded to I/O device  604   a  on lines  630   c , through switch  632 , and then on lines  630   a . Similarly, if the partition ID in the first modified transaction  620  identifies partition  704   b , the transaction router  622  may generate and transmit a second modified transaction  626   b  to I/O port  628   b  on lines  624   b , through which the second modified transaction  626   b  may be forwarded to I/O device  604   b  on lines  630   b . Note that the mapping  700  illustrated in  FIG. 7 , in which there is a one-to-one mapping between ports  628   a - b  and partitions  702   a - b , is provided merely as an example and does not constitute a limitation of the present invention. Techniques disclosed herein may, for example, be used in conjunction with mappings of multiple ports to a single partition, as may be accomplished by using additional bits of the physical address as part of the partition ID. 
     Examples of techniques will now be described for enabling the I/O devices  604   a - b  to perform outgoing communications through the I/O controller  602  when the I/O devices  604   a - b  are allocated to different partitions. Assume once again that I/O port  628   a  (and therefore I/O device  604   a ) is mapped to partition  704   a  and that I/O port (and therefore I/O device  604   b ) is mapped to partition  704   b  (as shown in  FIG. 7 ). Now consider an example in which an outgoing transaction  636   a  is generated by I/O device  604   a  on lines  634   a  (through I/O port  628   a ). Transaction  636   a  includes a physical address (a) and data (d). 
     I/O controller  602  includes a plurality of partition ID registers  606   a - b  associated with the I/O ports  628   a - b , respectively. In particular, partition ID register  606   a  is associated with I/O port  628   a  and represents mapping  702   a  ( FIG. 7 ). Similarly, partition ID register  606   b  is associated with I/O port  628   b  and represents mapping  702   b . Each of the partition ID registers  606   a - b  includes at least enough bits to distinguish among the partitions to which I/O ports  628   a - b  are allocated. 
     Each of the partition ID registers  606   a - b  stores a unique partition ID value that uniquely identifies the partition to which the corresponding one of the I/O ports  628   a - b  is allocated. For example, referring again to the example illustrated in  FIG. 7 , the value 0 (binary 00) may be stored in partition ID register  606   a , thereby indicating that I/O port  628   a  is allocated to partition  0  ( 704   a ). Similarly, the value 1 (binary 01) may be stored in partition ID register  606   b , thereby indicating that I/O port  628   b  is allocated to partition  1  ( 704   b ). The I/O controller  602  may be configured so that the partition ID values stored in the partition ID registers  606   a - b  cannot be changed by the operating system executing on the computer system  100 . 
     Referring to  FIG. 9 , a flowchart is shown of a method  900  that is performed by bit substitution circuit  638   a  according to one embodiment of the present invention when outgoing transaction  636   a  is transmitted on lines  636   a  by device  604   a . The bit substitution circuit  638   a  receives the outgoing transaction  636   a  (step  902 ). In response to receiving the transaction  636   a , the bit substitution circuit  638   a  reads the partition ID value from partition ID register  606   a  on lines  640   a  (step  904 ). The bit substitution circuit  638   a  writes the partition ID value into the physical address, thereby producing a partition-identifying address (step  906 ). 
     The partition-identifying address produced in step  906  may, for example, have the layout illustrated in  FIG. 12B . The example partition-identifying address  1210  illustrated in  FIG. 12B  is 64 bits wide. Portion  1212  (bits  0 - 54 ) of partition-identifying address  1210  contains the physical address contained in the original transaction  636   a . In one embodiment of the present invention, bit substitution circuit  638   a  writes the partition ID value obtained from transaction  636   a  into portion  1214  (bit  55 ) of the partition-identifying address  1210  (step  906 ). In other words, bit substitution circuit  638   a  appends the partition ID value to the original physical address. Portion  1218 , which includes both portions  1212  and  1214 , therefore unambiguously identifies the system memory address indicated by the original transaction  636   a . Portion  1216  (bits  56 - 63 ) of the partition-identifying address  1200  are unused. Portion  1218  therefore represents the “used portion” of address  1210  because the combination of the partition ID portion  1214  and the physical address portion  1212  are used to specify a unique address in the system  100 . 
     Note that the partition ID field  1214  of address  1210  is only one bit wide, in contrast to the partition ID field  1204  of address  1200  ( FIG. 12A ), which is two bits wide. The partition ID field  1214  of address  1210  need only be wide enough to distinguish among the partitions to which I/O ports  628   a - b  are allocated. Because I/O ports  628   a - b  are allocated to two ports in the example illustrated in  FIGS. 6A-6B , partition ID field  1214  need only be one bit wide. Partition ID field  1204  of address  1200  ( FIG. 12A ), in contrast, is two bits wide because it must be capable of distinguishing among all partitions  504   a - d  in the system. The required minimum width of the partition ID fields  1204  and  1214  may, of course, vary depending on the number of unique partitions they are required to represent. 
     The particular layout of the partition-identifying address  1210  in  FIG. 12B  is shown merely for purposes of example and does not constitute a limitation of the present invention. Rather, partition-identifying addresses of any size and having any layout may be used in conjunction with embodiments of the present invention. The bit substitution circuit  638   a  generates a first modified transaction  642   a  containing the partition-identifying address generated in step  906  (step  908 ). The bit substitution circuit  638   a  transmits the first modified transaction  642   a  to cache  646  on lines  644   a  (step  910 ). 
     Referring to  FIG. 10 , a flowchart is shown of a method  1000  that is performed by the cache  646  in response to receipt of the first modified transaction  642   a  according to one embodiment of the present invention. The cache  646  receives the first modified transaction  642   a  from the bit substitution circuit  638   a  (step  1002 ). The cache  646  determines whether the first modified transaction  642   a  can be satisfied using cache data stored locally in cache lines  648  (step  1004 ). If there is a cache hit, the cache  646  performs the transaction locally (i.e., within the cache lines  648 ) (step  1006 ) and the method  1000  terminates. Data is written into the cache from the IO card via lines  650 . If the transaction  636   a  is a read command, the cache  646  may transmit the read values to the device  604   a  on lines  650 . 
     If there is a cache miss, the cache  646  transmits a second modified transaction  654  to an address mapper  656  on lines  654  (step  1008 ). In one embodiment of the present invention, the second modified transaction  652  contains the partition ID value and physical address from the first modified transaction  642   a.    
     Referring to  FIG. 11 , a flowchart is shown of a method  1100  that is performed by the address mapper  656  when it receives the second modified transaction  652  in one embodiment of the invention. The address mapper  656  receives the second modified transaction  652  on lines  654  (step  1102 ). The address mapper  656  maintains a mapping  658  of address-partition ID pairs to destination terminus IDs. The address mapper  656  uses the mapping  658  to map the partition ID and address in the second modified transaction  652  into a destination terminus ID (step  1104 ). 
     The address mapper  656  generates and transmits a third modified transaction  670  to the system fabric  116  on lines  672  (step  1106 ). The third modified transaction  670  includes: (1) the destination terminus ID identified in step  1104 ; (2) the physical address from the second modified transaction  652 ; and (3) the data from the second modified transaction  652  (if any). Note that the third modified transaction  670  does not include the partition ID identified in step  904  ( FIG. 9 ), because in the embodiment illustrated in  FIGS. 6A-6B  the partition ID is only used to distinguish internally (i.e., within the I/O controller  602 ) among different partitions. 
     As described above, router  228  routes the third modified transaction  670  to the memory controller or other device having the destination terminus ID contained in the third modified transaction  670  using the techniques described above with respect to  FIG. 2 . 
     Although the examples described above relate to partition  704   a  and corresponding I/O port  628   a , the same or similar techniques may be used in conjunction with partition  704   b  and corresponding I/O port  628   b . For example, bit substitution circuit  638   b  may receive outgoing transaction  636   b  from device  604   b  on lines  634   b  and substitute therein the value of partition ID register  606   b , thereby generating and transmitting a first modified transaction  642   b  on lines  644   b . The first modified transaction  642   b  may then be processed in the manner described above. 
     Among the advantages of the invention are one or more of the following. 
     Existing partitionable computer architectures typically allocate resources to partitions on a per-chip basis. In other words, in a conventional partitionable computer, all of the resources (such as processor cores) in a single chip must be allocated to at most one partition. As the number and power of resources in a single chip increases, such per-chip resource allocation imposes limitations on the degree of granularity with which resources may be allocated to partitions in a partitionable computer system. Such limitations limit the extent to which resources may be dynamically allocated to partitions in a manner that makes optimal use of such resources. 
     The techniques disclosed herein address this problem by providing the ability to allocate resources on a sub-chip basis. The ability to allocate multiple resources on a single chip to multiple partitions increases the degree to which such resources may be allocated optimally in response to changing conditions. Sub-chip partitioning allows partitionable computer systems to take full advantage of the cost and size reductions made possible by the current trend in computer chip design of providing an increasing number of functions on a single chip, while still providing the fine-grained resource allocation demanded by users. 
     Furthermore, embodiments of the present invention enable sub-chip partitioning to be accomplished using relatively localized modifications to existing circuitry, thereby enabling a substantial portion of existing circuitry to be used without modification in conjunction with embodiments of the present invention. For example, in the system illustrated in  FIG. 2 , the cores  204   a - n , cache  208 , and fabric  116  may be prior art components. As a result, embodiments of the present invention may be implemented relatively easily, quickly, and inexpensively. 
     A further advantage of techniques disclosed herein is that the bit substitution circuits  212   a - n  and  638   a - b  may enforce inter-partition security by preventing the operating system in the corresponding partition from accessing addresses in other partitions. As described above, such security may be provided by overwriting any values the operating system writes into the upper bits of addresses it generates (e.g., bits in portions  1204  or  1206  of address  1200  ( FIG. 12A ) and bits in portions  1214  or  1216  of address  1210  ( FIG. 12B )). The techniques disclosed herein thereby provide a degree of hardware-enforced inter-partition security that cannot be circumvented by malicious or improperly-designed software. 
     It is to be understood that although the invention has been described above in terms of particular embodiments, the foregoing embodiments are provided as illustrative only, and do not limit or define the scope of the invention. Various other embodiments, including but not limited to the following, are also within the scope of the claims. For example, elements and components described herein may be further divided into additional components or joined together to form fewer components for performing the same functions. 
     The term “resources” refers herein to hardware resources in a computer system, such as processor cores ( FIG. 2 ) and I/O ports ( FIG. 6A-6B ). A chip may contain one or more hardware resources. Although processor cores and I/O ports are provided herein as examples of hardware resources that may individually be allocated to partitions in embodiments of the present invention, embodiments of the present invention may be used to allocate other kinds of hardware resources to partitions on a sub-chip basis. Furthermore, the techniques illustrated by the example in  FIG. 2  are applied to a plurality of CPU cores  206   a - n , such techniques may be applied to I/O ports or to any other kind of resource. Similarly, although the techniques illustrated by the example in  FIGS. 6A-6B  are applied to a plurality of I/O ports  628   a - b , such techniques may be applied to CPU cores or to any other kind of resource. 
     In general, techniques disclosed herein may be used in system including a cache to allocate the cache among multiple partitions. Furthermore, any resource which is accessed using memory-mapped transactions may be allocated to a particular partition in a partitionable computer system using techniques disclosed herein. 
     For example, general purpose event registers (GPEs) typically are allocated to particular partitions. A particular GPE, therefore, typically is addressable within the address space of the partition to which it is allocated. Techniques disclosed herein may be employed to enable the GPEs of each partition accessible over the system fabric  116  at unique system (fabric) addresses. 
     Although certain examples provided above involving allocating a plurality of resources on a single chip (integrated circuit) to a plurality of partitions, the techniques disclosed herein are not limited to use in conjunction with resources on a single chip. Rather, more generally, techniques disclosed herein may be used to allocated a plurality of resources in a computer system to a plurality of partitions in the computer system. 
     Although only a single memory controller is shown in each of the cell boards  102   a - d  in  FIG. 1 , this is not a requirement of the present invention. Rather, a cell board may contain multiple memory controllers, each of which may have its own terminus ID. Those of ordinary skill in the art will appreciate how to implement embodiments of the present invention in systems including multiple memory controllers on a single cell board. 
     Although in the example illustrated in  FIG. 2  the core  204   a  issues memory write command  230   a , the memory write command  230   a  is just one example of a memory access request, which is in turn merely one example of a transaction to which the techniques disclosed herein may apply. 
     Although partition ID values are stored in partition ID registers  210   a - n  in  FIG. 2 , partition ID values may be represented and stored in any manner. For example, partition ID values need not each be stored in a distinct register and need not be represented using the particular numbering scheme described herein. 
     Although various embodiments of the present invention are described herein in conjunction with symmetric multiprocessor computer architectures (SMPs), embodiments of the present invention are not limited to use in conjunction with SMPs. Embodiments of the present invention may, for example, be used in conjunction with NUMA (non-uniform memory access) multiprocessor computer architectures. 
     Although four cell boards  102   a - d  are shown in  FIG. 3 , this is not a requirement of the present invention. Rather, the techniques disclosed herein may be used in conjunction with multiprocessor computer systems having any number of cell boards. Furthermore, each cell board in the system may have any number of processors (including one). The term “cell board” as used herein is not limited to any particular kind of cell board, but rather refers generally to any set of electrical and/or mechanical components that allow a set of one or more processors to communicate over a system fabric through an interface such as an agent chip. 
     Although the fabric agent chip  114   a  and memory controller  110  are illustrated as separate and distinct components in  FIG. 1 , this is not a requirement of the present invention. Rather, the fabric agent chip  114   a  and memory controller  110   a  may be integrated into a single chip package.