# Patent

## Title
Memory interleaving

## Abstract

Memory interleaving includes providing a non-power of two number of channels in a computing system and interleaving memory access among the channels.

## Background
BACKGROUND
A channel generally refers to a pathway between a computer system and other computing systems and/or other devices. Each of a computing system's channels is an independent unit that can transfer data at the same time as other channels. Each channel is typically assigned a segment of memory address space and can transfer data corresponding to its assigned memory address space. In this way, the computing system's processor may access different segments of memory via different channels without idling while the memory completes an access to one segment before beginning another memory access. This type of memory access is generally called interleaving.



DESCRIPTION OF DRAWINGS
FIG. 1 is a block diagram of an example channel control system.
FIG. 2 is a flowchart of an example process of memory interleaving.
FIG. 3 is a flowchart of an example process of determining whether a region is within a channel.
FIG. 4 is a flowchart of an example process of reducing an address.
FIG. 5 is a block diagram of an example address reduction.
FIG. 6 is a flowchart of an example process of adjusting an address.
FIG. 7 is a flowchart of an example process of address remapping.
FIG. 8 is a block diagram of an example machine system.



DESCRIPTION
Referring to FIG. 1, an example channel control system 100 can interleave access to a memory using channel controllers 102(1)-102(X) each associated with one channel. X can equal any positive whole number greater than one, including non-power of two positive whole numbers (three, five, six, etc.). Whatever the value of X, interleaving may be performed without using one or more bits of a memory address to indicate which channel to use in accessing the memory. Because no address bits need to be used in channel selection, the number of channels for interleaving is not restricted to a power of two number of channels as in traditional channel interleaving. Addresses may be mapped to the channel controllers 102(1)-102(X), and accesses to the memory may be concurrently made for addresses mapped to different channels.
Each of the channel controllers 102(1)-102(X) includes one of X match detect mechanisms 104(1)-104(X) and one of X address and count remapping mechanisms 106(1)-106(X). Each of the channel controllers 102(1)-102(X) receives information regarding a region to access in a memory and determines whether its associated channel is mapped to access data included in that region. The channel controllers 102(1)-102(X) typically make their determinations in parallel although they could process the information according to some pre-programmed priority or ordering scheme.
The information received by the channel controllers 102(1)-102(X) can include a data pair including a start address indicating where to begin accessing data in the memory and a count indicating how much data to access starting at the start address. The count is typically provided in bytes although any data measurement or size indicator may be used. The start address and the count define a region beginning at the start address and extending for a number of bytes indicated by the count (or as otherwise indicated by the count depending on how the count is provided).
Each of the match detect mechanisms 104(1)-104(X) determines if its associated channel is mapped to any of the addresses included in the region. Addresses may be mapped to the channels using a traditional scheme that assigned segments of address space to channels and/or using a scheme described further below that can spread adjacent addresses over the channels such that adjacent or nearby addresses may be spread over multiple channels. The channel may be mapped to access no addresses included in the region, all addresses included in the region, or some of the addresses included in the region. If a channel is mapped to some of the addresses included in the region, then part of the region lies within the channel and at least two channels are mapped to that region and may both access the region.
If one of the match detect mechanism 104(1)-104(X) determines that part of the region lies within its associated channel, then the one of the address and count remapping mechanisms 106(1)-106(X) associated with that channel determines a remapped start address and a remapped count indicating a part of the access for its associated one of the channel controllers 102(1)-102(X) to complete. The address and count remapping mechanisms 106(1)-106(X) may determine remapped addresses and remapped counts while the match detect mechanisms 104(1)-104(X) determine which channel or channels (if any) map to the region to save on processing time.
Once the channel controllers 102(1)-102(X) have determined if their associated channels map to the region, the appropriate one or ones of the channels may access the data in indicated region. In this way, addresses may be mapped to channels such that multiple channels may be used to access data at relatively close addresses. Furthermore, an interleave scheme can include using two or more channels, including a non-power of two number of channels.
In other examples, the match detect mechanisms 104(1)-104(X) and/or the address and count remapping mechanisms 106(1)-106(X) may be external to the channel controllers 102(1)-102(X). Furthermore, some or all of the channel controllers 102(1)-102(X) may use the same one or ones of the match detect mechanisms 104(1)-104(X) and/or the address and count remapping mechanisms 106(1)-106(X).
FIG. 2 shows an example process 200 of memory interleaving. In the process 200, the channel controller 102 receives 202 address and count information about a region to access in memory. (Each of the channel controllers 102(1)-102(X) receives the same information.)
The channel controllers 102(1)-102(X), the match detect mechanisms 104(1)-104(X), and the address and count remapping mechanisms 106(1)-106(X) each function similar to their like-named counterparts. For simplicity, the match detect mechanism 104(1) (“match detect 104”) and the address and count remapping mechanism 106(1) (“remap 106”) included in the channel controller 102(1) (“channel controller 102”) are used as representative examples.
The match detect 104 determines 204 how much of the region lies with the channel associated with the channel controller 102, channel one in this example. An example of how the match detect 104 may make such a determination is discussed further below.
If none of the region lies within the channel, then the process 200 ends 206 because the data to be accessed cannot be accessed through that channel.
If the region lies entirely within the channel, then the channel controller 102 triggers 208 an access of an amount of data equal to the count starting at the start address through the channel. The channel controller 102 may itself retrieve the data.
If the region lies partially within the channel, then the remap 106 adjusts 210 the address and the count to an adjusted address and adjusted count so that the channel only accesses data in its assigned region. The channel controller 102 may then trigger 212 an access of an amount of data equal to the adjusted count starting at the adjusted start address through the channel. The channel controller 102 may itself retrieve the data. For example, the start address may be indexed within one channel but the count extends the region into an area mapped to another channel, and the other channel needs to adjust the start address to reflect where the region begins in its indexed area. The adjusted start address may be the same as the start address while the adjusted count differs from the count in the case where the count extends beyond a channel's mapped area.
FIG. 3 shows an example determination process 300 of how the match detect 104 may determine whether and how much of a region lies within a channel. In the determination process 300, the match detect 104 calculates 302 an upper address of the region corresponding to the end of the data included in the region. The match detect 104 may calculate the upper address as the start address plus the count minus one. The one is subtracted to account for the data at the start address.
The match detect 104 reduces 304 the upper address and the start address to two bits each. With the addresses reduced to two bits each, the match detect 104 can determine if the region lies at least partially within the channel associated with the match detect. The addresses may be reduced to two bits each because in an example including three channels (X equals three), each of the channels can be represented by a different two bit combination (e.g., “01” for channel one, “10” for channel two, and “11” for channel three) and the two bit representation of the channel including the address can be used in reducing the address to help determine whether the channel associated with the match detect 104 is mapped to the region. If the system includes more than three channels, the addresses may be reduced to more than two bits because more than two bits may be necessary to represent each of the different channels. An example of how the match detect 104 may perform a reduction is discussed further below.
The match detect 104 determines 306 if the channel number associated with the match detect 104 (one in this example) matches either the reduced upper address or the reduced start address. If not, then none of the region lies within the channel.
If either reduced address matches the channel number, then the match detect 104 determines 308 if both the reduced upper address and the reduced start address match the channel number. If so, then the entire region lies within the channel. If not, then only one of the reduced addresses matches the channel number and only part of the region lies within the channel. The match detect 104 determines 310 if the reduced start address matches the channel number. If so, then a lower part of the data transfer (access), a part starting at the start address, lies within the channel. If not, then the match detect 104 concludes that an upper part of the data transfer, a part starting at an address higher than the start address and continuing through the upper address (unless the starting address of the upper part is the upper address in which case the upper part of the data transfer includes only the upper address), lies within the channel. The match detect 104 in this example checks for a match with the start address and makes an assumption about an upper address match based on that determination, but the match detect 104 may check the upper address and make an assumption about the start address in other examples.
FIG. 4 shows an example reduction process 400 that the match detect 104 may use to reduce an address. In the reduction process 400 generally, as shown in an example address reduction 500 in FIG. 5, the match detect 104 takes five levels of gating to reduce a thirty-one bit address 502 to a two bit output number 504. The match detect 104 may ignore one or more bits included in the address 502 that are not part of the start or upper address. In this example showing a start address, the match detect 104 ignores the seven bits included in the address that represent a byte offset (the interleave byte size) and considers the twenty-four bit start address. One of the gating levels takes as an input a channel number 506 of the channel that holds the address.
Referring back to FIG. 4, the match detect 104 recodes 402 the address using 2-bit to 2-bit recoding to produce a first number of bits including as many bits as the address (twenty-four bits) according to the following table:
















Input

Output













n + 1
n
2
1







0
0
0
0



0
1
0
1



1
0
1
0



1
1
0
0










The match detect 104 reduces 404 the first number of bits using a 4-bit to 2-bit reducer to produce a second number of bits including the half as many bits as the first number of bits (twenty-four bits reduced to twelve bits) according to the following table:

















00
01
10









00
00
01
10




(0)
(1)
(2)



01
01
10
00




(1)
(2)
(0)



10
10
00
01




(2)
(0)
(1)










The match detect 104 reduces 406 the second number of bits according to the above 4-bit to 2-bit reducer table to produce a third number of bits including half as many bits as the second number of bits (twelve bits reduced to six bits).
The match detect 104 reduces 408 the third number of bits plus the channel number including the address according to the above 4-bit to 2-bit reducer table to produce a fourth number of bits (eight bits reduced to four bits). If the match detect 104 did not include the channel number as an input in this reduction, there would be two unused inputs. The match detect 104 may receive the channel number from the remap 106, may look it up in an address mapping table that indexes addresses and their corresponding channel numbers, or may otherwise acquire the channel number.
The match detect 104 adds 410 the two resulting 2-bit recoded numbers. Sums of zero, one, and two do not change while a sum of four adds to zero. This addition produces the final two-bit output of the reduction.
The match detect 104 determines 412 if the final output indicates a match with the channel associated with the match detect 104. A final output of zero (00) indicates a match while a one (01) or a two (10) indicates a non-match. Including the channel number associated with the address (the start address or the upper address) as an input in the reduction process allows the match detect 104 to make a match determination from the final bit output of the reduction process.
FIG. 6 shows an example adjustment process 600 that the remap 106 may use to adjust the start address and the count if the match detect 104 determines that only part of the region lies within its associated channel (see FIG. 2).
In the adjustment process 600, the remap 106 calculates 602 a boundary address representing an address an interleaved amount beyond the start address. The interleaved amount is typically included in the address that includes the start address (see, e.g., the address 502 in FIG. 5), and in this example equals one hundred twenty-eight bytes. The remap 106 may calculate the boundary address by performing a logical AND operation on the start address and 0xFFFFFF80 and adding 0x80 (the interleaved amount).
The remap 106 also calculates 604 a lower count representing a number of bytes between the start address and the boundary address. The remap 106 may calculate the lower count by subtracting the start address from the boundary address.
The remap 106 also calculates 606 an upper count representing a number of bytes between the boundary address and a stop address (the start address plus the count). The remap 106 may calculate the upper count by subtracting the boundary address from the stop address and adding one (the one accounts for the fact that the lower count includes the boundary address).
If the channel associated with the remap 106 owns a lower part of the transfer (e.g., as determined through the determination process 300), then the remap 106 considers 608 the adjusted start address as the start address and the adjusted count as the lower count. If not (i.e., if the channel associated with the remap 106 owns an upper part of the transfer), then the remap 106 considers 610 the adjusted start address as the boundary address and the adjusted count as the upper count.
FIG. 7 shows an example remapping process 700 that the remap 106 may use to index an address within a channel. The remap 106 finds 702 a longest string of consecutive address bits with a value of one in the address to be indexed. The remap 106 may begin searching for the longest string with the least significant bit in the address. Once found, the remap 106 drops 704 the longest string from the address, right justifies 706 the remaining bits in the address. If the channel includes a power of two number of memory locations (e.g., addresses), then the remap 106 fills 708 the vacated bit positions (the most significant bits) with ones. This filling essentially adds three quarters to the remaining space of channels, e.g., starts the filling three quarters over from the address. The resulting bits form the remapped address.
If the channel is of a non-power of two, then a shifting of three quarters is not typically the same size as for a power of two channels (e.g., may not be two bit positions but some other number). The remap 106 determines 710 where to start adding ones in the vacated positions. The remap 106 may make such a determination by consulting one or more lookup tables including constants that indicate start addresses where the remap 106 may start filling 712 in ones. Each of the tables may be included in the remap 106 or be otherwise accessible to the remap 106 and may include constants for any number of channels and for any number of shifted bits. The resulting bits form the remapped address.
For example, in a three channel system, the lookup tables may be as follows, where the constant values are shown in hexadecimal format and where KO represents three quarters of a channel, K1 represents three quarters plus three quarters of a channel K2 represents three quarters plus three quarters plus three quarters of a channel, etc. The tables show constant values up to 768 MBytes, but the values in the table may be scaled as appropriate for a larger number of MBytes.















MBytes
















32
64
96
128
160
192
224
256





K11
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A


K10
N/A
N/A
N/A
N/A
13FFFF
17FFFF
1BFFFF
1FFFFF


K9
N/A
7FFFF
BFFFF
FFFFF
13FFFE
17FFFE
1BFFFE
1FFFFE


K8
3FFFF
7FFFE
BFFFD
FFFFC
13FFFB
17FFFA
1BFFF9
1FFFF8


K7
3FFFC
7FFF8
BFFF4
FFFF0
13FFEC
17FFE8
1BFFE4
1FFFE0


K6
3FFF0
7FFE0
BFFD0
FFFC0
13FFB0
17FFA0
1BFF90
1FFF80


K5
3FFC0
7FF80
BFF40
FFF00
13FEC0
17FE80
1BFE40
1FFE00


K4
3FF00
7FE00
BFD00
FFC00
13FB00
17FA00
1BF900
1FF800


K3
3FC00
7F800
BF400
FF000
13EC00
17E800
1BE400
1FE000


K2
3F000
7E000
BD000
FC000
13B000
17A000
1BE9000
1F8000


K1
3C000
78000
B4000
F0000
12C000
168000
1A4000
1E000


K0
30000
60000
9000
C0000
F0000
120000
150000
180000












MBytes
















288
320
352
384
416
448
480
512





K11
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A


K10
23FFFF
27FFFF
2BFFFF
2FFFFF
33FFFF
37FFFF
3BFFFF
3FFFFF


K9
23FFFD
27FFFD
2BFFFD
2FFFFD
33FFFC
37FFFC
3BFFFC
3FFFFC


K8
23FFF7
27FFF6
2BFFF6
2FFFF4
33FFF3
37FFF2
3BFFF1
3FFFF0


K7
23FFDC
27FFD8
2BFFD4
2FFFD0
33FFCC
37FFC8
3BFFC4
3FFFC0


K6
23FF70
27FF60
2BFF50
2FFF40
33FF30
37FF20
3BFF10
3FFF00


K5
23FDC0
27FD80
2BFD40
2FFD00
33FCC0
37FC80
3BFC40
3FFC00


K4
23F700
27F600
2BF500
2FF400
33F300
37F200
3BF100
3FF000


K3
23DC00
27D800
2BD400
2FD000
33CC00
37C800
3BC400
3FC000


K2
237000
276000
2B5000
2F4000
333000
372000
3B1000
3F0000


K1
21C000
258000
294000
2D0000
30C000
348000
384000
3C0000


K0
1B0000
1E0000
210000
240000
270000
2A0000
2D0000
300000












MBytes
















544
576
608
640
672
704
736
768





K11
43FFFF
47FFFF
4BFFFF
4FFFFF
53FFFF
57FFFF
5BFFFF
5FFFFF


K10
43FFFE
47FFFE
4BFFFE
4FFFFE
53FFFE
57FFFE
5BFFFE
5FFFFE


K9
43FFFB
47FFFB
4BFFFB
4FFFFB
53FFFA
57FFFA
5BFFFA
5FFFFA


K8
43FFEF
47FFEE
4BFFED
4FFFEC
53FFEB
57FFEA
5BFFE9
5FFFE8


K7
43FFBC
47FFB8
4BFFB4
4FFF80
53FFAC
57FFA8
5BFFA4
5FFFA0


K6
43FEF0
47FEE0
4BFED0
4FFEC0
53FEB0
57FEA0
5BFE90
5FFE80


K5
43FBCO
47FB80
4BFB40
4FFB00
53FAC0
57FA80
5BFA40
5FFA00


K4
43EF00
47EE00
4BED00
4FEC00
53EB00
57EA00
5BE900
5FE800


K3
43BC00
47B800
4BB400
4FB000
53AC00
57A800
5BA400
5FA000


K2
43BC00
47B800
4BB400
4FB000
53AC00
57A800
5BA400
5FA000


K1
3FC000
438000
474000
4B0000
4EC000
528000
564000
5A0000


K0
330000
360000
390000
3C0000
3F0000
420000
450000
480000









The remap 106 indexes all available addresses, e.g., all addresses handled by the channel control system 100 (see FIG. 1). As a simple example, in a system including three channels with eight addresses each and twenty-four addresses, address locations in the channels would be remapped as shown:
















Index in






Channel
Channel 0
Channel 1
Channel 2





















0
0
1
2



1
6
4
5



2
9
10
8



3
12
13
14



4
18
16
17



5
21
22
20



6
3
7
11



7
15
19
23










Although the processes in FIGS. 2, 3, 4, 6, and 7 are each described with reference to the elements included in the example channel control system 100 of FIG. 1, these or similar processes, each including the same, more, or fewer elements, reorganized or not, may be performed in the channel control system 100 or in another, similar system. Additionally, the processes in FIGS. 2, 3, 4, 6, and 7 are described systems using 128-byte interleaving and including three channels and thirty-one bit addresses, but the processes may be used (with any appropriate modifications) for any size interleaving, for any number of channels, and for addresses of any size. Furthermore, the processes in FIGS. 2, 3, 4, 6, and 7 need not all be performed together in the same system but may find applicability alone or in partial combination with two or more of the other processes.
Referring to FIG. 8, a machine 800 includes a processing system 802 including a memory controller 804 that may include or be configured similar to the channel control system 100 (see FIG. 100). The elements described with reference to FIG. 8 can be implemented in a variety of ways.
A consuming device 806 may need information stored at a location in a main memory 808. The consuming device 806 typically connects to the machine 800 via input/output (I/O) ports, bays, and/or slots 810 and requests data from the main memory 808 through a chipset 812 and a processor 814.
The memory controller 804 may control access to and map addresses in the main memory 808 as described above, interleaving reads/writes using multiple memory channels. The main memory 808 can include any memory mechanism capable of storing data. Examples of the main memory 808 include random access memory (RAM) such as dynamic RAM or static RAM, read only memory (ROM), flash memory, tapes, disks, buffers, and other types of similar storage mechanisms. The main memory 808 may include one storage mechanism, e.g., one RAM chip, or any combination of storage mechanisms, e.g., multiple RAM chips. For example, memory may include SDRAM. SDRAM generally refers to a type of DRAM that can run at much higher clock speeds than conventional memory. SDRAM can synchronize itself with a bus associated with a processor included in the computing system (e.g., the processor 814). DDR-SDRAM generally refers to a type of SDRAM that supports data transfers on both edges of each clock cycle (the rising and falling edges), effectively doubling the memory's data throughput.
The machine 800 can include any mechanism or device capable of processing data. Examples of the machine 800 include workstations, stationary personal computers, mobile personal computers, servers, personal digital assistants, pagers, telephones, and other similar mechanisms and devices.
The consuming device 806 can include an I/O device, network interface, or other mechanism that may communicate with or be included in the machine 800. I/O devices generally include devices used to transfer data into and/or out of a computer system. Examples of I/O devices include mice, keyboards, printers, display devices such as monitors, disk drives, graphics devices, joysticks, paddles, Zip drives, scanners, CD drives, DVD drives, modems, cameras, video devices, microphones, and other similar types of internal, external, and internal/external devices. One consuming device is shown but the machine 800 may communicate with more than consuming device.
The I/O ports, bays, and/or slots 810 may include any mechanisms or interfaces capable of connecting one or more consuming devices to the machine 800. For example, the I/O ports, bays, and/or slots 810 may include peripheral component interconnect (PCI) slots, parallel ports, serial bus ports, disk drive bays, and other similar types of mechanisms and interfaces.
The processor 814 can include any processing mechanism such as a microprocessor or a central processing unit (CPU). The processor 814 may include one or more individual processors. The processor 814 may include a network processor, a general purpose embedded processor, or other similar type of processor.
The chipset 812 can include any number of chips/integrated circuits that can provide interfaces between the machine's subsystems.
Instructions and data are typically communicated to and from the main memory 808 in blocks. A block generally refers to a collection of bits or bytes communicated or processed as a group. A block may include any number of words, and a word may include any number of bits or bytes.
Data can be communicated between elements on communication links. The communication links can include any kind and any combination of communication links such as buses (of any type and size), physical ports, wireless links, and other similar links. For bus communication links, the buses can have any width, e.g., sixteen bits, thirty-two bits, sixty-four bits, etc, and may run at any speed, e.g., thirty-three Mega Hertz (MHz), 100 MHz, etc. A bus may have a sideband feature in which the bus includes parallel channels that can each simultaneously carry data and/or address information. Additionally, each of the communication links may include one or more individual communication links.
The memory controller 804 generally includes any mechanism capable of communicating with and managing the main memory 808. The memory controller 804 may include one or more chips and may be included in the chipset 812 or may be a mechanism independent from the chipset 812. The memory controller 804 may include any number of and any type of instructions, routines, applications, and/or programs.
Furthermore, the machine 800 is simplified for ease of explanation. The machine 800 may include more or fewer additional elements such as communication links, processors, storage mechanisms (buffers, caches, memories, databases, etc.), display mechanisms, consuming devices, bridges, chips, and other similar types of machine elements.
The techniques described here are not limited to any particular hardware or software configuration; they may find applicability in any computing or processing environment. The techniques may be implemented in hardware, software, or a combination of the two. The techniques may be implemented in programs executing on programmable machines such as mobile computers, stationary computers, personal digital assistants, and similar devices that each include a processor, a storage medium readable by the processor (including volatile and non-volatile memory and/or storage elements), at least one input device, and one or more output devices. Program code is applied to data entered using the input device to perform the functions described and to generate output information. The output information is applied to one or more output devices.
Each program may be implemented in a high level procedural or object oriented programming language to communicate with a machine system. However, the programs can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language.
Each such program may be stored on a storage medium or device, e.g., compact disc read only memory (CD-ROM), hard disk, magnetic diskette, or similar medium or device, that is readable by a general or special purpose programmable machine for configuring and operating the machine when the storage medium or device is read by the computer to perform the procedures described in this document. The system may also be considered to be implemented as a machine-readable storage medium, configured with a program, where the storage medium so configured causes a machine to operate in a specific and predefined manner.
Other embodiments are within the scope of the following claims.

## Claims

1. A method comprising:
mapping an address to one of a plurality of memory channels by 
determining a longest string of consecutive bits having a value of one in the address, 
dropping the longest string from the address, 
justifying remaining bits in the address, and 
filling vacated bits in the address with ones to create a remapped address indexed to one of the memory channels.

2. The method of claim 1 further comprising providing a non-power of two number of memory channels.

3. The method of claim 1 further comprising distributing an equal number of remapped addresses to each of the memory channels.

4. The method of claim 1 in which the address includes a main memory address.

5. The method of claim 1 in which the address includes any address that may be mapped to a machine resource.

6. A The method of claim 1 further comprising filling the vacated bits starting at a start address, the start address equaling the address plus a constant value.

7. The method of claim 6 further comprising, if there are a power of two number of memory channels, providing a constant value of three quarter.

8. The method of claim 6 further comprising, if there are a non-power of two number of memory channels, determining the constant value by consulting a table including constant values for different numbers of memory channels.

9. An article comprising a machine-accessible medium which stores executable instructions, the instructions causing a machine to:
map an address to one of a plurality of memory channels by 
determining a longest string of consecutive bits having a value of one in the address, 
dropping the longest string from the address, 
justifying remaining bits in the address, and 
filling vacated bits in the address with ones to create a remapped address.

10. The article of claim 9 further causing a machine to distribute an equal number of remapped addresses to each of the memory channels.

11. The article of claim 9 in which the address includes a main memory address.

12. The article of claim 9 in which the address includes any address that may be mapped to a machine resource.

13. The article of claim 9 further causing a machine to fill the vacated bits starting at a start address, the start address equaling the address plus a constant value.

14. The article of claim 9 further causing a machine to, if there are a power of two number of memory channels, provide a constant value of three quarters.

15. The article of claim 9 further causing a machine to, if there are a non-power of two number of memory channels, determine the constant value by consulting a table including constant values for different numbers of memory channels.