Memory caches and large block copy - Page 1
Forum section: electronics / general

author: SiliconWizard
author_rank: Super Contributor
timestamp: February 25, 2021, 04:51:05 pm
content: I'm currently working on the implementation of memory caches, and here's something I'm musing over.For block copy operations of large blocks (> cache size), a block copy through cache can be very inefficient due to the cache having to execute a lot of write-back operations. Overall, large block copy is more inefficient through a cache than it would be if directly made using the memory controller.Apart from write-backs, another source of inefficiency through a cache is that the cache can't "know", when you're writing a block, that you are actually going to fill a whole line - and thus, when it has to replace a line, it will not only need to write it back first, but then READ the older content from memory for the destination location before writing it, even though this read operation is useless as the entire line is going to be rewritten anyway.An alternative of course is to bypass caches for large copies (for instance using some kind of DMA). Another is to use several levels of cache (which is a common approach.)Any thoughts on this?

author: FenTiger
author_rank: Regular Contributor
timestamp: February 25, 2021, 05:44:30 pm
content: Write combining. Maintain a "dirty" bit for each byte (or whatever the granularity is) in your cache line. When you want to evict it, if all the dirty bits are set then you can skip the read.

author: ejeffrey
author_rank: Super Contributor
timestamp: February 25, 2021, 05:44:40 pm
content: One standard way to address the read penalty is write-combining. You have a buffer the size of 1 cache line, but it has flags for which bytes have been written. If the line buffer fills up completely, you can flush it out to the cache directly without ever executing a fetch. If the buffer gets evicted before being full then you need to fetch the existing data before applying the write. Of course you have to implement this in a way that respects your memory ordering and cache coherence requirements.Vector instructions (if big enough) can operate on an entire cache line directly which is another way to avoid this penalty.

author: asmi
author_rank: Super Contributor
timestamp: February 25, 2021, 06:14:22 pm
content: Cache is useless for bulk operations, as it was designed for the exact opposite scenario, so I always bypass it for such operations. For Microblaze, if you can use write-through mode, it will be somewhat more efficient because in this mode cache won't load line into cache during write if they weren't in the cache to begin with, and written data will not be stored in cache. if you use AXI4 bus for interconnect and AXI4-compliant cache implementation (for example System Cache IP), you can control the write mode using AxCACHE signals, if ACE is enabled for MB ports, it will automatically invalidate relevant CPU cache lines when a write occurs through "generic" port.In practice, if a module requires significant memory bandwidth, I typically connect it directly to memory interconnect and utilize CPU's cache controls to flush/invalidate any cache lines which are associated with the target memory address range. For example, in case of video output, I use 2 framebuffers - one which is being written to by the CPU, and another one which is streamed out to whatever output module for displaying, and once CPU completes the work on a frame, I flush the cache, switch the framebuffer pointer for an output DMA, invalidate the cache for this address range and wait for DMA's interrupt which indicates that old framebuffer is now safe to access and modify. And then the cycle repeats. Basically idea is very simple - at one point of time a memory buffer is "owned" either by CPU, or by DMA. Incidentally, that's how DMA subsystem works in Linux, so it certainly isn't anything new.

author: SiliconWizard
author_rank: Super Contributor
timestamp: February 25, 2021, 07:21:03 pm
content: Write combining: from the description above, I can't see how that would help here.One inefficiency I was mentioning was the following scenario:Write word to cache -> cache already taken for corresponding location but with a different tag -> if corresponding line is dirty, write it back -> read ext memory at new location to update cache line -> write word to the right offset in cache lineI can't see how setting dirty bits individually would prevent the cache from reading memory at the new location to fill the cache line before actually updating the written word in it. It would require to know beforehand that we're going to write the whole line.One idea though that ejeffrey exposed was "vector instructions". If you have instructions that can actually write a whole cache line, then it's possible to instruct the cache to avoid the reading step.Which gets me to a more general solution: an extra input signal for the cache that would signal that the cache doesn't need to read data from memory when you're writing to it. The "user" of the cache usually "knows" this beforehand. Of course, for that to be error-free, the corresponding cache line should be written entirely, which can't always be guaranteed unless the cache write port has a data bus the width of a whole cache line, in which case, there is no question at all anyway.But yes, as I mentioned in the first post and as asmi confirmed, caches are no good for those large bulk operations. As I mentioned, I guess cache inefficiency for large blocks is usually mitigated on typical CPU-based systems with several levels of cache. That's of course an option, but it looks overkill for a lot of applications.Still, I'll keep the idea above and do some experiments.

author: asmi
author_rank: Super Contributor
timestamp: February 25, 2021, 07:36:46 pm
content: Quote from: SiliconWizard on February 25, 2021, 07:21:03 pmThis is one of the functions of AxCACHE signals in AXI4 bus (ARCACHE and AWCACHE for reads and writes respectively).

author: SiliconWizard
author_rank: Super Contributor
timestamp: February 25, 2021, 07:38:31 pm
content: Quote from: asmi on February 25, 2021, 07:36:46 pmI'm not using AXI4 so far, but I'll definitely have a look at this.

author: NorthGuy
author_rank: Super Contributor
timestamp: February 25, 2021, 08:53:43 pm
content: Quote from: SiliconWizard on February 25, 2021, 07:21:03 pmx86 has instructions with "non-temporal hint", which signals that caching is not needed, which avoids the read from the memory. Using such instructions increases the performance of block writing (memset) and copying (memcpy).MIPS uses high bits of the address to distinguish between cached and non-cached spaces.You can do either of these two methods (or even both together) to pass the caching hints from the user to CPU/cache.

author: asmi
author_rank: Super Contributor
timestamp: February 25, 2021, 09:00:03 pm
content: Quote from: NorthGuy on February 25, 2021, 08:53:43 pmIn case of Microblaze, you can specify cacheable address range during IP instantiation (you can set it up to entire address space if you so desire).If your platform has virtual memory support, most MMUs have ability to set caching properties on a per-page basis in PDE.

author: ejeffrey
author_rank: Super Contributor
timestamp: February 25, 2021, 10:19:06 pm
content: Quote from: SiliconWizard on February 25, 2021, 07:21:03 pmThat's not the way it works. The write combining buffer is separate from a conventional caches. Let's say all caches start empty and you write a single byte. That write request goes into the write combining buffer but doesn't touch any of the standard caches. Now you write a few more bytes. They goes into the buffer as well. There are two ways this can end. Option 1: you write enough to touch every byte in a cache line. Now you can flush that whole buffer out to main memory. Option 2: you only write part of a cache line and then move on. Now you need to read from RAM to fill the cache before you can process the write buffer.

author: SiliconWizard
author_rank: Super Contributor
timestamp: February 25, 2021, 10:48:50 pm
content: Quote from: ejeffrey on February 25, 2021, 10:19:06 pmOk, so that basically falls into the "multi-level" cache category. It's somewhat a small cache along with the cache.I just implemented the idea above - a signal indicating whether a fetch from memory in case of write miss is required or not. This is most commonly referred to as "no-fetch-on-write-miss".The additional logic is minimal and that cut the write time for large blocks in half.

author: asmi
author_rank: Super Contributor
timestamp: February 25, 2021, 11:07:28 pm
content: Quote from: SiliconWizard on February 25, 2021, 10:48:50 pmThe real question is - why do you insist on going through the cache if you have no intentions to use facilities provided by said cache? Since you said your TX size is more than cache size, it guarantees that entire cache will be effectively empty once TX is completed (assuming the TX is contiguous).

author: SiliconWizard
author_rank: Super Contributor
timestamp: February 26, 2021, 12:02:48 am
content: Quote from: asmi on February 25, 2021, 11:07:28 pmJust looking to "optimize" a bit some operations that have to use caches. Why? In some contexts that's handy. In a simple system with a CPU and data+instruction caches, any memory operation from the CPU (load/store) will go through the cache. I'm working on such a system for instance. Uncached access (load/store), for peripheral access for instance, is handled at the address decoding level only - I have no kind of MMU whatsoever. Selectively going through the cache or direct to memory from the CPU itself requires very significant additional logic that I'm not ready to add, and that is not necessarily required anyway. But for more complex systems, that could certainly be an option, although writing memcpy()-like functions that would mess with cached or uncached access depending on block size would not be necessarily a picnic, or even something that I've seen very often.Now of course, for moving large blocks of data without CPU intervention, I can still use a DMA controller directly tied to the memory controller. And sure I can use DMA for memory-to-memory copy as well if required. But I think optimizing basic data access from the CPU is still worth it, and clearly this one optimization was simple (and almost obvious). I'm not going to attempt anything fancier as indeed, caches are not meant for handling that very efficiently anyway.

author: Evan.Cornell
author_rank: Regular Contributor
timestamp: March 01, 2021, 01:35:42 pm
content: Quote from: FenTiger on February 25, 2021, 05:44:30 pmIs this why Xilinx BRAM on Spartan7 has a max data width of 36 bits?

author: asmi
author_rank: Super Contributor
timestamp: March 01, 2021, 02:05:15 pm
content: Quote from: Evan.Cornell on March 01, 2021, 01:35:42 pmNo, that's for ECC.

author: NorthGuy
author_rank: Super Contributor
timestamp: March 01, 2021, 03:01:41 pm
content: Quote from: Evan.Cornell on March 01, 2021, 01:35:42 pmThe extra bits are for ECC, but they often come handy for other purposes.