# High-Throughput Line Buffer Microarchitecture for Arbitrary Sized Streaming Image Processing
## 2 Background
### BRAM-Misalignment Challenge and SWIM Framework  
An increase in processing throughput demands a more complex buffer that relies on the parallel pixel access using multi-pixel blocks. This, however, introduces potential memory alignment issues when utilizing BRAMs in the buffer design. An example in Figure 3a illustrates this problem, where the original image lines are sequenced into a high-throughput \(1\)D pixel stream, and then clipped to pixel blocks by the serial-to-parallel hardware (deserializer) inside the FPGA. The image-processing logic accepts one pixel block in each cycle. Complication due to memory block misalignment arises when the pixel number of one image line (denoted as \(N_{line}\)) is not an integer multiple of the pixel number in an input block (\(N_{blk}\)). In this case, some of the blocks ended up encapsulating pixels from two image lines. As an example, in Figure 3a, we have \(N_{line}=36\) and \(N_{blk}=16\). Thus, blk2, blk4, blk6, at the end of line0, line1, line2, contains \(12\), \(8\), and \(4\) pixels, respectively, that belong to the start of the next line. These pixels are labeled as _remainder pixels_ in the example. As Figure 3b shows, the general pixel buffer is composed of multiple line buffers (LBs) and each LB stores an entire image line. The LB is implemented with one BRAM with a size of \(N_{blk}\) to fulfill the parallel pixel access. We annotated the BRAM index (\(n\)) and the address (\(addr\)) in the diagram as \(Bn(addr)\) to present the storage pattern. Note that the remainder of the pixels within the last block of each line will be stored in the following LB. Therefore, the storage of subsequent blocks may inherit an alignment offset relative to the BRAM boundary. For example, in Figure 3b, the last block of line0(blk2) contains \(12\) remainder pixels that are  
Figure 2: Motivation for arbitrary sized image processing: (**a**) user-defined Region of Interest (ROI) processing; (**b**) arbitrary sized image processing in cloud computing.  
written to LB1. To store the blk3 continuously in LB, two addresses of LB1 (\(B1(0),B1(1)\)) are accessed. However, this behavior overwrites the pixels of blk2 stored in \(B1(0)\).  
To address the misalignment issue, [PERSON] et al. [9] proposed SWIM, a BRAM partition method for the pixel buffer design. With the same case, Figure 3c shows the SWIM buffer architecture and the pixel storage pattern. Each LB is composed of two BRAMs, and the width of the first BRAM is equal to the number of remainder pixels. For example, LB0 is composed of only one BRAM because there is no remainder pixel at the end of the previous line; LB1 is partitioned into BRAM1 and BRAM2 with widths of 12 and 4, respectively. Thus, the 12 remainder pixels in blk2 are stored at \(B1(0)\), and blk3 are stored separately at \(B2(0)\) and \(B1(1)\). With this method, SWIM guarantees that the block storage is aligned to the BRAM boundary. Although the SWIM framework generates BRAM partition configurations that avoid the BRAM-misaligned access, the hardware configuration needs to be re-generated through FPGA synthesis flow for a different image width (\(N_{line}\)). Even if the FPGA configuration files for different \(N_{line}\) can be pre-generated before the run-time, the long disruption caused by FPGA reconfiguration for differently sized images significantly decreases the throughput.