# High-Throughput Line Buffer Microarchitecture for Arbitrary Sized Streaming Image Processing
## 2 Background  
### Streaming Architecture for Image Processing on FPGA  
Similarly to the traditional computer system, memory hierarchy exists in FPGA-centric systems. On-chip memory inside the FPGA has low access latency, but relatively small capacity. In contrast, off-chip memory (DRAM) has a larger capacity, but longer latency and lower bandwidth. Furthermore, DRAM access consumes significantly more energy than on-chip memory. Therefore, in the field of FPGA architecture for image processing, it is a hot topic to trade off the on-chip buffer cost and system performance. For streaming architecture, it is widely adopted that the FPGA receives the pixels line-by-line as they are captured by the image sensor. The on-chip buffer is employed to store multiple lines for the 2D pixel access in the computation. Note that the buffer is optimized to the minimum size, and only the stores the pixels if they will be reused in subsequent computations.  
Previous works presented general methods for designing a streaming architecture for image processing with a 2D access pattern [5; 7; 10]. Figure 1 shows an example. There are three components within this streaming architecture: _Buffer_ (BUF), _operator_, and _interconnections_. The BUF stores multiple image lines that arrive sequentially in a line-by-line manner from the input stream, and the _operators_ can simultaneously access pixels across multiple lines within a local area defined by a 2D window or stencil pattern. For instance, in Figure 1, the operator 1 (OP1) performs 2D filtering with a \(3\times 3\) sliding window, and the step size of sliding window is one pixel in both vertical and horizontal directions. Assuming the FPGA receives one new pixel from the input stream per cycle to sequentially fill the input Buffer 1 (BUF1). Concurrently, BUF1 outputs \(3\times 3\) pixels in a window that is needed by OP1 to produce one resultant pixel. In each clock cycle, the window advances to the right by one step, and the output pixel is stored in BUF2. Note that while the window is moving to the boundary of each line, the output window of BUF1 concatenates the pixel columns from both the end and start in the buffer, as Figure 1a shows. The windows across the boundary are invalid, and the corresponding resultant pixels are dropped, such that the line width of BUF2 is less than the width of BUF1.  
As illustrated, BUF1 dynamically maintains three image lines that will be reused. Note that the on-chip BUF can be optimized to a minimum size where the new pixels are consumed as soon as they become available in the BUF and the old pixels are discarded. The BUF design for the other 2-D stencil patterns follows a similar principle. _Operator_ is composed of arithmetic units (adder, multiplier, etc.) tailored for the image application. The results from the operators can be either output as the final result, or stored in another BUF which provides the input of the subsequent operator. The _interconnections_ are the dedicated data paths that follows the data flow graph (DFG) of the application. For instance, in Figure 1d, Operator 2 (OP2) uses both the pixels in the initial BUF1 and the output of OP1 stored in BUF2 for further processing. In addition to the direct wires, _first-in, first-out (FIFO)_ was inserted on the data flow path to guarantee all pixels of the required pattern arrived at the operator in the correct clock cycle.