dataesr/scientific-paragraphs-categorization huggingface dataset:
Event-based vision sensors produce asynchronous event streams with high temporal resolution based on changes in the visual scene. The properties of these sensors allow for accurate and fast calculation of optical flow as events are generated. Existing solutions for calculating optical flow from event data either fail to capture the true direction of motion due to the aperture problem, do not use the high temporal resolution of the sensor, or are too computationally expensive to be run in real time on embedded platforms. In this research, we first present a faster version of our previous algorithm, ARMS (Aperture Robust Multi-Scale flow). The new optimized software version (fARMS) significantly improves throughput on a traditional CPU. Further, we present hARMS, a hardware realization of the fARMS algorithm allowing for real-time computation of true flow on low-power, embedded platforms.
We then present a hardware acceleration architecture of the fARMS flow algorithm using a hybrid system-on-chip (SoC) embedded platform containing a field programmable gate array (FPGA). We will refer to this architecture as hardware ARMS (hARMS). fARMS and hARMS were developed to complement the asynchronous nature of event-based vision sensors and the events they output. The hARMS architecture allows for flexible configuration based on application specific needs and provides significant improvements in latency and throughput compared to both the existing ARMS flow algorithm and the fARMS software baseline. This improved performance allows for real-time operation in a variety of visual scenarios, opening up the possibility of more widespread optical-flow-based application deployment on embedded platforms.
However, for some datasets and scenarios, real-time performance could not be achieved due to the deluge of data. The performance of the original ARMS algorithm is constrained by two primary factors:  (1)  The ARMS algorithm requires computation of the average local-flow vectors on all events within each window leading to repetitive averaging as the larger windows already encompass the events of the smaller neighborhoods and (2) the averaging considers all pixel locations in the η windows even if no new recent events occurred at these pixels. This means that the computational complexity of ARMS depends on both the maximum search window size (W m ) and the number of windows (η). Further analysis of the ARMS algorithm complexity is presented in Section. III-B. To mitigate these issues with ARMS algorithm, in this paper, we propose optimization of the ARMS algorithm to achieve significantly higher throughput.
C. SYSTEM-ON-CHIP DEVELOPMENT We used Xilinx's Zynq-7000 series SoC to implement the hARMS architecture presented in this paper. Xilinx's Zynq SoC is a hybrid computing platform that couples a traditional ARM processing system (PS) with a programmable logic (PL) FPGA fabric region. This architecture allows for algorithms to be split across both the regions of the SoC to achieve optimal performance. To streamline the development of accelerated applications targeting the Zynq platform, Xilinx provides the Software Defined System-on-Chip (SDSoC) development tool. SDSoC uses high-level synthesis (HLS) to allow hardware designs to be written in C/C++ for rapid development and verification of designs. SDSoC and HLS were used in the development of hARMS to allow for effi- cient design iteration and increase the configurability of the design.
First, maximum component distance between the current event and the RFB event is found. Then a tag is assigned based on the pre-computed window bin that the maximum component distance falls into. It is known that an event that falls into a given spatial window will also belong to all larger spatial windows. This means that only η + 1 unique window tags are needed to encode all possible windows along with the scenario where an event is not included in any windows. Once the windows that an event falls into are determined, the averaging can be performed as shown in part 2b of Algorithm 1. With the use of window arbitration and the RFB the algorithm only requires iteration over all of the N events in the RFB as opposed to costly searches over each of the expanding spatial windows. The true-flow results from fARMS are calculated in the same way as ARMS.
The PL is used to realize the hARMS accelerator using custom hardware blocks implemented in the FPGA fabric. The architecture developed for validation and testing computes the local flow in software on the PS. However, the local flow is only considered as an input to the hARMS design and can therefore be computed using any method, including PL acceleration as required by the application. As events are generated and the local flow is computed, the PS is used to accumulate local-flow events and collect the true-flow results. The accumulated events for which a valid local flow exists, and thus true flow can be calculated, are transferred to temporary RAM in the PL fabric using a direct memory access (DMA) controller and a block RAM (BRAM) interface. In addition to being stored in temporary RAM, the accumulated events are also added to the RFB to be used in the processing of future events.
Further reduction in the number of bits used to represent data could be considered in some cases depending on the expected range of velocities in the scene or tolerable loss in flow accuracy. We, however, designed the standard hARMS configuration with the goal of being robust to wide variations in scene dynamics, while maintaining accuracy equivalent to the fARMS algorithm. The architecture includes all the configurable parameters available in the fARMS algorithm, with the addition of P as outlined by Table  1. Unlike the fARMS algorithm in software, which can define these parameter at runtime, the hardware nature of the hARMS architecture requires configuration before compilation. The value of P is used to specify  the number of parallel accelerators to be used in the PL fabric. The high levels of parallelism that can be achieved through the use of this parameter have substantial impact on the design's performance and resource utilization.
The N parameter dictates the number of events stored in the RFB and processed in the PL. The ring buffer allows for new events to be added while replacing the oldest events at the same time. This way the most recent N events are always available for each true-flow calculation. When the true-flow calculations begin, the data stored in the RFB is streamed into the accelerator modules as shown in Fig. 2. Since the hARMS accelerators are designed for streaming inputs, one value from the RFB can be read each clock cycle, achieving an initiation interval of one and preventing the need for resource-expensive array partitioning. The hARMS accelerators are designed to perform asynchronous computation, such that, as the ARMS true flow is being calculated using the FPGA accelerator, the EAB can begin to be filled asynchronously as the previous EAB events are being processed.
However, in reality the value of W m is not restricted due to the resolution indifferent design of the hARMS architecture. As discussed in Section III-A, there are η + 1 possible window tags that could be assigned to any recent event as it is streamed into the accelerator, with the additional tag representing a recent event that does not belong to any of the defined windows. The tag can then be represented using only log 2 (η + 1) bits in hardware. These tag bits are appended to the input stream, while the x and y coordinate data is removed from the stream, as all required location information is now encoded in the window tag. The window arbiter hardware is fully pipelined to allow one new recent-flow event to be read from the internal input stream generated from the RFB on each cycle. The window edge values are statically declared in the tagLU T module and the window search is fully unrolled.
C. STREAM AVERAGING Averaging is the fundamental operation of multi-scale pooling. It involves averaging the values of recent local-flow events within each window. This operation is performed by the stream averaging module implemented in the PL fabric. The stream averager is a modified stream-based FPGA implementation of the averaging performed in the fARMS algorithm. It differs in its streaming nature and modular design, which allows parallelism between multiple instances of the module. As events from the RFB are streamed into the module, they are added to an internal array of window sums, where there is one sum for each of the η windows used in the design. This addition, however, is only performed for the sums of the windows that the event falls into. The tag assigned by the window arbitration module is used to select the window sums to which the event value should be added.
The average x and y components of the local flow in that spatial window are then returned as the true-flow results for the EAB event being processed. The ARMS compute module functionality described is realized as shown in Algorithm 3. The control flow is modified from that of fARMS to efficiently use streaming interfaces, as well as to capitalize on available parallelism. The process input stream loop is fully pipelined such that one event is read from and written to the window arbiter and averager modules respectively on each clock cycle. The calls to the stream averaging modules in lines 17 to 19 are performed in parallel using task-level, dataflow pipelining. Once the stream averaging is completed, the window index, w max , corresponding to the maximum local-flow magnitude is found and the trueflow results are returned and stored in temporary RAM before the DMA transfer of the results back to the PS for collection.
V. EXPERIMENTS AND RESULTS We evaluated the presented fARMS and hARMS designs against the same datasets presented in  [6]  for a direct comparison of accuracy and performance between the hardware and software designs. These datasets span a variety of scenes and sensor resolutions, allowing for a detailed investigation of the design space. We also evaluate how resource utilization and performance change when different configurable parameters are modified, and analyze design trade-offs when selecting a hardware configuration. A real-time performance comparison for the different datasets used is also provided to show successful achievement of real-time operation across a variety of visual scenes, event rates, and sensor resolutions. The Xilinx ZC706 development board, which includes the Zynq-7045 SoC, was used for all embedded software and hardware benchmarks.
A. TRIVIAL PATTERN The same trivial pattern dataset presented in  [6]  was used for evaluation of the developed hARMS accelerator. The dataset was recorded with a qVGA resolution event-based sensor and features a square and bars moving up and down in front of the stationary event camera. This dataset is denoted "Bar-Square" data. In the recording, the bars are always moving perpendicular to the true direction of motion, meaning that the ARMS algorithm should achieve an accurate estimate of the true flow in most cases. This dataset was used to test more than 60 different hardware design configurations. The number of spatial windows (η), maximum size of the spatial windows (W m ), and the number of parallel accelerators (P ) were all varied and the results evaluated. The direction estimation accuracy, throughput, throughput speedup, FPGA resource utilization, and estimated power usage were all collected to evaluate tradeoffs in the design configuration selections.
1) Direction Estimation Accuracy The bar-Square data involves motion of the scene in only one direction-either moving upward or downward. This means that for any of these movements, an ideal optical flow algorithm should output a direction distribution with one peak and with a standard deviation of zero. We use this to quantify the performance of the different implementations of the flow algorithms. Thus, the direction estimation error is quantified as the standard deviation of flow angle results across all the events. Larger standard deviation of angles indicate larger error in correcting the direction of motion from normal to true direction. A low standard deviation indicates that the true flow calculated has only one primary direction of motion, which is what is expected from the dataset.
There is also a certain point beyond which increasing N will not improve accuracy because all additional events will be filtered by the temporal window constraint. This is seen in Fig. 5  between buffer lengths of 1000 and 2000 where there is no change in accuracy. Despite the increase in standard deviation as N decreases, the accuracy was below the of the original software ARMS algorithm. This proves that even with a significant reduction in the amount of data considered, the hARMS and fARMS designs can achieve accurate results.
2) Throughput Design throughput is measured in true-flow events per second (evt/s). As many events will not have viable local-flow results such as those generated due to noise, the event rate coming from the sensor could be significantly higher than the maximum design throughput without overwhelming the design. However, we consider only the throughput of the hARMS design after filtering and prepossessing of events occurs. This allows us to evaluate the worst-case scenario where every event generated by the sensor is a valid true- flow event. In Section VI-D, a real-time evaluation of the hARMS architecture is provided, which includes information regarding total events and true-flow events in each dataset considered. Fig. 6  shows the throughput results for fARMS and hARMS with varying P for different number of spatial windows (η).
The results shown were generated for a selected benchmark configuration where W m = 320 and N = 1000. The figure shows that the throughput of the fARMS software design decreases as the number of spatial windows is increased. However, as expected, the throughput for each hardware configuration is nearly constant across varying numbers of spatial windows, with only small decreases as the number of windows increases. This behavior is due to the streaming architecture and fully unrolled window searching implemented in hardware. The small decreases in throughput that are observed as η increases are a result of the number of dividers being limited to four per averager. The highest throughput achieved with N = 1000 is 1. 21 Mevt/s when η = 4 and P = 24. Higher throughput speedups were achieved for larger values of η due to the decreasing performance of the fARMS baseline with large numbers of spatial windows.
That poor performance results from the need for sequential iteration through the windows in software. The parallel nature of hARMS and streaming architecture eliminate this bottleneck and allow for substantial speedup over the software baseline. As shown in Fig. 7 , speedup ranges from 4. 6× for P = 1 and η = 4 to 269. 2× for P = 8 and η = 32. The value of W m only affects the hardware implementation of the predetermined edges in the tagLU T module, and therefore has no impact on the latency or throughput of the design. The final parameter that has significant impact on the throughput of the design is N. Its value was set at 1000 for the primary benchmark because analysis of the Bar-Square dataset showed that a RFB with length 1000 is sufficient to capture all the recent-flow events within the τ temporal window of 5 ms used.
VI. HARMS ON REAL-WORLD DATASETS Based on the results from the Bar-Square dataset, we implemented the hARMS configurations to compute aperture robust optical flow on more complex real-world scenarios such as the DAVIS dataset  [19] , the MVSEC data  [20]  and a VGA resolution recording  [6]. The results for these are presented in the next sections. Note that because fARMS and hARMS produce flow results with only small differences due to quantization, only hARMS results will included in the following sections, however, results would be equivalent if using fARMS.
A. DYNAMIC ROTATION DATASET To ensure that the redesigned algorithm used for fARMS and hARMS maintains accuracy in dynamic scenes, the DAVIS dataset presented in  [19]  was used. This dataset provides the event stream from the 180×240 px resolution DAVIS along with timestamped grayscale images. The dataset also includes inertial measurement unit (IMU) data collected at a rate of 1000 Hz, which provides the angular velocity of the camera as the scene is being recorded. The dataset recording selected is the dynamic rotation scene as it allows for easier mapping from optical flow to the angular velocities from the IMU. In this scene the DAVIS is rotating along its axes while recording an office scene. The dynamic nature of the scene can be seen in Fig. 13 , which shows the local-flow direction estimates (left) and the hARMS flow direction estimates (right) for three distinct directions of motion.
We observe that even in a dynamic scene, the hARMS design performs well, producing the expected true direction of motion as its output, even from noisy local-flow direction estimates. For comparison to the software design study performed in  [6] , the same algorithm parameters were used, therefore, the hARMS configuration used to process the DAVIS dataset was (W m = 100, η = 10, τ = 5ms, P = 16, N = 1500). The size of N was intentionally set below the maximum required size to avoid losing events in the temporal window. This was done to show that even with a buffer length of less than half the minimum size to capture all relevant events, the fARMS algorithm and hARMS accelerator can still provide accurate output flow in dynamic scenes. The IMU data provided for the dynamic rotation scene was used to quantify the accuracy of the hARMS results in this dynamic scene.
B. MVSEC DATASET For further verification of the hARMS design for optical flow estimation in real-world environments, we applied it to scenes in the Multi Vehicle Stereo Event Camera (MVSEC) dataset  [20]. The MVSEC dataset utilizes an array of sensors to provide dense ground truth optical flow at specified frame intervals. We follow the same approach used in  [6]  to map This is especially noticeable during 22 to 32 second interval when the hARMS results overshoot the V y ground truth and undershoot the V x ground truth with high noise in both cases. As was observed in the original software realization presented by  [6] , this loss of accuracy can occur in areas with poor local-flow estimates or rapid changes in direction for which the ARMS flow cannot quickly correct.
C. PENDULUM VGA DATASET To verify the performance of the hARMS design for multiple directions of motion and occluded objects within a scene, we use the pendulum dataset introduced in  [6]. This scene uses a VGA (640×480 px) resolution sensor to record two pendulums of the same length oscillating in front of the sensor. One pendulum is placed further from the sensor such that it will appear smaller as it passes behind the other pendulum. For this dataset, we use the following hARMS configuration for processing: (W m = 50, η = 5, τ = 5ms, P = 16, N = 2000). A smaller value of W m was chosen based on the observation from  [6]  that such selection improves flow results in the presence of occlusion. Fig. 16  shows the results of hARMS processing on the pendulum dataset. At 0 ms in the sequence, the pendulums are swinging towards each other and hARMS produces accurate flow estimates for both objects.
1) Embedded Performance Evaluating the performance of an asynchronous, event-based algorithm for real-time operation can be difficult due to the inherent dependency on the event rate at any moment in time. Scenes with high activity, and therefore high event rates will require higher algorithm processing throughput to achieve real-time operation. For this research we will consider realtime operation to be the case where the fARMS or hARMS compute rate exceeds the true-flow rate of a specific dataset, where true-flow rate is the number of events per second that require a true-flow calculation to be performed. The dependence of hARMS and fARMS throughput on the buffer length, N , adds further difficulty to making performance comparisons across different datasets. Higher throughput can be achieved by reducing the input buffer length, however, this artificially reduces the amount of data available to correct the local-flow direction.
To avoid this we select the minimum buffer length required to capture all of the relevant events within the temporal window τ. This requires pre-evaluation of the dataset to determine the buffer length, which is equal to the maximum number of true-flow events within a τ = 5ms window. In a real-time application where the events are not pre-recorded, the effective buffer length can be dynamically changed in real time based on the event rate as long as it does not exceed the initial value of N used for hARMS configuration. However, for the purpose of evaluation we utilize the advance knowledge of the dataset to select the worst case buffer length for each dataset and use that value for processing of the entire set. For evaluation we chose the best performing configuration on the Bar-Square trivial motion dataset which used η = 4.
While the use of four spatial windows may not be optimal in all cases, it proved best for a singular direction of motion and is therefore used in all benchmarks to maintain consistency across datasets. The value of W m does not impact the throughput of the design and can be adjusted based on the sensor resolution and operating environment. For the embedded benchmark testing it was kept at 160. The value of P was set to 16 for all hARMS tests due to its high throughput and reduced resource footprint compared to the 24 core configuration that has high throughput, but can only be utilized when η is four. The embedded tests use the same plane-fitting local-flow method as in  [6]  and in the prior experiments for consistency. However, in applications where the local-flow computation could become a bottelneck, we propose the use of a more computationally efficient method for regularizing the timing of events called Savitzky-Golay plane-fitting presented in  [21].
The results of embedded performance benchmarking are shown in Table  3. Real-time performance is achieved using fARMS for two of the datasets, however, as expected most fall short of real time operation. The hARMS architecture is, however, able to easily achieve real time performance across all datasets tested. The dataset for which real-time performance is most difficult is the dynamic rotation set. This set has by far the highest average true-flow event rate of all datasets evaluated and requires the largest RFB. Despite this, the hARMS compute rate exceeded the true-flow rate by 2. 35× using the defined benchmark parameters. It is also important to note that flow accuracy can be maintained even with a more than 50% reduction in the buffer length, as shown in Section VI-A. Leveraging that finding could allow for even more comfortable achievement of real-time processing.
The results on the other datasets show that not only can real-time performance be achieved, it can also be achieved with reduced resource utilization depending on the scene. Observing the true-flow rates in Table  3  and the throughput results discussed in Section V-A2, it can be seen that real-time operation can be achieved with as little as one or two hARMS cores for most datasets, depending on the configuration. Another takeaway from the results shown in Table  3  is the independence of fARMS and hARMS throughput from sensor resolution. In fact, the lowest resolution sensor has the slowest compute rate, while higher resolution datasets could be processed much faster. This indicates that the most important factor in determining the fARMS and hARMS throughput performance is the true-flow event rate, which directly impacts the required buffer length.