<figure><img src="image_1.png" /></figure>

HAL
open science

# Embedded Computer vision software stack and development process for Autonomous Driving

Senthil Yogamani

► To cite this version:

Senthil Yogamani. Embedded Computer vision software stack and development process for Autonomous Driving. 2024. hal-04663564

HAL Id: hal-04663564

https://hal.science/hal-04663564v1

Preprint submitted on 29 Jul 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Embedded Computer vision software stack and development process for Autonomous Driving

Senthil Yogamani

**Abstract.** Autonomous driving systems have their own unique criteria due to functional safety and traceability issues. In this work, we provide an overview of autonomous driving system and the development process involved. In particular, we focus on machine learning algorithms which started to dominate the AD landscape which needs a revisit to the development process.

## 1 Introduction

The complexity of the next generation algorithms have increased greatly and the compute power has increased as well with the advent of HW accelerators for Dense Optical Flow and CNN. Computer vision algorithms have to be carefully designed to be optimal for the heterogeneous compute platforms. There are a variety of perception tasks which are deployed in real world systems namely object classification [5, 31], 2D object detection [19, 30], 3D object detection [21, 16, 15], depth estimation [9, 11, 10], localization [26], motion estimation [14, 18, 22], semantic segmentation [23, 20, 24], blockage and adverse weather detection [28, 27, 4], multi-task learning [2, 12, 25], sensor fusion [29, 1, 8] and perception systems [6, 17, 7, 3].

The motivation of this document is a self contained reference of the computer vision system detailing all aspects related to hardware, software and algorithms. The flow of the design goes from hardware to algorithms and then to software. Rephrasing it, software is designed based on understanding on the specifications of hardware and algorithms, limitations in particular.

Diagram in Figure 1 shows an abstraction of the entire application stack including Hardware, Software and Applications which are dealt with in more detail in each of the sections. Section 2 covers details of all the hardware aspects. Section 3 covers the software aspects. Section 4 covers the applications which form the end user functions. Section 5 discusses the software development process.

## 2 Embedded Systems

### 2.1 SOCs

The following discussion is based on the experience working with various SOCs (System on Chip) targeted for ADAS high-end market. This is by no means an exhaustive comparison. We have summarized the different types of processing units which are relevant for vision algorithms below.

GPP (General Purpose Processor) is typically the master control processor of an SOC. Some flavour of ARM Cortex Application processors is commonly used as the
<figure><img src="image_3.png" /><figcaption>Fig. 1. Hardware and Software components in an automotive system</figcaption></figure>

GPP. The flavours vary from A9, A15 to A53 and A57. The former two are the popular GPP in the current generation devices and the latter two are 64-bit roadmap processors. NEON is a SIMD engine which can accelerate image processing. Because of the ubiquitous nature of multimedia, importance of NEON has grown and it is becoming more tightly integrated.

**GPU (Graphics Processing Unit)** was traditionally designed for graphics acceleration. It can be used for view rendering/remapping and for drawing output overlays using OpenGL. Some of the GPUs can perform only 2D graphics via OpenVG. They are slowly being re-targeted to be used as additional resource for vision algorithms through OpenCL. Nvidia is leading this way providing a General Purpose GPU processor for vision. With respect to Vision algorithms, CUDA (Compute Unified Device Architecture) of Nvidia is significantly more powerful than other GPUs provided by ARM (Mali), PowerVR (SGX) and Vivante. The performance power of Nvidia GPUs are growing at a significantly faster rate compared to other processors to suit the growth in the field of automotive vision applications. CUDA uses SIMT (single instruction multiple threads) with threading done in hardware. It has a limitation of dealing with load/store intensive operations which is improved by high data-bandwidth provided to it.

**SIMD** engines are quite popular in the application areas of image processing. This is because the input data is 8-bit fixed point and the initial stages of image processing are
typically embarrassingly parallel. These processors are typically designed to be power-
efficient by avoiding floating-point and having a simplified processing pipeline. For
instance, performance/watt of TI's Embedded Vision Engine (EVE) [13] is ~8X than
that of A15. On the other hand, it lacks flexibility and it is typically used for the pixel-
level initial stages of the pipeline.

**DSPs** (Digital Signal Processors) are traditional processors designed for low-power acceleration of signal processing algorithms. Most of them including TI's C66x and Toshiba's MPE are VLIW (Very Large Instruction Word) based superscalar processors. VLIW is an efficient architecture scheme to provide flexibility compared to SIMD. They also exhibit a form of SIMD using packed arithmetic i.e. aliasing 4 byte instructions in a word instruction. The gap of performance normalized to cost of DSP is becoming closer because of lowered cost and performance improvements of NEON and its ubiquity.

**ASIC** (Application-specific Integrated Circuit) implements the entire algorithm in hardware with minimal flexibility like modification of parameters coded via register settings. If the algorithm is standardized, ASIC provides the best performance for the given silicon. But it requires a lot of initial investment which impacts time to market and there is a risk of being redundant particularly for the area of computer Vision where the algorithms are not standardized and could change drastically.

**FPGA** (Field Programmable Gate Array) is closely related to ASICs but has a key advantage of being re-configurable at run-time. From the application perspective, this means the HW can adaptively transform to an accelerator needed for that particular scenario. For instance, if the algorithm has to be drastically different for low and high speeds, the FPGA can transform into one of these as required. But because ASIC is hard-coded to be optimized for one algorithm, FPGA will be lagging in performance. Due to recent progress in FPGA technology, the gap is closing. Xilinx and Altera are the major FPGA suppliers who offer very similar performance FPGAs.

**Others:** Many of the SOCs provide video codecs (H.264), JPEG and ISP as ASICs. The video codec provides Motion Estimation (Block-matching Optical Flow) which is very useful for the vision algorithms. Some SOCs provide a safety compliant micro-processor like Cortex-M4 or R4 for AUTOSAR (AUTomotive Open System ARchitec-ture) and improved ASIL.

**Summary:** Typical design constraints for SOC selection for embedded systems are performance (MIPS, utilisation, bandwidth), cost, power consumption, heat dissipation, high to low end scalability and programmability. Unlike hand-held devices, power consumption is not a major criterion as it is powered by car battery. Heat dissipation matters only up to a threshold and might add to costs through better heat sinks. Programmability is becoming abstracted via SW frameworks like OpenCL and is not a major factor in cost. Hence for ADAS, the main factors finally boil to cost and performance. Because of the diverse nature of the processors, this is typically a difficult decision to make. Usually comparing the processor via MIPS is not useful as the utilization is heavily dependent on the nature of the algorithm. Hence a benchmark analysis for the given list of applications based on vendor libraries and estimates is critical for choosing an appropriate SOC. A hybrid architecture which combines fully programmable, semi programmable and hard-coded processors could be a good amortized risk option.
## 2.2 Peripherals

At high level, there are two type of camera system. Standalone camera with a small embedded system tightly integrated in the camera housing. This is sufficient for smaller applications like a Rear View camera. But for more complex applications, the camera is typically connected to an external powerful SOC. There is a lot of interfacing and electronics that goes in between a camera and the SOC. As illustrated in the Fig 3, for a typical surround view system with 4 camera inputs, the spatially separated cameras have to be connected to a central ECU and fed to the ECU. The data bandwidth requirements for video is high compared to other systems which brings in a lot of challenges and limitations in SOC. The raw digital output from a sensor is typically 10/12 bit but the video input port of the SOC might be just 8 bits. This mandates an external ISP to compress it to 8 bits. Other simple factors like resolution and frame-rate could double the system requirements and hence they are important. The connectivity between SOC and camera is typically wired via twisted pair or co-axial cable.

<table><caption>**Table 1.** Common Peripherals and Interfaces to SOC</caption><thead><tr><td>Peripheral/Interface</td><td>Options</td></tr></thead><tbody><tr><td>SOC VidIn</td><td>CSI2/LVDS, VIP, Parallel</td></tr><tr><td>Protocol/Bus</td><td>SerDes Coaxial cable, Ethernet twisted pair cable</td></tr><tr><td>SOC VidOut</td><td>DSI/OLDI, Parallel/HDMI</td></tr><tr><td>Vehicle Interface (VEH IF)</td><td>CAN/CAN FD, FlexRay, Ethernet</td></tr><tr><td>Comm Interface (COM IF)</td><td>PCIe, SPI, QSPI, UART, GPIO, I2C, Ethernet, USB 3.0.</td></tr><tr><td>Memory</td><td>RAM (L2/L3), DDR3/DDR4, Flash/EEPROM, MEMIF, DMA/Cache</td></tr><tr><td>Debug IF</td><td>UART, ETH, JTAG</td></tr></tbody></table>

Use of Frencher and deserializer (together known as SerDes) and signaling via co-axial cable is more common because of the high bandwidth of 1 Gbps/lane. Co-axial cable interfaces use Fakra connectors which is used by European OEMs. Ethernet interface and twisted pair cable is a cheaper alternative but it has a relatively limited bandwidth of 100 Mbps. To compensate for it, MJEG is performed before transmission which causes a limitation of having the complete ISP and conversion to 8-bit for MJPEG before transmission. The other alternative can leverage the SOC ISP. Ethernet cameras also require more complex electronic circuitry on both ends as shown in the figure. Gigabit Ethernet could be used to achieve higher bandwidth but it is more expensive and defeats the purpose of lower cost.

Table 1 summarizes the various interfaces and peripherals. Most of the modern SOC interfaces are digital and serial. MIPI (Mobile Industry Processor Interface) standardized the serial interfaces for camera input CSI (Camera serial interface) and DSI (Display serial interface). Physically these use LVDS connectors underneath. CSI2 is the current generation whose bandwidth is 1 Gbps/lane. OLDI is the open LVDS interface which works on bare-metal LVDS. Some SOCs provide parallel interfaces in additional to the serial interfaces. Although parallel interfaces provide a larger bandwidth, it re-
quires larger wiring and more complex circuitry and not typically scalable. Sometimes DVP is used as an umbrella term for various types of digital video ports.

Vehicle interfaces are unique to automotive space. These carry the signals from the car to the SOC. With respect to ADAS systems, odometry related signals like wheel speed, yawrate, etc are useful for motion related algorithms like SFM. It could also provide signals like Ambient light levels, fog/rain sensor, etc which could be helpful to adapt the algorithm. The common communication protocols used are CAN and Flexray as they are low payload data. For, high payload signals, sometimes Ethernet protocol is used. Flexray is an improved version of CAN (faster and more robust) and hence more expensive as well. CAN FD (flexible data-rate) is an improved second generation of CAN. Many of the automotive SOCs have direct interface to CAN and some of them support Flexray additionally. In some cases, it is not available and a companion MCU is required.

As mentioned before, memory is a critical factor in vision systems. There are several types of memory involved – main memory is usually DDR which is typically start at 256 MB and could go to several GBs. Image and all the intermediate processing data resides here. The high end current generation systems use DDR3 and the trend seem to be moving towards DDR4. There is also Flash/EEPROM memory for storing persistent data like bootup code, config parameters and sometimes statistics of the algorithm outputs. On the SOC, there is on-chip memory (L3) in the order of few MBs which is shared across the different cores on the chip. It could be used as a high speed buffer to stream from DDR. There is also cache or internal memory inside the processors(L1 and L2) which has access rates close to the clock frequency of the processor. DMAs are common in vision systems for ping-pong buffering of data from DDR to L2/L3 memories. It is important to appreciate the hierarchies of memory which have opposite gradation in size and speed. There are arbitrated via a memory interface MEMIF in the SOC. Memory could become a serious bottleneck in such a system and sometimes it could get overlooked. A detailed bandwidth analysis of the algorithms is necessary to decide the speed of memories and the bandwidth of MEMIF.

Debugging is typically done via JTAG and an IDE and this is usually not supported in the native ECU and a breakout board is necessary during development stage. For Ethernet systems, there is direct exposure of ECU memory via file systems. Sometimes debugging is also done through logging via UART. The other peripherals like SPI (for serial comm), I2C (master-slave electronics sync), GPIO(general purpose pins),etc are standard as in other electronic systems

## 3 Embedded Software

In this section, we discuss the software which is related to the embedded system typically provided by SOC vendors and third parties which are re-targeted to particular SOCs.
## 3.1 OS

**Drivers & Services:** We make this distinction for the following reason. Drivers are just linked in without active involvement in the development. Services can also be drivers but the developer has to interact with it. For instance, EDMA driver has to be used to manage data or IPC has to be used to synchronize across cores.

**Operating System/Frameworks:** Traditionally ADAS systems has been using real-time OSes (RTOS) to meet the real-time requirements where-as multimedia infotainment systems and mobile have moved on to high level OS (mostly linux based like Android). Greenhill Integrity and Vxworks are some of the popular RTOS. Integrity OS comes with ASIL D and AutoSAR compliance. Functional safety standards require low overheads of scheduling because when the switch to a fail-safe mechanism should be quick. The latter supports ease of application development and the former latency. A dual-kernel approach using a hypervisor bridges the gap between the both and achieves the best of both worlds.

## 3.2 Algorithm Frameworks

**OpenCL:** Investigation of SOC abstraction frameworks like OpenCL and OpenVX. OpenCL is a new technology from the company Khronos and the uncertainty was whether it produces the level of abstraction to exploit the architecture differences and do efficient optimization. Investigative analysis was done for various architectures SIMD, SIMT, VLIW, etc. OpenCL performed reasonable on SIMD architectures but not efficient on other architectures. OpenCL performed well for easily parallelizable algorithms like Harris detector or Sobel filter but for more complex data flows like connected components, it did not perform well.

**SYCL:** SYCL is a cross-platform abstraction layer that builds on the underlying concepts, portability and efficiency of OpenCL that enables code for heterogeneous processors to be written in a "single-source" style using completely standard C++. SYCL retains the execution model, runtime feature set and device capabilities of the underlying OpenCL standard. The OpenCL C specification imposes some limitations on the full range of C++ features that SYCL is able to support. This ensures portability of device code across as wide a range of devices as possible. As a result, while the code can be written in standard C++ syntax with interoperability with standard C++ programs, the entire set of C++ features is not available in SYCL device code. In particular, SYCL device code, as defined by this specification, does not support virtual function calls, function pointers in general, exceptions, runtime type information or the full set of C++ libraries that may depend on these features or on features of a particular host compiler. SYCL single-source programming enables the host and kernel code for an application to be contained in the same source file, in a type-safe way and with the simplicity of a cross-platform asynchronous task graph.

SYCL includes templates and generic lambda functions to enable higher-level application software to be cleanly coded with optimized acceleration of kernel code across
the extensive range of shipping OpenCL 1.2 implementations. Developers program at a higher level than OpenCL C or C++, but always have access to lower-level code through seamless integration with OpenCL, C/C++ libraries, and frameworks such as OpenCV or OpenMP. The current version of SYCL is SYCL 1.2.1. SYCL programs are explicitly parallel and expose the full heterogeneous parallelism of the underlying machine model of OpenCL. This includes exposing the data-parallelism, multiple execution devices and multiple memory storage spaces of OpenCL. However, SYCL adds on top of OpenCL a higher level of abstraction allowing developers to hide much of the complexity from the source code, when a developer so chooses.

**OpenVX:** OpenVX is intended to be used as either directly by applications or as the acceleration layer for higher level vision frameworks, engines or platform APIs. OpenVX is designed as a framework of standardized computer vision functions able to run on a wide variety of platforms and potentially to be accelerated by a vendor's implementation on that platform. It can improve the performance and efficiency of vision applications by providing an abstraction for commonly used vision functions and an abstraction for aggregations of functions (graph), there by providing the implementer the opportunity to minimize the run-time overhead.

**Kernels:** This is often the overlooked part of OpenVX. OpenVX kernels is an ambitious attempt to do functional standardization of computer vision. This is what OpenGL achieved for Graphics, a standard algorithm API which is functionally the same on any HW or SW implementation and deviations have to be reported if any. Obviously, vision is much harder and there are hundreds of algorithms even for something as simple as edge detection and it is hard to standardize functionality and accuracy. At the moment, there are roughly 100 standardized kernels in OpenVX and this is very limited compared to 1000s of functions in OpenCV. This is the reason, it is not emphasized much. On that note, there is standardization work being done for Deep Learning as there are various libraries, interfaces and data formats. This could be attractive for us to have a portable solution for deep learning across various platforms.

**Graph Scheduler for partitioning and scheduling:** The holy grail of scheduling is to have each node implemented for different targets and the scheduler automatically figures out the mapping. In practice, it is difficult to get optimized algorithms on various processors let alone on a single one. Realistically, we will have to use a manual scheduler where we specify which nodes go to which processors. The framework helps us to integrate different components easily (especially ones from third parties and SOC suppliers) and handle scheduling and synchronization across the execution of nodes.

## 3.3 Algorithm Libraries

**OpenCV 3.1:** Commonly used for preprocessing and classical computer vision algorithms.

**TensorFlow:** Design of TensorFlow consists of functional nodes which interact via data flow graphs. The functional nodes are symbolic functors which are type free, stateless and efficiently combined using composition. It can take in OpenCL functions un-
derneath. There is also an attempted to link it with OpenVX APIs.

**Eigen:** Expression template based template meta programming for efficient composition of operators via lazy evaluation. Being a C++ template Library for Linear Algebra the main attractions of Eigen are: 1) Purely headers, no compilers required & supports multi-platform. 2) Provides good speed & simple interface to use. 3) A single unified library. Most libraries specialized in one of the features or module. Eigen satisfy all these criteria: free, fast, versatile, reliable, decent API, support for both sparse and dense matrices, vectors and array, linear algebra algorithms, geometric transformations.

**Target optimized Vision Libraries:** Embedded Optimization is very important as the system has to operate real-time, one could get 10x improvement in runtime. TI provides DSP optimized VLIB/IMGLIB, Nvidia provides CUDA SDK with all libraries bundled together. Xilinx/Altera provide their own vision IP blocks for standard functionality. In terms of software, OpenVX promises to provide a processor agnostic vision API standard but it is still in early stages. OpenCL could potentially be used for writing HW agnostic optimizations but not all processors doesn't support it.

# 4 Application Software

In this section, we discuss the software which is typically needed for the end application development by Tiers or OEMs.

**Prototyping:** OpenCV is the most popular open-source algorithm library. Traditionally it has been used for prototyping, after it has been taken over by Itseez, there seem to be a push for commercialization. It has strong GPU acceleration and a recent effort for NEON acceleration. Matlab is the popular among researchers for rapid prototyping. It does provide some generic support for embedded through RTW C++ code generation and linking with optimized target libraries. But this wouldn't be useful for end system. Python is becoming a popular alternative because of cost of matlab and wrapping of other libraries (OpenCV) and symbolic math and tight integration with optimizer compilers.

**Compliance:** Because it is a safety system, testing is extremely important and challenging. Software process has to comply to SPICE process and ISO 26262 functional safety standard. ISO, SPICE, ASIL, AUTOSAR

**Testing Tools:** Test data collection and manual annotation is a significant portion of the project. Exhaustive coverage of all scenarios (different weather conditions, object interactions) is difficult and sometimes simulated data is used. KITTI has become a standard dataset for benchmarking ADAS algorithms. They have become the go-to centralized place for benchmarking basic building block algorithms for ADAS like stereo, optical flow, odometry, road detection, object detection and tracking. The software has to be rigorously static tested to meet MISRA/SPICE compliance. MISRA sometimes conflicts with good code quality, maintenance and performance. Tools like LINT can
be used to do static testing.

**Development Tools:** ADTF (Automotive Data and Time-Triggered Framework) is a development and testing framework popular amongst Automotive OEMs. It provides a modular environment similar to Mathworks Simulink to enable plug and play of different vendor modules. For instance, a TIER1 or a 3P developing an algorithm would develop an ADTF wrapper compatible with the OEMs' requirements. The main advantage of ADTF is to have native support for Can/FR/LIN so that it can interface with the car directly. It also provides time synchronization of Mechanical and image signals. Canoe for simulation of flexray/can signals, variant coding and Inchron for system profiling.

# 5 Software Development Process

## Design Driven Development (D3):

**Algorithm Design Principles:** Don't re-invent the wheel for algorithm model. Re-use as much as possible from previous projects. Re-use from standard proven models which are open-source. Need to give strong justification not to re-use! No model is perfect, assumptions and weakness have to be documented in detail. Decoupling of model - need clear separation of standard parts and novelty would benefit IP process, optimization using standard 3P libraries, sharing with 3P, etc. Need clear separation of well-defined model and heuristics.

**Algorithm Design Review :** Developer defends his design with a review committee. It is important for the process and very effective in practise. Review and criticism helps strengthen and validate the proposed model. Technical lead to track the high level aspects. Embedded developer to understand the algorithm from the beginning and contribute. External Experts domain experts from other teams to help get a neutral perspective. Need to ensure it is understandable for the Embedded developer to contribute actively rather than port and test. Reviewers need to come prepared and share review comments prior to the meeting to have a productive discussion. The review process might need a few iterations to converge.

**SW Design Review:** Once the algorithm design is agreed upon, the next stage would be software design. The Algorithm and Embedded developer will work together and prepare a software design. The implementation will be shared between the two.

**Unit Testing:** Unit testing vision algorithms is not the same as regular software because of the challenges listed below. Due to these challenges, it is difficult to write tests manually. The typical approach is to run images through the algorithm and use manually annotated or visually verified outputs as ground truth. In cases of a complex pipeline of algorithm modules, it is only possible to generate I/O by passing images through the whole pipeline and verifying against the output for the pipeline. A good example would be Pedestrian detection algorithm. The test cases for modules can only be
generated by functional tests for entire pipeline based on annotated pedestrian outputs. It is difficult or impossible to generate abstract I/O for intermediate stages of algorithm pipeline on its own. This is referred to as a top down approach in creating test sets for the vision module and this is the preferred approach for computer vision algorithms.

### Challenges in unit-testing Computer Vision algorithms:

- Large Input/Output data: The input typically consists of transformed images or its features which can be of the order of millions of values. Hence it is impossible to write the test cases manually. Also due to combinatorial explosion, it is impossible to cover all possible combinations. For example, for the code that will threshold the raw binary code for a 1024 × 1024 binary image, $2^{(1024×1024)}$ test cases would be required, which is clearly beyond the realm of practicality.

- Large parameters: Number of parameters for one module can be large especially for machine learning algorithms. For instance, CNN has millions of parameters and its challenging to populate it manually.

- Automated code generation via Machine Learning: ML is becoming more common in our development where the features are picked from data. For instance, code for decision trees for classifiers are auto generated, network for deep learning is auto-generated from Python framework. Writing test cases for code that is not written by a developer but generated means it's very difficult to write manual test cases for.

- Abstract Mathematical data: The input data could be abstract features learnt by a Machine Learning classifier(eg: CNN features) or a feature transform (eg: HOG). It is difficult to generate these inputs through mocking and hence there is a need for tight coupling of the tests. Generated data is the only realistic source as human cannot generate realistic input data.

- Semantic Ambiguity: Functionality of an algorithm is not well defined and it is highly dependent on other modules in the chain. This creates a tight coupling of an algorithm pipeline. For instance, there is no optimal edge detector, segmentation or clustering. It highly depends on the use case.

## References

1. Shubhankar Borse, Marvin Klingner, Varun Ravi Kumar, Hong Cai, Abdulaziz Almuzairee, Senthil Yogamani, and Fatih Porikli. X-align: Cross-modal cross-view alignment for bird's-eye-view segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3287-3297, 2023.

2. Sumanth Chennupati, Ganesh Sistu, Senthil Yogamani, and Samir A Rawashdeh. Multi-net++: Multi-stream feature aggregation and geometric loss strategy for multi-task learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0-0, 2019.

3. Ashok Dahal, Jakir Hossen, Chennupati Sumanth, Ganesh Sistu, Kazumi Malhan, Muhammad Amasha, and Senthil Yogamani. Deeptrailerassist: Deep learning based trailer detection, tracking and articulation angle estimation on automotive rear-view camera. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0-0, 2019.
4. Arindam Das, Pavel Křížek, Ganesh Sistu, Fabian Bürger, Sankaralingam Madasamy, Michal Uřičář, Varun Ravi Kumar, and Senthil Yogamani. Tiledsoilingnet: Tile-level soiling detection on automotive surround-view cameras using coverage metric. In *2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC)*, pages 1–6. IEEE, 2020.

5. Arindam Das and Senthil Yogamani. Evaluation of residual learning in lightweight deep networks for object classification. In *Proceedings of the 20th Irish Machine Vision and Image Processing Conference*, pages 205–208, 2018.

6. Ciarán Eising, Jonathan Horgan, and Senthil Yogamani. Near-field perception for low-speed vehicle automation using surround-view fisheye cameras. *IEEE Transactions on Intelligent Transportation Systems*, 23(9):13976–13993, 2021.

7. Jonathan Horgan, Ciarán Hughes, John McDonald, and Senthil Yogamani. Vision-based driver assistance systems: Survey, taxonomy and advances. In *2015 IEEE 18th International Conference on Intelligent Transportation Systems*, pages 2032–2039. IEEE, 2015.

8. Marvin Klingner, Shubhankar Borse, Varun Ravi Kumar, Behnaz Rezaei, Venkatraman Narayanan, Senthil Yogamani, and Fatih Porikli. X3kd: Knowledge distillation across modalities, tasks and stages for multi-camera 3d object detection. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 13343–13353, 2023.

9. Varun Ravi Kumar, Marvin Klingner, Senthil Yogamani, Markus Bach, Stefan Milz, Tim Fingscheidt, and Patrick Mäder. Svdistnet: Self-supervised near-field distance estimation on surround view fisheye cameras. *IEEE Transactions on Intelligent Transportation Systems*, 23(8):10252–10261, 2021.

10. Varun Ravi Kumar, Marvin Klingner, Senthil Yogamani, Stefan Milz, Tim Fingscheidt, and Patrick Mader. Syndistnet: Self-supervised monocular fisheye camera distance estimation synergized with semantic segmentation for autonomous driving. In *Proceedings of the IEEE/CVF winter conference on applications of computer vision*, pages 61–71, 2021.

11. Varun Ravi Kumar, Stefan Milz, Christian Witt, Martin Simon, et al. Near-field depth estimation using monocular fisheye camera: A semi-supervised learning approach using sparse lidar data. In *CVPR Workshop*, volume 7, page 2, 2018.

12. Varun Ravi Kumar, Senthil Yogamani, Hazem Rashed, Ganesh Sitsu, et al. Omnidet: Surround view cameras based multi-task visual perception network for autonomous driving. *IEEE Robotics and Automation Letters*, 6(2):2830–2837, 2021.

13. Dipan Kumar Mandal, Jagadeesh Sankaran, Akshay Gupta, Kyle Castille, Shraddha Gondkar, Sanmati Kamath, Pooja Sundar, and Alan Phipps. An embedded vision engine (eve) for automotive vision processing. In *2014 IEEE International Symposium on Circuits and Systems (ISCAS)*, pages 49–52. IEEE, 2014.

14. Eslam Mohamed, Mahmoud Ewaisha, Mennatullah Siam, Hazem Rashed, Senthil Yogamani, Waleed Hamdy, Mohamed El-Dakdouky, and Ahmad El-Sallab. Monocular instance motion segmentation for autonomous driving: Kitti instancemotseg dataset and multi-task baseline. In *2021 IEEE Intelligent Vehicles Symposium (IV)*, pages 114–121. IEEE, 2021.

15. Sambit Mohapatra, Heinrich Gotzig, Senthil Yogamani, Stefan Milz, and Raoul Zollner. Exploring deep spiking neural networks for automated driving applications. *arXiv preprint arXiv:1903.02080*, 2019.

16. Sambit Mohapatra, Senthil Yogamani, Heinrich Gotzig, Stefan Milz, and Patrick Mader. Bevdnet: bird's eye view lidar point cloud based real-time 3d object detection for autonomous driving. In *2021 IEEE International Intelligent Transportation Systems Conference (ITSC)*, pages 2809–2815. IEEE, 2021.

17. Maximilian Pöpperli, Raghavendra Gulagundi, Senthil Yogamani, and Stefan Milz. Capsule neural network based height classification using low-cost automotive ultrasonic sensors. In *2019 IEEE Intelligent Vehicles Symposium (IV)*, pages 661–666. IEEE, 2019.
18. Hazem Rashed, Ahmad El Sallab, Senthil Yogamani, and Mohamed El Helw. Motion and depth augmented semantic segmentation for autonomous navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0–0, 2019.

19. Hazem Rashed, Eslam Mohamed, Ganesh Sistu, Varun Ravi Kumar, Ciarán Eising, Ahmad El-Sallab, and SK Yogamani. Fisheyeyolo: Object detection on fisheye cameras for autonomous driving. In Proceedings of the Machine Learning for Autonomous Driving NeurIPS 2020 Virtual Workshop, Virtual, volume 11, 2020.

20. Hazem Rashed, Senthil Yogamani, Ahmad El-Sallab, Pavel Krizek, and Mohamed El-Helw. Optical flow augmented semantic segmentation networks for automated driving. In Proceedings of the International Conference on Computer Vision Theory and Applications, 2019.

21. Khailash Santhakumar, B Ravi Kiran, Thomas Gauthier, Senthil Yogamani, et al. Exploring 2d data augmentation for 3d monocular object detection. arXiv preprint arXiv:2104.10786, 2021.

22. Mennatullah Siam, Heba Mahgoub, Mohamed Zahran, Senthil Yogamani, Martin Jagersand, and Ahmad El-Sallab. Modnet: Moving object detection network with motion and appearance for autonomous driving. arXiv preprint arXiv:1709.04821, 2017.

23. Mennatullah Siam, Sepehr Valipour, Martin Jagersand, Nilanjan Ray, and Senthil Yogamani. Convolutional gated recurrent networks for video semantic segmentation in automated driving. In 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), pages 1–7. IEEE, 2017.

24. Ganesh Sistu, Sumanth Chennupati, and Senthil Yogamani. Multi-stream cnn based video semantic segmentation for automated driving. arXiv preprint arXiv:1901.02511, 2019.

25. Ganesh Sistu, Isabelle Leang, Sumanth Chennupati, Senthil Yogamani, Ciarán Hughes, Stefan Milz, and Samir Rawashdeh. Neurall: Towards a unified visual perception model for automated driving. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pages 796–803. IEEE, 2019.

26. Nivedita Tripathi and Senthil Yogamani. Trained trajectory based automated parking system using Visual SLAM. In Proceedings of the Computer Vision and Pattern Recognition Conference Workshops, 2021.

27. Michal Uricár, David Hurych, Pavel Krizek, et al. Challenges in designing datasets and validation for autonomous driving. In Proceedings of the International Conference on Computer Vision Theory and Applications, 2019.

28. Michal Uricár, Jan Ulicny, Ganesh Sistu, Hazem Rashed, et al. Desoiling dataset: Restoring soiled areas on automotive fisheye cameras. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0–0, 2019.

29. Ravi Yadav, Ahmed Samir, Hazem Rashed, Senthil Yogamani, and Rozenn Dahyot. Cnn based color and thermal image fusion for object detection in automated driving. Irish Machine Vision and Image Processing, 2, 2020.

30. Lucie Yahiaoui, Jonathan Horgan, Brian Deegan, Senthil Yogamani, Ciarán Hughes, and Patrick Denny. Overview and empirical analysis of isp parameter tuning for visual perception in autonomous driving. Journal of Imaging, 5(10):78, 2019.

31. Lucie Yahiaoui, Jonathan Horgan, Senthil Yogamani, Ciarán Hughes, and Brian Deegan. Impact analysis and tuning strategies for camera image signal processing parameters in computer vision. In Irish Machine Vision and Image Processing conference (IMVIP), volume 2, 2011.