# Towards Optimal Design of Datasets and Validation
Scheme for Autonomous Driving

Michal Uricar, Pavel Křížek, David Hurych, Senthil Yogamani

<figure><img src="image_1.png" /></figure>

To cite this version:

Michal Uricar, Pavel Křížek, David Hurych, Senthil Yogamani. Towards Optimal Design of Datasets and Validation Scheme for Autonomous Driving. 2024. hal-04576083

HAL Id: hal-04576083

https://hal.science/hal-04576083

Preprint submitted on 15 May 2024

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Towards Optimal Design of Datasets and Validation
Scheme for Autonomous Driving

Michal Uřičář, Pavel Křížek, David Hurych

Valeo R&D DVS
Prague, Czech Republic
{michal.uricar, pavel.krizek, david.hurych}@valeo.com

Senthil Yogamani

Valeo Vision Systems
Tuam, Ireland
senthil.yogamani@valeo.com

## Abstract

Autonomous driving is getting a lot of attention in the last decade and will be the hot topic at least until the first successful certification of a car with Level 5 autonomy [16]. There are many public datasets in the academic community. However, they are far away from what a robust industrial production system needs. There is a large gap between academic and industrial setting and a substantial way from a research prototype, built on public datasets, to a deployable solution which is a challenging task. In this paper, we focus on bad practices that often happen in the autonomous driving from an industrial deployment perspective. Data design deserves at least the same amount of attention as the model design. There is very little attention paid to these issues in the scientific community, and we hope this paper encourages better formalization of dataset design. More specifically, we focus on the datasets design and validation scheme for autonomous driving, where we would like to highlight the common problems, wrong assumptions, and steps towards avoiding them, as well as some open problems.

## 1 Introduction

We have the privilege to live in the exciting era of high pace research and development aiming for the full autonomy in the ground transportation, involving all major automotive industries. Nowadays, the sort of standard is the autonomy level 2 [16]. We can see the progress towards levels 3 and 4, and the ultimate goal is, of course, to achieve level 5, i.e., the real full autonomy. There has been rapid progress due to deep learning in various perception tasks such as object detection [26, 35, 22], depth estimation [18, 19], localization [32], motion estimation [21], semantic segmentation [6, 5, 27], blockage and adverse weather detection [34, 8], trajectory prediction [15, 30], multi-task learning [29, 20] and near-field perception systems [10, 33].

Naturally, there is a high motivation and willingness to speed up the progress in combination with the recent success of deep neural networks. However, this leads to the development of certain bad practices, which are progressively more and more visible in research papers. The goal of this paper is to determine some of the bad practices, especially those related to the issue of dataset design and validation scheme, and propose the ideas for fixing them. Apart from that, we would also like to identify several open problems for which the standardized solution is yet to be discovered.

The importance of dataset design is often overlooked in the computer vision community; the problem was addressed in detail in the ECCV workshop in 2016 [1]. In [17] the authors discuss issues with dataset bias and how to address them. In general, we can say that having a good and representative data is the crucial problem of virtually all machine learning techniques. Often, the applied algorithms come together with a requirement for data to be independent and identically distributed (i.i.d). However, this requirement is frequently broken and not checked for. Either the dataset parts are obtained from different distributions or their independence is questionable.
# Towards Optimal Design of Datasets and Validation
Scheme for Autonomous Driving

Michal Uřičář, Pavel Křížek, David Hurych

Valeo R&D DVS
Prague, Czech Republic
{michal.uricar, pavel.krizek, david.hurych}@valeo.com

Senthil Yogamani

Valeo Vision Systems
Tuam, Ireland
senthil.yogamani@valeo.com

## Abstract

Autonomous driving is getting a lot of attention in the last decade and will be the hot topic at least until the first successful certification of a car with Level 5 autonomy [16]. There are many public datasets in the academic community. However, they are far away from what a robust industrial production system needs. There is a large gap between academic and industrial setting and a substantial way from a research prototype, built on public datasets, to a deployable solution which is a challenging task. In this paper, we focus on bad practices that often happen in the autonomous driving from an industrial deployment perspective. Data design deserves at least the same amount of attention as the model design. There is very little attention paid to these issues in the scientific community, and we hope this paper encourages better formalization of dataset design. More specifically, we focus on the datasets design and validation scheme for autonomous driving, where we would like to highlight the common problems, wrong assumptions, and steps towards avoiding them, as well as some open problems.

## 1 Introduction

We have the privilege to live in the exciting era of high pace research and development aiming for the full autonomy in the ground transportation, involving all major automotive industries. Nowadays, the sort of standard is the autonomy level 2 [16]. We can see the progress towards levels 3 and 4, and the ultimate goal is, of course, to achieve level 5, i.e., the real full autonomy. There has been rapid progress due to deep learning in various perception tasks such as object detection [26, 35, 22], depth estimation [18, 19], localization [32], motion estimation [21], semantic segmentation [6, 5, 27], blockage and adverse weather detection [34, 8], trajectory prediction [15, 30], multi-task learning [29, 20] and near-field perception systems [10, 33].

Naturally, there is a high motivation and willingness to speed up the progress in combination with the recent success of deep neural networks. However, this leads to the development of certain bad practices, which are progressively more and more visible in research papers. The goal of this paper is to determine some of the bad practices, especially those related to the issue of dataset design and validation scheme, and propose the ideas for fixing them. Apart from that, we would also like to identify several open problems for which the standardized solution is yet to be discovered.

The importance of dataset design is often overlooked in the computer vision community; the problem was addressed in detail in the ECCV workshop in 2016 [1]. In [17] the authors discuss issues with dataset bias and how to address them. In general, we can say that having a good and representative data is the crucial problem of virtually all machine learning techniques. Often, the applied algorithms come together with a requirement for data to be independent and identically distributed (i.i.d). However, this requirement is frequently broken and not checked for. Either the dataset parts are obtained from different distributions or their independence is questionable.
<table><caption>Table 1: Popular automotive datasets for semantic segmentation.</caption><thead><tr><th>Dataset</th><th>CamVid [3]</th><th>Citiscapes [7]</th><th>Synthia [28]</th></tr></thead><tbody><tr><th>Annotation</th><td>700 images</td><td>5000 images</td><td>200,000 images</td></tr><tr><th>Note</th><td>Cambridge captures</td><td>Germany captures</td><td>Synthetic data</td></tr><tr><th>Dataset</th><td>Virtual KITTI [11]</td><td>Mapillary Vistas [24]</td><td>ApolloScape [14]</td></tr><tr><th>Annotation</th><td>21,260</td><td>25,000 images; 100 classes</td><td>143,000 images; 50 classes</td></tr><tr><th>Note</th><td>Synthetic data</td><td>Six continents</td><td>China captures</td></tr></tbody></table>

Frequently, we can also see that researchers blindly follow the results of the evaluation, represented by a key performance indicator (KPI), without making the trouble to check for the experiment correctness. With the increasing size of the datasets, the errors in annotations become significant, and the absence of a careful inspection might be dangerous. Especially, when we achieved the state where the improvements in decimals point are considered essential. A lot of models are discarded during the design, because of the lack of systematic analysis and the effort of getting a real insight. However, such a complete and systematic analysis might render some of these models more robust or to better generalize across different datasets.

Last but not least, we should emphasize the importance of fair comparisons regarding the used resources or a model complexity. Taking deep neural networks as a gentle example, what if we use an ensemble of simpler classifiers, trying to call the complexity of a neural net? Would we still see the same performance gap?

The rest of the paper is structured as follows. Section 2 discusses the current bad practices in dataset design, emphasizes its importance and lists open issues. Section 3 summarizes issues related to the validation of a safety system. Finally, Section 4 concludes the paper and provides future directions.

## 2 Dataset Design

Typically in academia, test and validation datasets are provided, and the goal is to get the best accuracy. However, in the industry deployment setting, datasets have to be designed interactively along with the model design. Unfortunately, there is minimal systematic design effort as it is difficult to formulate the problem and quantify the quality of datasets. First, there is a data capture process where cameras mounted on a vehicle capture necessary data. This process has to be repeated across many scenarios like different countries, weather conditions, times of the day, etc.

Table 1 lists the popular automotive datasets for semantic segmentation task. There is visible progress towards the increased number of samples as well as a change in direction to realistic data.

**Typical scenarios of dataset design in autonomous driving.** There are three scenarios currently possible for automated driving researchers: (i) academic setup where public datasets are used as they are. However, these have commercial licensing restrictions and cannot be used freely for industrial research nor for final production, (ii) proof of concept setup where dataset is collected for a restricted scope, e.g. one city, regular weather conditions, and (iii) a production system where dataset has to be designed for all scenarios, e.g., a large set of countries, weather conditions, etc.

Here, we focus our discussion on the third type. A typical process for dataset design comprises of the following steps. Firstly, the requirements are created for coverage of countries, weather conditions, object diversity, etc. Then video captures are acquired and their frames arbitrarily sampled without a systematic sampling strategy. Next step is creating the training, validation and test splits from all gathered images. This can be done either randomly, or in a better case by stratified sampling, retaining the class distributions among the splits. Then, after the model is trained, one has to evaluate the KPIs, such as the mean intersection over union (mIoU). Last but not least step is a naïve search for corner cases in the test split and addition of such infected samples to the training split (in best case by obtaining new data, which look similar to the testing ones).

**Open issues and suggestions for improvement.** Here we list several important open issues and suggestions for improvement.
<table><caption>Table 1: Popular automotive datasets for semantic segmentation.</caption><thead><tr><th>Dataset</th><th>CamVid [3]</th><th>Citiscapes [7]</th><th>Synthia [28]</th></tr></thead><tbody><tr><th>Annotation</th><td>700 images</td><td>5000 images</td><td>200,000 images</td></tr><tr><th>Note</th><td>Cambridge captures</td><td>Germany captures</td><td>Synthetic data</td></tr><tr><th>Dataset</th><td>Virtual KITTI [11]</td><td>Mapillary Vistas [24]</td><td>ApolloScape [14]</td></tr><tr><th>Annotation</th><td>21,260</td><td>25,000 images; 100 classes</td><td>143,000 images; 50 classes</td></tr><tr><th>Note</th><td>Synthetic data</td><td>Six continents</td><td>China captures</td></tr></tbody></table>

Frequently, we can also see that researchers blindly follow the results of the evaluation, represented by a key performance indicator (KPI), without making the trouble to check for the experiment correctness. With the increasing size of the datasets, the errors in annotations become significant, and the absence of a careful inspection might be dangerous. Especially, when we achieved the state where the improvements in decimals point are considered essential. A lot of models are discarded during the design, because of the lack of systematic analysis and the effort of getting a real insight. However, such a complete and systematic analysis might render some of these models more robust or to better generalize across different datasets.

Last but not least, we should emphasize the importance of fair comparisons regarding the used resources or a model complexity. Taking deep neural networks as a gentle example, what if we use an ensemble of simpler classifiers, trying to call the complexity of a neural net? Would we still see the same performance gap?

The rest of the paper is structured as follows. Section 2 discusses the current bad practices in dataset design, emphasizes its importance and lists open issues. Section 3 summarizes issues related to the validation of a safety system. Finally, Section 4 concludes the paper and provides future directions.

## 2 Dataset Design

Typically in academia, test and validation datasets are provided, and the goal is to get the best accuracy. However, in the industry deployment setting, datasets have to be designed interactively along with the model design. Unfortunately, there is minimal systematic design effort as it is difficult to formulate the problem and quantify the quality of datasets. First, there is a data capture process where cameras mounted on a vehicle capture necessary data. This process has to be repeated across many scenarios like different countries, weather conditions, times of the day, etc.

Table 1 lists the popular automotive datasets for semantic segmentation task. There is visible progress towards the increased number of samples as well as a change in direction to realistic data.

**Typical scenarios of dataset design in autonomous driving.** There are three scenarios currently possible for automated driving researchers: (i) academic setup where public datasets are used as they are. However, these have commercial licensing restrictions and cannot be used freely for industrial research nor for final production, (ii) proof of concept setup where dataset is collected for a restricted scope, e.g. one city, regular weather conditions, and (iii) a production system where dataset has to be designed for all scenarios, e.g., a large set of countries, weather conditions, etc.

Here, we focus our discussion on the third type. A typical process for dataset design comprises of the following steps. Firstly, the requirements are created for coverage of countries, weather conditions, object diversity, etc. Then video captures are acquired and their frames arbitrarily sampled without a systematic sampling strategy. Next step is creating the training, validation and test splits from all gathered images. This can be done either randomly, or in a better case by stratified sampling, retaining the class distributions among the splits. Then, after the model is trained, one has to evaluate the KPIs, such as the mean intersection over union (mIoU). Last but not least step is a naïve search for corner cases in the test split and addition of such infected samples to the training split (in best case by obtaining new data, which look similar to the testing ones).

**Open issues and suggestions for improvement.** Here we list several important open issues and suggestions for improvement.
Many visual perception tasks, like semantic segmentation, need a very expensive annotation, leading to unnecessarily smaller datasets. Synthetic datasets, like [28, 11, 9, 23], can be useful as potential mitigation of the lack of data. However, domain adaptation is usually required, and it is not clear what ratio of synthetic to real data would be still beneficial.

Due to the popularity of the AD, the available datasets are snowballing, and the choice of datasets starts to be a problem itself. Moreover, in research, there is no synchronization on datasets, and it is difficult to compare different works justly. It might be helpful if the community agrees on some standardized dataset (combining the strengths and weaknesses of all of them) to have a possibility to compare algorithms more thoroughly and honestly. For example, there is no available dataset with wide-angle fisheye camera images. Such a camera is a standard in the AD for capturing the 360° view around the vehicle. A publicly available dataset with multiple cocoon cameras, which are typical for the AD, is also missing.

An automated sampling mechanism for acquiring the training images the goal of which is to get rid of redundant samples and providing a maximal diversity is an open problem. The dataset and model design is done iteratively, and samples are added on the go to improve the model performance. This process is dangerous since such an approach might easily break the i.i.d. requirements on data. A corner case mining is a related topic, where difficult samples are identified, and their knowledge is used for improving the performance. Note, that this process usually takes into the account the sample similarity in the image space. However, one would benefit if the mining would be based on the similarity measured in the classifier's feature space.

Data augmentation is an important mechanism to obtain samples for difficult or rare scenarios. We can take an automated parking system with pedestrian detection as an example. We would like to have data where children are playing with a ball and sometimes blindly follow the ball which gets on the collision path with the vehicle. It is clear that we cannot record such scenario, due to the safety and legal issues. We see the possibility in bypassing such scenario by recording in a controlled environment and applying GANs [12] for the domain transfer to fit the AD needs [4, 13].

## 3 Validation

Validation scheme is another critical part of the current research. In the automotive industry, this topic is even more critical, due to the very stringent requirements on safety. However, we would like to emphasize here, that the common problems with validation are shared among other fields as well.

The AD systems have unique criteria due to functional safety and traceability issues. The artificial intelligence software for the AD has to comply with strict processes like ISO26262 to ensure functional safety. Thus, apart from accuracy validation, it is essential to do rigorous testing of software stability. However, unit testing of AI algorithms comes with additional challenges, like large dimensionality of data, or the abstract nature of the model and automated code generation. Due to these challenges, it is difficult to write tests manually. There have been attempts to generate tests automatically using deep learning like DeepXplore [25] and DeepTest [31] focused on the AD.

**The need for virtual validation and their limitations.** In the automotive industry, one tests the algorithms by recording a lot of hours of various scenarios which should ideally cover all possible real-world situations. Afterward, these hours of recordings are statistically evaluated, and the algorithm is allowed to go for the start of production only if fulfilling the strict requirements, which were formulated and ideally also fixed at the beginning of the project. Despite sounding completely legitimate at first, such an approach has several important flaws. The first, and probably also the most important one is the physical and practical impossibility to cover all real-world scenarios. Let us formulate an example explaining our claims: we want to test the automated parking functionality enhanced by pedestrian detection algorithm. The vehicle should send a command for braking if there is a pedestrian within critical distance and on the collision course present. Now, imagine a legitimate scenario, where the pedestrian should be a toddler sitting right behind the car. In most of the countries across the whole globe, their laws prohibit making of such recordings. The second problem is in setting up the requirements at the beginning of the project. If we agree that it is not possible to record all real-world situations, freezing the requirements tends to influence the scenarios to record as well as their complexity.
Many visual perception tasks, like semantic segmentation, need a very expensive annotation, leading to unnecessarily smaller datasets. Synthetic datasets, like [28, 11, 9, 23], can be useful as potential mitigation of the lack of data. However, domain adaptation is usually required, and it is not clear what ratio of synthetic to real data would be still beneficial.

Due to the popularity of the AD, the available datasets are snowballing, and the choice of datasets starts to be a problem itself. Moreover, in research, there is no synchronization on datasets, and it is difficult to compare different works justly. It might be helpful if the community agrees on some standardized dataset (combining the strengths and weaknesses of all of them) to have a possibility to compare algorithms more thoroughly and honestly. For example, there is no available dataset with wide-angle fisheye camera images. Such a camera is a standard in the AD for capturing the 360° view around the vehicle. A publicly available dataset with multiple cocoon cameras, which are typical for the AD, is also missing.

An automated sampling mechanism for acquiring the training images the goal of which is to get rid of redundant samples and providing a maximal diversity is an open problem. The dataset and model design is done iteratively, and samples are added on the go to improve the model performance. This process is dangerous since such an approach might easily break the i.i.d. requirements on data. A corner case mining is a related topic, where difficult samples are identified, and their knowledge is used for improving the performance. Note, that this process usually takes into the account the sample similarity in the image space. However, one would benefit if the mining would be based on the similarity measured in the classifier's feature space.

Data augmentation is an important mechanism to obtain samples for difficult or rare scenarios. We can take an automated parking system with pedestrian detection as an example. We would like to have data where children are playing with a ball and sometimes blindly follow the ball which gets on the collision path with the vehicle. It is clear that we cannot record such scenario, due to the safety and legal issues. We see the possibility in bypassing such scenario by recording in a controlled environment and applying GANs [12] for the domain transfer to fit the AD needs [4, 13].

## 3 Validation

Validation scheme is another critical part of the current research. In the automotive industry, this topic is even more critical, due to the very stringent requirements on safety. However, we would like to emphasize here, that the common problems with validation are shared among other fields as well.

The AD systems have unique criteria due to functional safety and traceability issues. The artificial intelligence software for the AD has to comply with strict processes like ISO26262 to ensure functional safety. Thus, apart from accuracy validation, it is essential to do rigorous testing of software stability. However, unit testing of AI algorithms comes with additional challenges, like large dimensionality of data, or the abstract nature of the model and automated code generation. Due to these challenges, it is difficult to write tests manually. There have been attempts to generate tests automatically using deep learning like DeepXplore [25] and DeepTest [31] focused on the AD.

**The need for virtual validation and their limitations.** In the automotive industry, one tests the algorithms by recording a lot of hours of various scenarios which should ideally cover all possible real-world situations. Afterward, these hours of recordings are statistically evaluated, and the algorithm is allowed to go for the start of production only if fulfilling the strict requirements, which were formulated and ideally also fixed at the beginning of the project. Despite sounding completely legitimate at first, such an approach has several important flaws. The first, and probably also the most important one is the physical and practical impossibility to cover all real-world scenarios. Let us formulate an example explaining our claims: we want to test the automated parking functionality enhanced by pedestrian detection algorithm. The vehicle should send a command for braking if there is a pedestrian within critical distance and on the collision course present. Now, imagine a legitimate scenario, where the pedestrian should be a toddler sitting right behind the car. In most of the countries across the whole globe, their laws prohibit making of such recordings. The second problem is in setting up the requirements at the beginning of the project. If we agree that it is not possible to record all real-world situations, freezing the requirements tends to influence the scenarios to record as well as their complexity.
One might say, that the solution is obvious— use virtual validation or some other workaround, like usage of dummy objects. However, doing so bring other problems, such as the realism of these artificial scenarios. It is clear that we will never be able to evaluate it for some of the real-world situations.

A common problem (not only in the automotive industry) is the design of the validation scheme itself. Typically, we can see that the algorithm was optimized for a specific criterion, using a particular loss function. Ideally, we should see the same loss function used in the evaluation, alas, it is not uncommon to see something not even similar to the original loss used in testing. The problem with this setup is that it is not possible to optimize for a beforehand unknown criterion.

**Model survivorship bias.** This mistreated optimization is connected to another significant problem— overfitting to (not only) standard datasets. In research, but also in industry, only the models which are obtaining the best results are reported or survive. Nobody tells, how many times the model failed on those data before it was tweaked enough to provide the best results. Not reporting the negative results is counter-productive [2]. With the increase of the deep learning models in the game, this problem is even worse. Deep networks are known to easily fit random labels, even for randomly generated images [36].

**Complete reporting and replicability.** Another, and unfortunately also very frequent, problem is that certain important statistics are not being reported. Only rarely, we can see that authors of the research paper give away also their splits of datasets to training/validation/testing parts. Quite frequently, they do not even bother mentioning the key statistics, such as the number of samples they used, or how the splits were obtained. This problem is connected with a choice of machine learning method, which is not well justified. For example, a lot of techniques come with an assumption about the data distribution. Important and integral part of each experimental evaluation should be its replicability by a non-involved party.

**Cross-validation and the law of small numbers.** The emphasis on early deployment in the automotive industry often leads to unjustified design choices, which do not have support in data. The infamous law of small numbers gets into the practice. Usually, the researchers and developers have to deal with the insufficient amount of data to do a statistical evaluation. Then, due to the lack of time, some early decisions are made, and the outcomes are wrong with a higher probability, the smaller sample of data was used for performing the evaluation. Just imagine you have a fair coin, so the expected probability of getting head after flipping it is 0.5. Let us conduct the following experiment. Flip the coin ten times and count the number of heads/tails. Very likely, you will not get the five heads. Now flip the same coin a thousand times, marking the number of heads. This time, the number of heads will be close to five hundred. If one does only the first experiment, he might be in temptation to question the fairness of the coin. While after the second one, such conduct feels unjustified. It is common to see quite a small number of samples for some particular tasks, as well as only one evaluation over a single split of data.

**The need for customized evaluation metrics.** Standard performance measures, such as mIoU for semantic segmentation, may not translate well to the end user application needs. Let us take the automated parking functionality with pedestrian detection as an example again. A perfect segmentation of a pedestrian is not necessary, and just a coarse detection is sufficient for initiating the braking. Another example is recognition of the lane markings— there are nice examples, where a higher mIoU does not necessarily lead to a better segmentation of the main shape of the marking, which is crucial for its recognition. In both of these cases, custom tailored evaluation metric is the key for a better algorithm. And visual checks of the results are a must.

# 4 Conclusions

In this paper, we attempt to emphasize the importance of dataset design and validation for the AD systems. Both dataset design and validation are highly overlooked topics which have created a large gap between academic research and industrial deployment setting. There is a considerable effort to go from a model which achieves state-of-the-art results in an academic context to the development of a safe and robust system deployed in a commercial car. Unfortunately, there is very little scientific effort spent in this direction. We have tried to summarize the bad practices and listed open research
One might say, that the solution is obvious— use virtual validation or some other workaround, like usage of dummy objects. However, doing so bring other problems, such as the realism of these artificial scenarios. It is clear that we will never be able to evaluate it for some of the real-world situations.

A common problem (not only in the automotive industry) is the design of the validation scheme itself. Typically, we can see that the algorithm was optimized for a specific criterion, using a particular loss function. Ideally, we should see the same loss function used in the evaluation, alas, it is not uncommon to see something not even similar to the original loss used in testing. The problem with this setup is that it is not possible to optimize for a beforehand unknown criterion.

**Model survivorship bias.** This mistreated optimization is connected to another significant problem— overfitting to (not only) standard datasets. In research, but also in industry, only the models which are obtaining the best results are reported or survive. Nobody tells, how many times the model failed on those data before it was tweaked enough to provide the best results. Not reporting the negative results is counter-productive [2]. With the increase of the deep learning models in the game, this problem is even worse. Deep networks are known to easily fit random labels, even for randomly generated images [36].

**Complete reporting and replicability.** Another, and unfortunately also very frequent, problem is that certain important statistics are not being reported. Only rarely, we can see that authors of the research paper give away also their splits of datasets to training/validation/testing parts. Quite frequently, they do not even bother mentioning the key statistics, such as the number of samples they used, or how the splits were obtained. This problem is connected with a choice of machine learning method, which is not well justified. For example, a lot of techniques come with an assumption about the data distribution. Important and integral part of each experimental evaluation should be its replicability by a non-involved party.

**Cross-validation and the law of small numbers.** The emphasis on early deployment in the automotive industry often leads to unjustified design choices, which do not have support in data. The infamous law of small numbers gets into the practice. Usually, the researchers and developers have to deal with the insufficient amount of data to do a statistical evaluation. Then, due to the lack of time, some early decisions are made, and the outcomes are wrong with a higher probability, the smaller sample of data was used for performing the evaluation. Just imagine you have a fair coin, so the expected probability of getting head after flipping it is 0.5. Let us conduct the following experiment. Flip the coin ten times and count the number of heads/tails. Very likely, you will not get the five heads. Now flip the same coin a thousand times, marking the number of heads. This time, the number of heads will be close to five hundred. If one does only the first experiment, he might be in temptation to question the fairness of the coin. While after the second one, such conduct feels unjustified. It is common to see quite a small number of samples for some particular tasks, as well as only one evaluation over a single split of data.

**The need for customized evaluation metrics.** Standard performance measures, such as mIoU for semantic segmentation, may not translate well to the end user application needs. Let us take the automated parking functionality with pedestrian detection as an example again. A perfect segmentation of a pedestrian is not necessary, and just a coarse detection is sufficient for initiating the braking. Another example is recognition of the lane markings— there are nice examples, where a higher mIoU does not necessarily lead to a better segmentation of the main shape of the marking, which is crucial for its recognition. In both of these cases, custom tailored evaluation metric is the key for a better algorithm. And visual checks of the results are a must.

# 4 Conclusions

In this paper, we attempt to emphasize the importance of dataset design and validation for the AD systems. Both dataset design and validation are highly overlooked topics which have created a large gap between academic research and industrial deployment setting. There is a considerable effort to go from a model which achieves state-of-the-art results in an academic context to the development of a safe and robust system deployed in a commercial car. Unfortunately, there is very little scientific effort spent in this direction. We have tried to summarize the bad practices and listed open research
problems based on our experience in this area for more than ten years. Hopefully, this encourages further scientific research in this area and places a seed for future improvement.

## References

[1] ECCV 2016 Workshop on Datasets and Performance Analysis in Early Vision. https://hci.iwr.uni-heidelberg.de/eccv16ws-datasets, 2016.

[2] Ali Borji. Negative results in computer vision: A perspective. Image Vision Comput., 69:1–8, 2018.

[3] Gabriel J Brostow, Jamie Shotton, Julien Fauqueur, and Roberto Cipolla. Segmentation and recognition using structure from motion point clouds. In *European conference on computer vision*, pages 44–57. Springer, 2008.

[4] Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A Efros. Everybody dance now. *arXiv preprint arXiv:1808.07371*, 2018.

[5] S. Chennupati, Ganesh Sistu., Senthil Yogamani., and Samir Rawashdeh. Auxnet: Auxiliary tasks enhanced semantic segmentation for automated driving. In *Proceedings of the International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISAPP)*, 2019.

[6] Sumanth Chennupati, Venkatraman Narayanan, Ganesh Sistu, Senthil Yogamani, and Samir A Rawashdeh. Learning panoptic segmentation from instance contours. In *2021 IEEE International Conference on Robotics and Automation (ICRA)*, pages 9586–9593. IEEE, 2021.

[7] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. *arXiv preprint arXiv:1604.01685*, 2016.

[8] Mahesh M Dhananjaya, Varun Ravi Kumar, and Senthil Yogamani. Weather and light level classification for autonomous driving: Dataset, baseline and active learning. In *2021 IEEE International Intelligent Transportation Systems Conference (ITSC)*, pages 2816–2821, 2021.

[9] Alexey Dosovitskiy, Germán Ros, Felipe Codevilla, Antonio López, and Vladlen Koltun. CARLA: an open urban driving simulator. In *1st Annual Conference on Robot Learning, CoRL 2017, Mountain View, California, USA, November 13-15, 2017, Proceedings*, pages 1–16, 2017.

[10] Ciarán Eising, Jonathan Horgan, and Senthil Yogamani. Near-field perception for low-speed vehicle automation using surround-view fisheye cameras. *IEEE Transactions on Intelligent Transportation Systems*, 23(9):13976–13993, 2021.

[11] Adrien Gaidon, Qiao Wang, Yohann Cabon, and Eleonora Vig. Virtual worlds as proxy for multi-object tracking analysis. In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, pages 4340–4349, 2016.

[12] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial networks. *CoRR*, abs/1406.2661, 2014.

[13] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. CyCADA: Cycle-consistent adversarial domain adaptation. In Jennifer Dy and Andreas Krause, editors, *Proceedings of the 35th International Conference on Machine Learning*, volume 80 of *Proceedings of Machine Learning Research*, Stockholm Sweden, 10–15 Jul 2018. PMLR.

[14] Xinyu Huang, Xinjing Cheng, Qichuan Geng, Binbin Cao, Dingfu Zhou, Peng Wang, Yuanqing Lin, and Ruigang Yang. The apolloscape dataset for autonomous driving. *arXiv preprint arXiv:1803.06184*, 2018.

[15] Ciarán Hughes, Sunil Chandra, Ganesh Sistu, Jonathan Horgan, Brian Deegan, Sumanth Chennupati, and Senthil Yogamani. Drivespace: towards context-aware drivable area detection. *Electronic Imaging*, 31:1–9, 2019.

[16] SAE International. The wayback machine, September 2017. https://web.archive.org/web/20170903105244/https://www.sae.org/misc/pdfs/automated_driving.pdf.

[17] Aditya Khosla, Tinghui Zhou, Tomasz Malisiewicz, Alexei A Efros, and Antonio Torralba. Undoing the damage of dataset bias. In *European Conference on Computer Vision*, pages 158–171. Springer, 2012.

[18] Varun Ravi Kumar, Marvin Klingner, Senthil Yogamani, Markus Bach, Stefan Milz, Tim Fingscheidt, and Patrick Mäder. Svdistnet: Self-supervised near-field distance estimation on surround view fisheye cameras. *IEEE Transactions on Intelligent Transportation Systems*, 23(8):10252–10261, 2021.
problems based on our experience in this area for more than ten years. Hopefully, this encourages further scientific research in this area and places a seed for future improvement.

## References

[1] ECCV 2016 Workshop on Datasets and Performance Analysis in Early Vision. https://hci.iwr.uni-heidelberg.de/eccv16ws-datasets, 2016.

[2] Ali Borji. Negative results in computer vision: A perspective. Image Vision Comput., 69:1–8, 2018.

[3] Gabriel J Brostow, Jamie Shotton, Julien Fauqueur, and Roberto Cipolla. Segmentation and recognition using structure from motion point clouds. In *European conference on computer vision*, pages 44–57. Springer, 2008.

[4] Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A Efros. Everybody dance now. *arXiv preprint arXiv:1808.07371*, 2018.

[5] S. Chennupati, Ganesh Sistu., Senthil Yogamani., and Samir Rawashdeh. Auxnet: Auxiliary tasks enhanced semantic segmentation for automated driving. In *Proceedings of the International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISAPP)*, 2019.

[6] Sumanth Chennupati, Venkatraman Narayanan, Ganesh Sistu, Senthil Yogamani, and Samir A Rawashdeh. Learning panoptic segmentation from instance contours. In *2021 IEEE International Conference on Robotics and Automation (ICRA)*, pages 9586–9593. IEEE, 2021.

[7] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. *arXiv preprint arXiv:1604.01685*, 2016.

[8] Mahesh M Dhananjaya, Varun Ravi Kumar, and Senthil Yogamani. Weather and light level classification for autonomous driving: Dataset, baseline and active learning. In *2021 IEEE International Intelligent Transportation Systems Conference (ITSC)*, pages 2816–2821, 2021.

[9] Alexey Dosovitskiy, Germán Ros, Felipe Codevilla, Antonio López, and Vladlen Koltun. CARLA: an open urban driving simulator. In *1st Annual Conference on Robot Learning, CoRL 2017, Mountain View, California, USA, November 13-15, 2017, Proceedings*, pages 1–16, 2017.

[10] Ciarán Eising, Jonathan Horgan, and Senthil Yogamani. Near-field perception for low-speed vehicle automation using surround-view fisheye cameras. *IEEE Transactions on Intelligent Transportation Systems*, 23(9):13976–13993, 2021.

[11] Adrien Gaidon, Qiao Wang, Yohann Cabon, and Eleonora Vig. Virtual worlds as proxy for multi-object tracking analysis. In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, pages 4340–4349, 2016.

[12] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial networks. *CoRR*, abs/1406.2661, 2014.

[13] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. CyCADA: Cycle-consistent adversarial domain adaptation. In Jennifer Dy and Andreas Krause, editors, *Proceedings of the 35th International Conference on Machine Learning*, volume 80 of *Proceedings of Machine Learning Research*, Stockholm Sweden, 10–15 Jul 2018. PMLR.

[14] Xinyu Huang, Xinjing Cheng, Qichuan Geng, Binbin Cao, Dingfu Zhou, Peng Wang, Yuanqing Lin, and Ruigang Yang. The apolloscape dataset for autonomous driving. *arXiv preprint arXiv:1803.06184*, 2018.

[15] Ciarán Hughes, Sunil Chandra, Ganesh Sistu, Jonathan Horgan, Brian Deegan, Sumanth Chennupati, and Senthil Yogamani. Drivespace: towards context-aware drivable area detection. *Electronic Imaging*, 31:1–9, 2019.

[16] SAE International. The wayback machine, September 2017. https://web.archive.org/web/20170903105244/https://www.sae.org/misc/pdfs/automated_driving.pdf.

[17] Aditya Khosla, Tinghui Zhou, Tomasz Malisiewicz, Alexei A Efros, and Antonio Torralba. Undoing the damage of dataset bias. In *European Conference on Computer Vision*, pages 158–171. Springer, 2012.

[18] Varun Ravi Kumar, Marvin Klingner, Senthil Yogamani, Markus Bach, Stefan Milz, Tim Fingscheidt, and Patrick Mäder. Svdistnet: Self-supervised near-field distance estimation on surround view fisheye cameras. *IEEE Transactions on Intelligent Transportation Systems*, 23(8):10252–10261, 2021.
[19] Varun Ravi Kumar, Stefan Milz, Christian Witt, Martin Simon, et al. Near-field depth estimation using monocular fisheye camera: A semi-supervised learning approach using sparse lidar data. In CVPR Workshop, volume 7, page 2, 2018.

[20] Isabelle Leang, Ganesh Sistu, Fabian Bürger, Andrei Bursuc, et al. Dynamic task weighting methods for multi-task networks in autonomous driving systems. In 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), pages 1–8. IEEE, 2020.

[21] Eslam Mohamed, Mahmoud Ewaisha, Mennatullah Siam, Hazem Rashed, Senthil Yogamani, Waleed Hamdy, Mohamed El-Dakdouky, and Ahmad El-Sallab. Monocular instance motion segmentation for autonomous driving: Kitti instancemotseg dataset and multi-task baseline. In 2021 IEEE Intelligent Vehicles Symposium (IV), pages 114–121. IEEE, 2021.

[22] Sambit Mohapatra, Senthil Yogamani, Heinrich Gotzig, Stefan Milz, and Patrick Mader. Bevdnet: bird's eye view lidar point cloud based real-time 3d object detection for autonomous driving. In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pages 2809–2815. IEEE, 2021.

[23] Matthias Mueller, Alexey Dosovitskiy, Bernard Ghanem, and Vladlen Koltun. Driving policy transfer via modularity and abstraction. In 2nd Annual Conference on Robot Learning, CoRL 2018, Zürich, Switzerland, 29-31 October 2018, Proceedings, pages 1–15, 2018.

[24] Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulò, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In ICCV, pages 5000–5009, 2017.

[25] Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. Deepxplore: Automated whitebox testing of deep learning systems. In Proceedings of the 26th Symposium on Operating Systems Principles, pages 1–18. ACM, 2017.

[26] Hazem Rashed, Eslam Mohamed, Ganesh Sistu, Varun Ravi Kumar, Ciarán Eising, Ahmad El-Sallab, and SK Yogamani. Fisheyeyolo: Object detection on fisheye cameras for autonomous driving. In Proceedings of the Machine Learning for Autonomous Driving NeurIPS 2020 Virtual Workshop, Virtual, volume 11, 2020.

[27] Hazem Rashed, Senthil Yogamani, Ahmad El-Sallab, Pavel Krizek, and Mohamed El-Helw. Optical flow augmented semantic segmentation networks for automated driving. In Proceedings of the International Conference on Computer Vision Theory and Applications, 2019.

[28] German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3234–3243, 2016.

[29] Ganesh Sistu, Isabelle Leang, and Senthil Yogamani. Real-time joint object detection and semantic segmentation network for automated driving. arXiv preprint arXiv:1901.03912, 2019.

[30] Fergal Stapleton, Edgar Galván, Ganesh Sistu, and Senthil Yogamani. Neuroevolutionary multi-objective approaches to trajectory prediction in autonomous vehicles. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, pages 675–678, 2022.

[31] Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. Deeptest: Automated testing of deep-neural-network-driven autonomous cars. In Proceedings of the 40th International Conference on Software Engineering, pages 303–314. ACM, 2018.

[32] Nivedita Tripathi and Senthil Yogamani. Trained trajectory based automated parking system using Visual SLAM. In Proceedings of the Computer Vision and Pattern Recognition Conference Workshops, 2021.

[33] Michal Uricár, David Hurych, Pavel Krizek, et al. Challenges in designing datasets and validation for autonomous driving. In Proceedings of the International Conference on Computer Vision Theory and Applications, 2019.

[34] Michal Uricár, Jan Ulicny, Ganesh Sistu, Hazem Rashed, et al. Desoiling dataset: Restoring soiled areas on automotive fisheye cameras. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0–0, 2019.

[35] Lucie Yahiaoui, Jonathan Horgan, Brian Deegan, Senthil Yogamani, Ciarán Hughes, and Patrick Denny. Overview and empirical analysis of isp parameter tuning for visual perception in autonomous driving. Journal of Imaging, 5(10):78, 2019.

[36] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. CoRR, abs/1611.03530, 2016.
[19] Varun Ravi Kumar, Stefan Milz, Christian Witt, Martin Simon, et al. Near-field depth estimation using monocular fisheye camera: A semi-supervised learning approach using sparse lidar data. In CVPR Workshop, volume 7, page 2, 2018.

[20] Isabelle Leang, Ganesh Sistu, Fabian Bürger, Andrei Bursuc, et al. Dynamic task weighting methods for multi-task networks in autonomous driving systems. In 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), pages 1–8. IEEE, 2020.

[21] Eslam Mohamed, Mahmoud Ewaisha, Mennatullah Siam, Hazem Rashed, Senthil Yogamani, Waleed Hamdy, Mohamed El-Dakdouky, and Ahmad El-Sallab. Monocular instance motion segmentation for autonomous driving: Kitti instancemotseg dataset and multi-task baseline. In 2021 IEEE Intelligent Vehicles Symposium (IV), pages 114–121. IEEE, 2021.

[22] Sambit Mohapatra, Senthil Yogamani, Heinrich Gotzig, Stefan Milz, and Patrick Mader. Bevdnet: bird's eye view lidar point cloud based real-time 3d object detection for autonomous driving. In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pages 2809–2815. IEEE, 2021.

[23] Matthias Mueller, Alexey Dosovitskiy, Bernard Ghanem, and Vladlen Koltun. Driving policy transfer via modularity and abstraction. In 2nd Annual Conference on Robot Learning, CoRL 2018, Zürich, Switzerland, 29-31 October 2018, Proceedings, pages 1–15, 2018.

[24] Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulò, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In ICCV, pages 5000–5009, 2017.

[25] Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. Deepxplore: Automated whitebox testing of deep learning systems. In Proceedings of the 26th Symposium on Operating Systems Principles, pages 1–18. ACM, 2017.

[26] Hazem Rashed, Eslam Mohamed, Ganesh Sistu, Varun Ravi Kumar, Ciarán Eising, Ahmad El-Sallab, and SK Yogamani. Fisheyeyolo: Object detection on fisheye cameras for autonomous driving. In Proceedings of the Machine Learning for Autonomous Driving NeurIPS 2020 Virtual Workshop, Virtual, volume 11, 2020.

[27] Hazem Rashed, Senthil Yogamani, Ahmad El-Sallab, Pavel Krizek, and Mohamed El-Helw. Optical flow augmented semantic segmentation networks for automated driving. In Proceedings of the International Conference on Computer Vision Theory and Applications, 2019.

[28] German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3234–3243, 2016.

[29] Ganesh Sistu, Isabelle Leang, and Senthil Yogamani. Real-time joint object detection and semantic segmentation network for automated driving. arXiv preprint arXiv:1901.03912, 2019.

[30] Fergal Stapleton, Edgar Galván, Ganesh Sistu, and Senthil Yogamani. Neuroevolutionary multi-objective approaches to trajectory prediction in autonomous vehicles. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, pages 675–678, 2022.

[31] Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. Deeptest: Automated testing of deep-neural-network-driven autonomous cars. In Proceedings of the 40th International Conference on Software Engineering, pages 303–314. ACM, 2018.

[32] Nivedita Tripathi and Senthil Yogamani. Trained trajectory based automated parking system using Visual SLAM. In Proceedings of the Computer Vision and Pattern Recognition Conference Workshops, 2021.

[33] Michal Uricár, David Hurych, Pavel Krizek, et al. Challenges in designing datasets and validation for autonomous driving. In Proceedings of the International Conference on Computer Vision Theory and Applications, 2019.

[34] Michal Uricár, Jan Ulicny, Ganesh Sistu, Hazem Rashed, et al. Desoiling dataset: Restoring soiled areas on automotive fisheye cameras. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0–0, 2019.

[35] Lucie Yahiaoui, Jonathan Horgan, Brian Deegan, Senthil Yogamani, Ciarán Hughes, and Patrick Denny. Overview and empirical analysis of isp parameter tuning for visual perception in autonomous driving. Journal of Imaging, 5(10):78, 2019.

[36] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. CoRR, abs/1611.03530, 2016.