EyeDAS: Securing Perception of Autonomous Cars Against the Stereoblindness Syndrome
Authors: Efrat Levy
Ben Nassi
Raz Swissa
Yuval Elovici
Venue: Submitted to ICLR 2023 (2023)

Abstract: The ability to detect whether an object is a 2D or 3D object is extremely important in autonomous driving, since a detection error can have life-threatening consequences, endangering the safety of the driver, passengers, pedestrians, and others on the road. 
Methods proposed to distinguish between 2 and 3D objects (e.g., liveness detection methods) are not suitable for autonomous driving, because they are object dependent or do not consider the constraints associated with autonomous driving (e.g., the need for real-time decision-making while the vehicle is moving).
In this paper, we present EyeDAS, a novel few-shot learning-based method aimed at securing an object detector (OD) against the threat posed by the stereoblindness syndrome (i.e., the inability to distinguish between 2D and 3D objects).
We evaluate EyeDAS's real-time performance using 2,000 objects extracted from seven YouTube video recordings of street views taken by a dash cam from the driver's seat perspective.
When applying EyeDAS to seven state-of-the-art ODs as a countermeasure, EyeDAS was able to reduce the 2D misclassification rate from 71.42-100% to 2.4% with a 3D misclassification rate of 0% (TPR of 1.0). 
We also show that EyeDAS outperforms the baseline method and achieves an AUC of over 0.999.

Under review as a conference paper at ICLR 2023
SECURING PERCEPTION OF AUTONOMOUS CARS AGAINST THE STEREOBLINDNESS SYNDROME
EYEDAS:
Anonymous authors
ABSTRACT
The ability detect whether an object 2D or 3D object is extremely important in autonomous driving, since a detection error can have life- threatening consequences, endangering the safety of the driver, passengers, pedestrians, and others on the road. Methods proposed to distinguish between 2 and 3D objects (e.g., liveness detection methods) are not. suitable for autonomous driving, because they are object dependent or do not consider the constraints associated with autonomous driving (e.g,, the need for real-time decision-making while the vehicle is moving). In this paper, we present EycDAS, a novel few-shot learning-based method aimed at securing an object detector (OD) against the threat posed by the stereoblindness syndrome (i.e., the inability to distinguish bet ween 2D and 3D objects). We evaluate EyeDAS's real-time performance using 2,000 objects extracted from seven YouTTub video recordings of street views taken by dash cam from the driver's seat perspective. When applying EyeDAS of-the-art ODs as a countermeasure, EyeDAS was able to reduce the 2D misclassification rate from 71.42-100% 2.4% with a 3D misclassification rate of 0% (TPR of 1.0). Also, EyeDAS outperforms the baseline method and achieves an AUC of over 0.999.
seven state-
INTRODUCTION
After years of research and development, automobile technology is rapidly approaching the point at which human drivers can be replaced, commercial cars are now capable of supporting semi-autonomous driving. To create a reality that consists of commercial semi-autonomous cars, scientists had to develop the computerized driver intelligence required to: (1) continuously create a virtual perception of the physical surroundings (e.g., detect pedestrians, road signs, cars, etc.), (2) make decisions, and (3) perform the corresponding action (c.g., notify the driver, turn the wheel, stop the car).
While computerized driver intelligence brought in terms of safety (I), recent incidents have shown that remi-autonomout cars suffer from the stereoblindness syndrome: they react to 2D objects if they were 3D) objects due to their inability to distinguish between these two types of objects. This fact threatens autonomous car safety, because 2D object (e.g., an image of car, dog, person) in nearby advertisement that is misdetected as a real object can trigger a reaction from a semi-aulonomous car (e.g., cause to stop in the middle of the road), as shown in Fig, Such undesired reactions may endanger drivers, passengers, and nearby pedestrians as well. As a result, there need to secure semi-antonomous cars against the perceptual challenge caused by the stereoblindness syndrome.
driving to new heights
The perceptual challenge caused by the stercoblindness syndrome stems from object detectors' (which obtain data from cars' video cameras) misclasiffiaation of 2D objects. One might argue that the stereoblindness syndrome can addressed by adopting a sensor fusion approach: by cross-coreelaing data from the video cameras with data obtained by sensors aimed at detecting depth (e.g., ultrasonic sensors, radar). However, due to safety concerns, a "safety first" policy is implemented in autonomous vehicles, which causes them to consider detected object as a real object even when detected by a single sensor without additional validation from another sensor
2 3) This is also demonstrated in Fig. I which shows how

---

Under review as a conference paper at ICLR 2023
Eleckric Pature
@laspni
Car kept jamming on the brakes thinking this was person & The NN waz dreaminn @egreentheonlly
This Billboard That Confuses Tesls Autopilot Good Reminder Of Why Seif-Driving la Still Long Way Off divr. divr.itRRxphhQ
STOP
B:57 Add Seep 26, 20000 frm Stddieyy Naw South Vikies- Teatber Androsd
262 Retweess
102 Quota Twweets
1.470 Likes
Figure 1: Two well-known incidents that demonstrate how Teslas misdetect. 2D objects as real objects.
Tesla's autopilot triggers sudden stop due to the misdetection of 2D object as real object, despite the fact that Teslas are equipped with radar, a set of ultrasonic sensors, and set of front-facing video cameras.
In addition, while various methods have used liveness detection algorithms to detect whether 4 5 0 the proposed methods do not provide the functionality required to distinguish between 2D/3D objects in an autonomous driving setup, because they are object dependent (they cannot generalize between different objects, e-g., cars and pedestrians) and do not take into account the real-time constraints associated with autonomous driving, As a result, there is a need for dedicated functionality that validates the detections of video camera based object detectors and considers the constraints of autonomous driving.
an object is 2D/3D
In this paper, we present EyeDAS, committee of models that validates objects detected by the on-board object detector. EyeDAS aims to secure a single channel object detector that obtains data from vidco camera and provides a solution to the stereoblindness syndrome, i.c., distinguishes between 2 and 3D objects, while taking the constraints of autonomous driving (both safety and real-time constraints) into account. EyeDAS can be deployed on existing advanced driver-assistance systems (ADASs) without the need for additional sensors.
EyeDAS based on few-shot learning and consists of four wight, unsupervised models, each of which utilizes unique feature extraction method and outputs 3D confidence score. Finally, a meta-classifier uses the output of the four models to determine whether the given object is 2 or 3D object.
We evaluate EyS using a dataset collected from seven YouTube video recordings of street views taken by dash cam from the driver's seat perspective; the 2D objects in the dataset were extracted from various billboards that appear in the videos. When applying FyeDAS to seven state-of-the-art ODs as countermeasure, FgeDAS was able to reduce the 2D misclassification rate from 71.42-100% to 2.4% with a 3D misclassification rate of 0% (TPR of 1.0). We also show that EyeDAS outperforms the baseline method and achieves an AUC of over 0.999.
In this research we make the following contributions: (1) we present a practical method for securing object detectors against the stereoblindness syndrome that meets the constraints of autonomous driving (safety and real-time constraints), and (2) we show that the method can be applied using few-shot learning, can be used to detect whether an inanimate object is a 2D or 3D object (i.e., distinguishes between a real car from an advertisement containing an image of car), and can generalize to different types of objects and between cities.
The remainder of this paper is structured as follows: In Section 2 we review related work. In Section 3 we present EyeDAS, explain its architecture, design considerations, and each expert in the committee of models. In Section 4 we evaluate EyeDAS's performance under the constraints of autonomous driving, based on various YouTube video recordings taken by a dash cam from several places around the world. In Section 5 we discuss the limitations of DySDAS, and in Section 6 we present. a summary.
2

---

(2) have high false negative Under review as a conference paper at ICLR 2023
2 RELATED WORK
The ability to detect whether an object a 2D or 3D object is extremely important in autonomous driving, since detection error can have life-threatening consequences, endangering the safety of the driver, passengers, pedestrians, and others on the road. Without this capability, Tesla's autopilot was unintentionally triggered, causing the car to: (1) continuously slam on the brakes in response to a print advertisement containing a picture of person that appeared on a bus (7), and (2) stop response to a billboard advertisement that contained a stop sign (8) Moreover, attackers can exploit the absence of this capability and intentionally trigger: (1) Tesla's autopilot to suddenly stop the car in the middle of road in response to a stop sign embedded in an advertisement on a digital billboard (2), and (2) Mobileye 630 to issue false notifications regarding a projected road sign
The need to detect whether an object is 2 or 3D is also important for authentication systems (e.g., face recognition systems) where the identity of a user can be spoofed using printed picture of the user. Various methods have been suggested for liveness detection (4] 5 [) however the two primary disadvantages of the proposed methods are that they: (1) fail to generalize to other objects s (e.g,, distinguish between real car and picture of car), since they mainly rely on dedicated features associated with humans (4) (e.g., eye movements (5) facial vein map ()) which makes them object dependent;
rates for pictures of objects that were not taken from a specific orientation, angle, or position (e.g., they fail to detect. liveness if the picture of the person was taken from the back). As result, these methods are not suitable for autonomous driving.
DyS
The requirements for method used to secure the perception of autonomous cars against stereoblindness syndrome are as follows. The method must be capable of: (1) operating under the constraints of autonomous driving, and (2) securing an object detector that obtains data from a single video camera, because a few commercial ADASs, including Mobileye 630 PRO, rely single video camera without any additional sensors, and (3) utilizing just small amount of training data; the fact that there may be just a small amount of 2D objects in each geographical area necessitates a method with high performance and minimum training so that it can be generalized to different types of objects and between geographical locations.
ARCHITECTURE
Fig. 2 provides an overview of EyeDAS; whenever an object detected by the vehicle's image recognition model, it is tracked during t consecutive frames sampled at frequency of frames per second (FPS), cropped from each frame and serially passed to EgeDAS. EyeDAS then predicts whether the object is 2D (e.g., an image of person) or 3D object (e.g., a real person).
Let x= [x] "..,,'" "t.,, ,x²) be time series of t identical RGB objects cropped from consecutive frames where each object is centered. To predict whether an object is 2D or 3D, we could build a supervised machine learning model which receives I, which consists of images of an object to classify, and predicts whether the object detected is 2D or 3D. However, such an approach would make the machine learning model reliant on specific features and thus would not generalize to objects extracted when the vehicle is traveling in different locations or at different speeds, or when the vehicle is approaching the object from different angles or distances.
To avoid this bias, we utilize the committee of experts approach used in machine learning applications (10), in which there is an eusemble of models, each of which has different perspective of interpreting the incoming data. By combining different perspectives, we (1) create a more resilient classifier that performs well even in cases where one aspect fails to capture the evidence, and (2) reduce the false alarm rate by focusing the classifier on just the relevant. input. features; DAS consists of an ensemble of unsupervised models (experts),
3

---

Under review as a conference paper at ICLR 2023
EyeDAS
confidence scores
Blurring Model
(iii)
Object
Sharpness Model
Meta-
Detector
Classification
2D 20/30
Color Model
'person'
Edge Model
Figure 2: EyeDAS's architecture. When an object is detected, (i) time series of the cropped object images is transferred to ByeDAS, (ii) four types unique features are processed by four unsupervised models (i.e., experts), resulting in four 3D confidence scores, and (iii) the meta-classifier model interprets the confidence scores and makes the final decision regarding the object (2D 3D).
each of which outputs 3D confidence score, and a supervised model (meta- classii-esssfeer), which produces the final outcome (decision) given the set of confidence scores.
Although each of the proposed unsupervised models (experts) focuses a different perspec- tive, they have common property: given I, which consists of images of an object classify as 2D or 3D, each model measures difference between each two conseculive elements in the series; this study, we show that the combination together with the proper distance metric (applied between each two consecutive elements) a good basis for building such a classifier. In addition, decisions based on a distance observed between two consecutive elements in given series allows EyeDAS to generalize; this approach minimizes dependency on object types or goographical locations. In addition, EyeDAS finds the optimal balances between time to decision and classification accuracy; compare EyeDAS utilizing / >> object frames to state-of-the-aa image classifier that is designed to process single frame a time.
proper feature rxcticn method
From the perspective of a sore module like EyeDAS, 3D object itself is not expected to change significantly within short period of time, even in cases in which the 3D object is a human, animal, or vehicle (i.e., a real object). Therefore, it is not trivial to find distance metrics that can detect statistically significant differences that: (1) allow accurate differentiation between 2D and 3D objects by considering just the object, and (2) can be computed quickly. Therefore, we suggest a time-efficient approach that considers objects entangled with their close surrounding background.
Each of the proposed unsupervised models utilizes a unique feature extraction method and a corresponding distance metric; these models designed to process image time series of any size (t > 1). This property is crucial, since the exact time point at which there a significant difference between two images (in cases in which the object detected is 3D) is unpredictable. In addition, the images to analyze should be represented in such a way that ensures that statistically significant difference be efficiently obtained for 3D objects, while the error rate for 2D objects is minimized. In other words, considering the objects entangled with their close surrounding background, we interested in feature raction methods whose outcomes for 2D objects and 3D objects are statistically distinguishable within a short period of time.
3.2 PROPOSED MODELS
Our committee consists of four unsupervised models, each focusing on a different perspective (see Fig. 3 for demonstration of each model's perspective); each model receives the time scries of consecutive images of an object to classify, extracts features from each image, measures type of difference between the features extracted from two consecutive images, and finally outputs a 3D confidence score. Additional details on the four models are provided below:
4

---

Under review as a conference paper at ICLR 2023
Blumiig
Sharpness
Color
Edge
Original (3D)
distance (raw) = 0,771
distance (raw) 0.583
distance (raw)
Features
distance 0.643
distance 0.165
distance 0.120
distance 0.100
Original (20)
distanue (tew) 0,723
distunce (raw) 0.510
distance (raw] 0.503
distance (raw) 0.520
Features
distance 0.148
distance 0.063
distance 0.001
distance
Figure 3: Examples of EyeDAS models' views on 2D and 3D objects and the corresponding distances (both raw and EyeDAS's distances are presented) calculated 3D confidence scores.
Blurring Model (B) This model utilizes the automatic focus (auto-focus) capability of video cameras commonly used to build 3D cameras (TI) Unlike 2D object images, for 3D object images, the blurring effect differs and is applied alternately
the object and its surrounding background during the auto-focus process; this is reflected in a large amount of contrast, which is observed when examining the blurring maps (12) corresponding raw images. Thus, using the structural image similarity measure proposed by Wang et al. [13,, the blurring model outputs the maximum value obtained by calculating the differences between each two consecutive blurring maps.
the
Sharpness Model (S) This model utilizes the possible image sharpness-level instability observed in 3D object images due to the objects' movements; the sharpness model extracts the overall sharpness level (a numeric value) from each raw image received, using the method described by Bansal et al. (14, and outputs the maximun value obtained by calculating the differences between each two consecutive sharpness levels.
Color Model (C) This model utilizes the possible movement expected in the surrounding environment of 3D objects; this movement is reflected in a large difference in the color distributions of the raw images. Thus, the color model extracts the color distributions from cach raw image using a clustering algorithm (15), computes the size of the largest cluster observed, and then outputs the maxinum value obtained by calculating differences between each two consecutive elements.
Edge Model (E) This model is based the Sobel edge detector 16), a gradient-based method for estimating the first order derivatives of the image separately for the horizontal and vertical axes; these derivatives are not expected change significantly for static objects like 2D objects. The edge model operates similarly to the blurring model except for one thing: given the raw images, the odge model extracts the edging maps instcad of the blurring maps for comparison; extraction is performed using the method described by Gao el al. (17).
To make a prediction as to whether or not object is 2D 3D object, we combine the knowledge of the unsupervised models described above in a final prediction; utilize gradient boosting (GB) based binary classifier (18) trained on the outputs the models. We choose the GB algorithm, since it can capture nonlinear function dependencies; in case, the above unsupervised models complement cach other, and their output scores form a nonlincar relationship.
Meta-Classifier-
The reader (the link will be added after the review, the project zip file has been already shared during paper submission).
the following link to download the code that implements the method
5

---

Under review as a conference paper at ICLR 2023
4 EVALUATION
The experiments described in this section were designed in recognition of the needs of autonomous driving in the real world; to ensure safety, solution needs to both accurately distinguish between 2D and 3D objects and make a fast dccision.
The experiments are aimed at evaluating: (1) the performance of each expert in the committee, (2) the performance of the entire commitlee, (3) improvement in the false positive rale of ODs when EyeDAS is applied for validation, (4) the practicality of EyeDAS in real-time environments (in terms of computations resources and speed), and (5) the ability to generalize between objects (i.e., humans, animals and vehicles) and geographical locations (i.e., cities).
All the experiments described in this section, including the speed and memory benchmark (described in Section 1.4,, were conducted on an Intel 2.9 GHz Intel Core i7-10700 and 32GB RAM. The machine's operating system was Windows 10.
4.1 EXPERIMENT SETUP
Datasct. Since some ADASs (e.g. Mobileye 630 PRO) rely on single channel camera, all of the data collected for our experiments was obtained from single-channel cameras; we utilized seven YouTube video recordings D of street views taken by a dash cam from the driver's seat perspective. The distribution of the dataset represents the real distribution of 2D and 3D objects encountered by aulonomous driving in the real world: 2,000 RGB objects (i.e., humans, animals, and vehicles) were extracted from driving view videos taken in seven cities: New York (NY, USA), San Francisco (CA, USA), Dubai ( (United Arab Emirates), Miami (FL, USA); London (UK), Los Angeles (CA, USA), and George Town (Singapore). of which approximately 95% are 3D objects and 5% are 2D objects extracted from billboards.
The objects were extracted and cropped using the highest performing OD described by Redmon et al. (19); in our experiments, each input instance æ associated with an object to classify contains up to five images laken 200 millisecond intervals, starting at the time point at which an object was delected by the OD. Input instance 2; is labeled as 'Truef if the instance represents 3D object and 'False' the instance represents 2D object.
Training. We denote T TRaD and TRRD as two training sets representing 3D and 2D objects respectively. To avoid an unbalanced training set, we extend TRop by utilizing known image data augmentation techniques (20) to randomly selected instances from T TR2D; we apply the rotation technique.
Given the outputs calculated by the unsupervised models for each imput instance (i.e., the 3D confidence scores), the final meta-classifier training was performed; the best hyperparameters for training were selected using the grid search algorithm (21) using the random shuffling split method and 10-fold cross-validation. We vary the number of estimators in the set of {20, .5, 40) trees, while we change the maximum depth of the trees in the set of {2, 3, 4). To select the best set of hyperparameters, we evaluated the meta-classifier's performance in terms of accuracy.
In all of the experiments described in this section: (1) [R = 150 and [R = 70, and (2) a decision was made within 200 milliseconds from the moment the object was detected by the OD (i.e., 2).
4.2 RESULTS
Performance. In Fig. 5 we present the receiver operating characteristic (ROC) plot and the area under the ROC (AUC) for different combinations of the blurring (B), sharpness (S), color (C), and edge (E) models; the combination of all four is our proposed method. The ROC plot shows the true positive rate (TPR) and false positive rate (FPR) for every possible prediction threshold, and the AUC provides an overall performance measure of classifier (AUC-0.) -0.5: random guessing, AUC-1: perfect performance).
New York City (USA, NY) San Francisco (USA, CA) Dubai (UAE) Miami (USA, FL) London
(UK)
Los Angeles (USA, CA)
George Town (Singapore)

---

Under review as a conference paper at ICLR 2023
In case, there is critical trade-off that must be considered: the classification threshold. lower threshold will decrease the FPR but often decrease the TPR as well. In Table [ provide the TPR and FPR of the models when the threshold is set at 0.5 and for the threshold value at which the 1. As can be seen, the use of all of the proposed models (B+S+C - E) in combination perforrm all other model combinations.
20 Mildcoossototiin Wote
1.0
Sye3M5 Courrmeuuee
Without
With
True Positive Rate
Object Deterto
TrreshoM60
TTMssokks//MM-11
yolors, @ [19]
100%
0.0%
2.4%
BeReCaf ALC 0.9004604
yolovd Sarinetta, 508 [19
78.57%
0.0%
2.4%
vilvo
L9)
0.4
BeSaC AUC n.9006Gaii
ReCoE AUC D 0.999963
Sater (os3e550223
AAC 240021166
faster 101
97.61%
0.0%
2.4%
0.2
79107218
sod, mobilenet_ee .2 126
78.57%
0.0%
2,4%
AUC 0.18117555
sod hwc.icccoioonoooo 88.099
0.0%
2.4%
0.0
0.2
0.4
0.6
0.8
1.0
False Positive Rate
Figure 4: 2D misclassification rates using stato-of-the-art object detectors.
Figure 5: The values of the ROC curve and AUC metrics for different model com- binations (B: blurring model, S: sharpness model, C: color model, E: edge model).
In Table 2 compare EyeDAS's performance to that of two other approaches: (1) baseline models based on the state-of-the-art pre-trained image classifies (i..., VGG16, VGG19 and Resnet50 (22);: ); utilize the known transfer learning technique (23) by re-training these models, and (2) an optimized model similar to EyeDAS, except that it is based on single expert model which considers the raw images as is (i.e., it computes the image similarity based distance (13) between the raw images directly, without extracting any features as EyeDAS does). In both cases, 220 instances were randomly selected for training; the distribution of 3D and 2D images is approximately 66.7% and 33.3% respectively, and the data augmenta ation technique described above was applied to avoid an unbalanced training set..
Each baselinc model was re-trained by (1) freczing all the layers except for the output layer which was replaced by two trainable fully connected layers, (2) randomly picking 50 instances from the training set to serve as the validation set, (3) pre-processing the input data (i.e., image resizing and scaling), and finally (4) minimizing the cotegorical crossentropy loss function on the validation set using the Adam optimizer. The first new layer contained 128 neurons and attached with the relu activation function, and the second layer (output layer) contained two neurons and attached with the softmax activation function. Increasing the first. layer's neurons count beyond 128 resulted in poorer results due to overfitting. As can be seen, EyeDAS outperforms all of the abovementioned models.
Table 1:
EyeDAS's TPR and FPR with different thresholds.
EyeDAS BiSaCaF
B+C
S+C
8-5eC
B:S1F
SiC4E
B+C+E
Threshold @0.5
TPR
0.950
0.803
0.963
0.950
0.704
0.959
FPR
0.000
0.095
0.000
0.100
0.000
0.000
0.095
0.187
0.033
0.000
0.000
0.100
0.000
0.000
Threshald
TPR
1.000
1.000
1.000
TPR-1]
FPR
1.000
1.000
1.000
1.000
0.586
0.355
0.478
1.000
1.000
1.000
0.119
1.000
1.000
0.500
0.024
TPR: Ratio of 3D objects detected; FPR: Ratio of 2D objects misclassified as 3D objects.
Table 2:
Comparison to other approaches.
VGG16
V6519
Thresho 20.5
TPR
0.855
0.893
0.766
0.741
0.968
FPR
0.071
0.143
0.286
0.024
Thresho d
1.000
1.000
@[TPR-
FPR
0.581
0.905
1,000
1.C00
0.024
7

---

Under review as a conference paper at ICLR 2023
Securing ODs with EyeDAS. To determine how effective EyeDAS is as part of system,
we evaluated 2D misclassification rates on seven state-of-the-art ODs (19 24 25 26). The results are presented in Table [ present the 2D misclassification rates obtained for each detector before and after applying EyeDAS as a counterneasure and the impact of the different thresholds. The results show that for most ODs, when the detector mistakenly classified a 2D object as real (i.e., 3D), ByeDAS provided effective mitigation, even for the threshold value which the TPR-1.
Table 3:
Disagroements between the models.
Disagreement
5.C
B.S.C.
Threshold
20
0.3%
0.0%
1.0%
0.0%
0.5%
1.1%
0.3%
1.3%
1.0%
0.9%
1.3%
@0.5
3D
31.5%
41.7%
20.7%
37.2%
33.5%
26.6%
84.1%
50.7%
65.0%
17.8%
72.8%
Threshold
2D
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.3%
0.0%
0.7%
0.0%
1.3%
@[TPR=1]
38.2%
87.8%
24.3%
43.1%
44.2%
42.7%
67.5%
55.4%
58.7%
67.4%
75.9%
Ablation study. In order for the committee of experts approach to be effective, there must be some disagreements belween the models; although there is a clear indication that each model (i.e., blurring, sharpness, color, and edge) contributes unique perspective on the final prediction, we perform an ablation study to provide further evidence that each model has a unique contribution. For that, utilize the SHAP (SHapley Additive exP sllanatioons framework to explain the model's predictions; each input feature is assigned a score (i.c., a Shapley value) which represents its contribution to the model's outcome. In casc, interested in the contributions of each expert (i.e., each input feature to our meta-classifier to the final prediction. In Table 3 we present the measure of disagreement between each possible sub-committee of experts: given an input instance we say that a sub-committee disagrees on x if there at least one expert whose Shapley value is negative and least one expert whose Shapley value is positive, Interestingly, the full combination of experts (B+S+C-F) has the highest number of disagreements and results in the lowest FPR at TPR-1.
Table 4: Evaluation results for different combinations of locations (NY: New York, GT: George Town, MI: Miami, SF: San Francisco, LA: Los Angeles, DU: Dubai, LO: London).
Trained in...
Tested in...
FPR @0.5
TPR
FPR Threshold @[TPR=1]
TPR @[TPR=1]
@D..5
0.000
0,000
1.000
0.000
0.953
0.000
1.000
CT
SF
0.000
0.962
0.000
1.000
DU
0.000
0.979
0,000
1.000
NY
0.000
0:962
0.154
1.000
LA
0.000
0.958
0.000
1.000
DU
0.000
0.973
0.000
1.000
LO
0.000
0.990
0.000
1.000
4.3 GENERALIZATION
We also evaluate how EyeDAS's approach generalizes to different geographical locations and even different types of objects.
Generalization to other geographical locations. To evaluate EyeDAS's geographical location generalization ability, we trained and tested the models on complementary groups of location types (i.e., cities). For training, took minimum-sized city type combinations in which there are at least 56 2D objects and at least 120 3D objects, since we observed that less than that amount is insufficient. for training the models and thus meaningful
8

---

Under review as a conference paper at ICLR 2023
present the evaluation results obtained for different geographical location (i.e.; cities) combinations. As can be seen, EyeDAS not dependent on the geographical location in which the training data was collected, as it is capable of using the knowledge gained during training to distinguish between and 3D objects.
conclusions could be derived. In Table 4
Generalization to other types of objects. To evaluate EyeDAS's 's object type generaliza- tion ability, we trained and tested the models on complementary groups of object types. We locused on humans (HU), animals (AN), and vehicles (VE). As previously done, for training we used minimum-sizzed object type combinations in which there are at least 56 2D objects and at least 120 3D objects. In Table [ we present the evaluation results obtained for each object type combination. As can be seen, EyeDAS is independent of the type of the object appeared in the training set, as it is capable of using the knowledge gained during training to distinguish between 2 and 3D objects.
Table 5: Evaluation results for different combinations of object types.
Trainedon...
lested on...
FR
TPR
FPR
TPR
Threhaid gobs
Thrwchuld
Thmeduii e[(%=1]
Thrnhhld e[tee]]]
Animmals
0.018
Vehidles Aniimals
Humans Vehicles
Animals
0.100
1.000
0.100
1.000
4.4 SPEED AND MEMORY PERFORMANCE
We performed a speed benchmark experiment to assess EyeDAS 's performance. We found that EyeDAS took average of 19 ms to process single object frame,, with a standard deviation of ms. This translates to an approximately 220 ms time to decision from the time the object is detected. We also note that the meta-classffier is relatively small (45 kilo byles) and only utilizes less than 0.01% of the CPU per object frame.
LIMITATIONS
Despite the high performance achieved, EyeDAS has some limitations. First, technical factors may infiuence the performance of EyeDAS, For example, EyeDAS may perform poorly when image resolution is low or images are taken in low lighting. However, adequate lighting conditions are expected roads on which autonomous vehicles can typically drive. Second, for 3D objects, both the detected object and its close surrounding background are stationary (e.g., the object does not move its surrounding background does not change), then ByeDAS may perform poorly. However, the vehicle is moving or the camera's auto-focus process is operating, then EyeDAS 's errors will likely to decrcase significantly even for 3D stationary objects. If the vehicle is not moving, the concern for the safety of passengers does not exist.
6 SUMMARY
In this paper, we proposed a novel countermeasure which can be used to secure object detectors (ODs) against the stercoblindness syndrome; this syndrome can have life- tffe-threatening consequences, endangering the safety of an autonomous car's driver, passengers, redestrians, and others on the road.
Designed in recognition of the needs of autonomous driving in the real world, the proposed method based on few-shot learning, making it very practical in terms of collecting the required training set. As presented in the previous sections, EyeDAS outperforms the baseline method and demonstrates excellent performance and specifically its: (1) ability to maintain misclassification rate for 3D objects, (2) ability to improve the performance of ODs, (3) practicality in real-time conditions, and (4) ability to generalize between objects (i.e., humans, animals, and vehicles) and geographical locations (i.e., cities).
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
[ "Tesla updates autopilot safety numbers; almost 9x
safer
than
average driving." [Online]. Available:
https: /electrek.co/ 2019 10/0/////)
asll--utoopilot-sae
9x- safer- than- than-averaae--driinng
[2] B. Nassi, Y. Mirsky, D. Nassi, R. Ben-Netanel, O. Drokin, and Y. Elovici, "Phantom of the adas: Securing advanced driver-assistance systems from split-second phantom attacks," " in Procecdings of the 2020 ACM SIGSAC Conference on Computer and Communications Security, 2020, pp. 293 308.
[3] C. Yan, W. Xu, and J. Liu, "Can you trust autonomous vehicles: Contactless attacks against sensors of self-driving vehicle," DEF CON, vol. 24, 2016.
[4] G. Pan, Z. Wu, and L. Sun, "Liveness detection for face recognition," Recent advances in face recognition, pp. 109-124, 2008.
[5] O. V. Komogortsev, A. Karpov, and C. D. Holland, "Attack of mechanical replicas: Liveness delection with eye movements," IEEE Transactions on Information Porensics and Securily, vol. 10, no. 4, pp. 716 725, 2015.
[6] S.-Y. Wang, S.-H Yang, Y.P. Chen, and J.-W. Huang, "Face liveness detection based on skin blood flow analysis," " symmetry, vol. 9, no. 12, p. 305, 2017.
[7] "Tesla autopilot amusingly confuses ben fordhams bus poster ad real pedestrian." OOlline]. Available: ben-es--bbn--ordhaassss hatpllaans-------ssttt---d-----s posterr-adlass--cal-pppottttiiii
https: http://wwwwwwttooooo/o//ooo
tesla
[8] "This billboard that confuses tesla antopilot. is a good reminder of why self-driving is still a long way off." [Online]. Available: lhtps: jatp://jooonni..oom. bhbb-billbard-tthe tha.rarr-thtt--ofusss-----sss-se tetesssessllalallliiee autopilot- aasss---goo g-good- 1846698527
(9] B. Nassi, D. Nassi, R. Ben-Netanel, and Y. Flovici, "Spoofing mobileye 630's video camera using projector," " in Workshop on Automotive and Autonomous Vehicle Security (AutoSec), vol. 2021, 2021, p. 25.
[10] Y. H. Hu and J.-N. Hwang, "Handbook of neural network signal processing." " 2002.
[11] K. C. Zheng, A. Colburn, A. Agarwala, M. Agrawala, D. Salesin, B. Curless, and M. Cohen, "Parallax photography: creating 3d cinematic effects from stills," in Proceedings of Graphics Interface 2009, 2009, pp. 111-118.
[12] S. A. Golestanch and L. J. Karam, "Spatially-varyine blur detection based on multiscale fused and sorted transform coefficients of gradient magnitudes." in CVPR, 2017, pp. 596 605.
[13] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, "Image quality assossment: from error visibility structural similarity," IEEE transactions on image processing, vol. 13, no. 4, pp. 600 612, 2004.
[14] R. Bansal, G. Raj, and T. Choudhury, "Blur image detection using laplacian operator and open-cv," " in 2016 International Conference System Modeling & Aduancement in Research Trends (SMART). IEFE, 2016, pp. 63-67.
[15] A. Likas, N. Vlassis, and J. J. Verbeek, "The global k-means clustering algorithm," " Pattern recognition, vol. 36, no. 2, pp. 351-461, 2003.
[16] N. Kanopoulos, N. Vasanthavada, and R. L. Baker, "Desigu of an image edge detection filter using the sobel operator," " TFF.F. Journal of solid-state circuils, vol. 23, no. 2, pp. 358-367, 1988.
[17] W. Gao, X. Zhang, L. Yang, and H. Liu, "An improved sobel edge detection," " in 2010 3rd International conference on computer science and information technology, vol. TEEF, 2010, pp. 67-71
10

---

Under review as a conference paper at ICLR 2023
[18] G. Ke, Q. Meng. T. Finley, T. Wang, W. Chen, W. Ma, Q Ye, and T.-Y. Liu, "Lightgbm: A highly efficient gradient boosting decision tree," " Advances in neural information processing systems, vol. 30, 2017.
[19] J. Redmon and A. Farhadi, "Yolov3: An incremental improvement," 2018.
[20] C. Shorten and T. M. Khoshgoftaar, "A survey on image data augmentation for deep lcarning," Journal of big data, vol. 6, no. 1, pp. 1-48, 2019.
[21] J. Bergstra, D. Yamins, and D. Cox, "Making science of model search: Hyperparame- ter optimization hundreds of dimensions for vision architectures, in International conference machine learning. PMLR, 2013, pp. 115-123.
[22] S. Mascarenhas and M. Agarwal, "A comparison between vgg16, vgg19 and resnet50 architecture frameworks for image classification," in 2021 International Conference on Disruptive Technologies for Multi- DistipDDicpp Research and Applications (CENTCON), vol. 1. IEEE, 2021, pp. 96-99.
[23] S. Tammina, "Transfer learning using "6g-16 with deep convolutional neural network for classifying images," " International Journal of Scientsfie and Research Publications (IJSRP), vol. 9, no. 10, pp. 143-150, 2019.
[24] S. Ren, K. He, R. Girshick, and J. Sun, "Faster r-cmn: Towards real-time object detection with region proposal worksr,,, IEFE Transactions on Pallern Analysis and Machine Intelligence, Jun 2017.
[25] A. G. Howard, M. Zhu, Chen, D. Kalenichenko, W. Wang. T. Weyand, M. Andreetto, and H. Adam, "Mobilenets: Efficient convolutional neural networks for mobile vision applications," arXiv preprint urXiv:1704.04811 2017.
[26] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. Berg, "Ssd: Single shot multibox detector," " ECCV, 2016.
11


Reviews:
[{"review_id": "eMLY8koyx5", "reviewer": "Reviewer_d1dP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles an important problem facing autonomous vehicle perception - how to distinguish between a 3d object and a 2d representation of a 3d object?  A useful insight of this paper is that having information of the target object over time leads to important features that help solve this problem.  The paper proposes a model based on 4 heuristic hand-crafted features together with a gradient boosting meta classifier.  ", "summary_of_the_review": "This is paper addresses an important problem and demonstrates on a dataset that the method works with accuracy that may be high enough to solve this problem in a practical setting.  The paper makes the insight that using sequences of images over time can help solve this problem.  Unfortunately it does not use a learned representation to achieve this but rather uses hand-crafted features, so is not directly relevant to ICLR.  The baselines that it uses to compare against did not have the advantage of the temporal sequence of images and therefore are not apples-to-apples comparisons. ", "strength_and_weaknesses": "Strengths:\n* Paper deals with an important problem\n* This paper presents a useful insight that considering the appearance of the target object over time can provide a strong cue for this problem\n* The paper shows that the technique performs well on the dataset they used.\n\nWeaknesses:\n* This paper uses hand-crafted rather than learned features and does not present a convincing case that there is a good reason to do so. \n* The baseline methods are straw-men that do not benefit from temporal information as the proposed method does.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IJFPgnHb9lu", "reviewer": "Reviewer_KB85", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work prorposes a few-shot learning-based method named EyeDAS for securing object detectors against the stereoblindness syndrome (i.e., the inability to distinguish between 2D and 3D objects). It leverages the low-level features of image to solve the problem. Four unsupervised models (for blurring, edge, color and sharpness) repectively predict 3D confidence scores and a meta-classifier interprets the confidence scores and makes the final decision. EyeDAS is evaluated on a dataset collected from seven YouTube video recordings.", "summary_of_the_review": "The evaluation is based on a self-collected  dataset with too few data samples, which makes the effectiveness of the proposed method less convincing.  And the few-shot method  is with less practical value, because annotated data are easily collected and supervised methods work well.\n\n", "strength_and_weaknesses": "Strength:\n--The proposed problem of stereoblindness syndrome. i.e., the inability to distinguish between 2D and 3D objects, is interesting and meaningful.  It’s a practical problem we should solve to improve the robustness of autonomous vehicles.\n- The paper is well written and organized. And it’s easy to follow.\n\n\nWeakness:\n- The major problem is the evaluation. Firstly, the whole experiment dataset is only collected from seven YouTube video recordings with hundreds of  annotated objects for training and testing. This experiment benchmark are not convincing enough because of the small data quantity.\n- And the baseline result used for comparison is poor and not reasonable.  The baseline model does not converge well with such few data. If trained with enough data, I think the baseline model can achieve comparable performance.\n- The proposed method is a few-shot method. But in the driving scenerios,  2D / 3D objects are common. It’s easy to get more annotated data for training the model. Few-shot methods are not practical for the problem of stereoblindness syndrome.\n- The proposed method is not lightweight enough. 200 ms latency is not enough for real-time applications.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HbZx86U6ql", "reviewer": "Reviewer_DucZ", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is interested in discriminating between a 2d view of a 3D object  and an actual  3D object, in the  context of autonomous  driving.\nThe goal is  to get rid of false detection of objects that are actually pictures of objects,  thereby increasing navigation safety.\n\nAlthough the author does not express it this way,  the  problem is essentially to  ascertain if the  scene is planar or not in a monocular sequence of two images.\nAlthough the problem is interesting, it  is  cast  in a very vague,  non geometrical way,  which  results  is  an ad hoc algorithm without justification,  especially with  regards to geometry.\n", "summary_of_the_review": "This paper has failed to properly identify the exact 3d  computer vision problem  it  tries to  solve, which  is  scene planarity under motion  stereo. Because of this, the proposed model is unjustified, and there  is no  basis to believe  that  it actually works, or would would work well in practice.\n\nThe problem is interesting and  important, and I hope the  authors  will consider it from the angle of 3d computer vision, as a planarity estimation problem. It would be ideal, in my  opinion, to explore the  problem in a synthetic framework first (simulating a camera motion in a scene with objects that  are planar and non  planar), before jumping to real images.\n\n\n", "strength_and_weaknesses": "The main weakness  of  the  paper is that the problem is not cast as a 3D computer vision problem,  but a vague 2D recognition  problem.\nWhile  it  is clear that the  problem is related  to  estimating scene planarity from a the  motion of  a monocular camera, the  problem is cast as a purely  2D task.\n\nThe proposed network  architecture, relying on four simple  models (Burring/Sharpness/Color/Edge) is not well justified and is  not related to the fundamentals of  the problem. This makes it  impossible  to  see how or why  it  would  work.\n\nThe choice of  5  images over a  1 second time lapse is  not  properly  justified,  especially when the  proposed  algorithm uses only 2 images to make a decision. The time interval  should be chosen in accordance to  the motion of the  camera,  to ensure that enough parallax is present.\n\nComparing this architecture, which uses 2 images, with other  architectures that use only one image (and are trained on a different problem) seems somewhat unfair. But maybe I misunderstood that result (Figure 4).\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9mIRHvK0ZT7", "reviewer": "Reviewer_NQv8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the author(s) proposed EyeDAS, a few-shot learning-based method to avoid stereoblindness syndrome for object detection (ie. wrongly detect printed objects on billboard/screen as they were in 3D). The proposed method works as a post-processing step. Once any object detection model detects an object with t (t>1) frames, it selects 4 non-learning-based distance metrics (blurring, sharpness, color and edge) to compute a ``3D confidence score''. It uses a gradient-boosted tree to classify if the object is a 3D object. Experimentally, the authors show that the proposed method can achieve close to perfect classification performance on 7 videos (all recorded by a dash cam from the driver’s seat perspective) collected on YouTube, outperforming deep-learning-based transfer learning methods.", "summary_of_the_review": "Please refer to the Weaknesses section. I am mostly concerned about the dataset details and the minimal contribution.", "strength_and_weaknesses": "Strengths:\n- The paper is well-written and easy to understand.\n- The method is simple, fast, and is shown with good empirical performance.\n- The authors also conduct a number of ablation studies to show the generalizability and the effectiveness of each component of the proposed method.\n\nWeaknesses:\n- Missing details about the dataset and the labeling process: the experiments use a custom dataset collected on YouTube and I think it would be better to provide more details about it. Specifically, \n  - Each of the 7 videos is actually quite long (~an hour), but the resulting train/test datasets only have 2000 RGB objects (line 206). What is the process of extracting the objects? Would about 100 (2000 * 5%) non-3D objects be sufficient for evaluation?\n  - What is the process of labeling true 3D objects and false 3D objects?\n- About the setting: This paper actually proposes a new application problem: few-shot planar/3D object classification. But I am not entirely convinced by its practical soundness in that:\n  - I am not sure the few-shot is necessary for such a setting since planar/3D object classification can be object agnostic and can potentially have large enough data for training. Gradient boosted method can work quite well on few data with good feature engineering, but I am not sure it can be better than transfer learning methods with enough data.\n- It would be good to also show some qualitative examples of the classification results.\n- I am concerned about the minimal contribution: this paper is about a specific and new setting, and the proposed method is simple (gradient boosted tree with good feature engineering) and not new. Though I think it is good to have a simple method that works well, I think it should also show new ideas/insights/understandings to the machine learning community. I do not see enough such contributions from this paper and it looks more like a good technical report.\n\nMinor:\n- Styling of the tables: all tables in the paper are from screenshots and I would suggest the authors follow the ICLR style guide (https://github.com/ICLR/Master-Template/raw/master/iclr2023.zip) to present the tables.\n- About the evaluation metric: I got confused initially by the 2D misclassification rate and TPR in Table 1: if I understand correctly, TPR is specifically referred to as the true positive rate of 3D objects, and 2D misclassification rate is computed as (FP / (FP + TN) = FPR). I suggest the authors clarify the metrics in the experiment section, and it feels more natural to me to use more conventional names (https://en.wikipedia.org/wiki/Sensitivity_and_specificity)\n- Some of the columns/rows in the tables are redundant, e.g., in table 2 and table 3 the TPR row under Threshold @ [TPR=1], in table 5 and table 6 the TPR Threshold @ [TPR=1] column.\n- The right image of Figure 1 might not be the right example: the stop sign is also a planar object. Can it be detected by the proposed method?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject