Identifying Perturbed Roadway Signs: Perception of AI Capabilities

Artificial Intelligence (AI) computer vision is necessary for multiple safety and assistance functions for driving automation systems (DAS). Even though AI can execute complicated tasks, it is vulnerable to adversarial inputs, such as minimally perturbed noise added onto images that can be identified by human eyes (Qiu et al., 2019; Goodfellow et al., 2015). Humans can identify the contents of an image that has been maliciously altered through perturbed noise, but AI computer vision cannot (Goodfellow et al., 2015). In SAE Level 2 and 3 vehicles, humans are still required in the driver’s seat to be ready to take over whenever the vehicle’s system falters (SAE, 2021). To best prepare the driver, they should know when the DAS will likely fail by understanding its capabilities and limitations (Mishler et al., 2021). This study aimed to examine humans’ perception of AI’s capabilities of identifying 11 unique road sign images that were maliciously manipulated through two different adversarial attacks, the projected gradient descent (PGD; Deng & Karam, 2020) and the physical attack (Eykholt et al., 2018), in addition to a control (pixel-scrambled) image and a baseline (original) image. For both the PGD and physical-attack images, the human should be able to correctly identify the sign, but the AI would misclassify both. The scrambled manipulation should make it difficult for both the AI and the human to correctly classify the image, and both the human and AI should correctly identify the original image with no attack. A total of 230 participants were recruited through Amazon Mechanical Turk (MTurk) to participate in this study. Participants completed a pre-task questionnaire measuring trust (Jian et al., 2000) and self-confidence, an experimental task, and a post-task questionnaire measuring safety criticality of the 11 road signs and nine demographic related questions. The experimental task used a within-subjects design of attack type (no attack/baseline, PGD attack, physical attack, scrambled manipulation) and agent type (human vs. AI). Each experimental question contained an image in one of the four conditions, and the participant was required to rate their perception of how well the agent could classify the image correctly, on a scale of 1 to 7. The results showed that participants correctly rated their own human ability to be more capable at correctly identifying the road-sign images than the AI agent for the baseline, PGD attack, and physical-attack image. Participants also rated the AI’s ability similarly between the baseline, PGD attack and physical-attack images, which was unexpected. This may be because participants were unaware of the limitations of the AI for the malicious attacks. If they were educated about the AI’s limitations, then they would have rated the AI’s capability lower for the physical-attack image than the PGD attack since it is more visible to the human eye. Our results imply that participants understood that AI is vulnerable to the scrambled images because they rated both the human and AI to be unable to identify these signs. There was a positive correlation between trust and AI capabilities, those who trusted AI more believed it was more capable at identifying the signs. Together, these results indicate overtrust (Parasuraman & Riley, 1997) in the AI when encountering malicious attacks and a lack of computer vision knowledge, which can lead to potentially dangerous situations (Halim et al., 2016).