# Towards Explainable Pedestrian Behavior Prediction:
## 1 Introduction and Related Work
A dataset like Targeted Action Priors Network (TITAN) [20] concentrates on a range of pedestrian actions, including motion, communicative, and contextual actions but does not explicitly incorporate the crossing action in their dataset approach. Conversely, datasets such as Stanford-TRI Intent Prediction (STIP) [21], JAAD [22], and PIE [23] explicitly include crossing actions within their dataset approach. The crossing action labels are crucial to this study, as they enable the identification of pedestrian behavior in specific road situations. These labels are used as the ground truth in this work. We discarded the use of STIP for this research, as the dataset could not be shared by its owners at the time of the study. Therefore we have chosen to focus on the two widely used and publicly available datasets within the research community: JAAD is publicly available at [[https://data.nvision2.eecs.yorku.ca/JAAD_dataset/](https://data.nvision2.eecs.yorku.ca/JAAD_dataset/)]([https://data.nvision2.eecs.yorku.ca/JAAD_dataset/](https://data.nvision2.eecs.yorku.ca/JAAD_dataset/)) (accessed on 10 May 2024) and PIE is publicly available at: [[https://data.nvision2.eecs.yorku.ca/PIE_dataset/](https://data.nvision2.eecs.yorku.ca/PIE_dataset/)]([https://data.nvision2.eecs.yorku.ca/PIE_dataset/](https://data.nvision2.eecs.yorku.ca/PIE_dataset/)) (accessed on 10 May 2024). * **JAAD Dataset:** The JAAD dataset is a richly annotated collection consisting of 348 short video clips. It features a diverse range of road users across various driving environments, including different locations, traffic scenarios, and weather conditions. The dataset's annotations are categorized into spatial, behavioral, contextual, and pedestrian-specific information. Approximately 72% of the pedestrians in the dataset actively engage in street crossing, while the remaining 28% do not. Figure 1: Explainability and perspectives under AV context. * **PIE Dataset:** The creators of JAAD released a new dataset called PIE, which contains over 300K labeled video frames recorded in Toronto in clear weather. In addition to the similar type of annotations found in JAAD, PIE stands out by incorporating ego-vehicle information derived from On-board diagnostics (OBD) sensors. Distinguished from the JAAD dataset which primarily focuses on pedestrians intending to cross, the PIE dataset provides annotations for all pedestrians in close proximity to the road, irrespective of whether they attempt to cross in front of the ego vehicle or not. Furthermore, in contrast to the annotated pedestrian action in the JAAD dataset, the PIE dataset also exhibits an imbalance where a larger proportion of pedestrians are observed not crossing the street. Specifically, approximately 39.2% of pedestrians actively cross the street, whereas the remaining 60.8% do not engage in street crossing. Building on the provided context and emphasizing the importance of both explainability and dataset selection into the autonomous driving context, this work proposes a novel neuro-symbolic approach that integrates symbolic reasoning with neural perception to enhance transparency and interpretability. We aim to develop an explainable pedestrian behavior predictor and evaluate the data selection process through a series of experiments conducted on this predictor. The proposed method employed multiple explainable features extracted from the JAAD and PIE datasets which were used for mining fuzzy rules. Subsequently, these rules were used to define a fuzzy inference system that provides the pedestrian behavior prediction. This approach is particularly novel, as it not only focuses on accurate prediction but also establishes a baseline for explainability. The rest of the paper proceeds as follows: Details about the proposed pipeline for the Neuro-symbolic approach are introduced in Section 2. Section 3 describes the implementation, and the experimental setup is presented in Section 4. The results and discussions are presented in Section 5.