Uncertainty and Traffic Light Aware Pedestrian Crossing Intention Prediction
Authors: Minali Upreti
Jayanth Ramesh
Chandan R Kumar
Bodhisattwa Chakraborty
VIKRAM BALISAVIRA
Phillip Czech
Vitali Kaiser
Markus Roth
Venue: Submitted to ICLR 2023 (2023)
Keywords: deep learning
computer vision
recurrent neural networks
uncertainty estimation
intention prediction
attention mechanism
autonomous driving

Abstract: Predicting Vulnerable Road User (VRU) crossing intention is one of the major challenges in automated driving. Crossing intention prediction systems trained only on pedestrian features underperform in situations that are most obvious to humans, as the latter take additional context features into consideration. Moreover, such systems tend to be over-confident for out-of-distribution samples, therefore making them less reliable to be used by downstream tasks like sensor fusion and trajectory planning for automated vehicles. In this work, we demonstrate that the results of crossing intention prediction systems can be improved by incorporating traffic light status as an additional input. Further, we make the model robust and interpretable by estimating uncertainty. Experiments on the PIE dataset show that the F1-score improved from 0.77 to 0.82 and above for three different baseline systems when considering traffic-light context. By adding uncertainty, we show increased uncertainty values for out-of-distribution samples, therefore leading to interpretable and reliable predictions of crossing intention.

Under review as a conference paper at ICLR 2023
UNCERTAINTY AND TRAFFIC LIGHT AWARE PEDES- TRIAN CROSSING INTENTION PREDICTION
Anonymous authors Paper under double-blind review
ABSTRACT
Predicting Vulnerable Road User (VRU) crossing intention is one of the major challenges in automated driving. Crossing intention prediction systems trained only on pedestrian features underperform in situations that are most obvious to hu- mans, as the latter take additional context features into consideration. Moreover, such systems tend to be over-confident for -fitt---iittiittoonnn samples, therefore making them less reliable be used by downstream tasks like sensor fusion and trajectory planning for automated vehicles. In this work, we demonstrate that the results of crossing intention prediction systems can be improved by incorporating traffic light status as an additional input. Further, we make the model robust and interpretable by estimating uncertainty. Experiments on the PIE dataset show that the Fi-score improved from 0.77 to 0.82 and above for three different baseline systems when considering traffic-light context. By adding uncertainty, we show increased uncertainty values for -istt--iistiiuutinn samples, therefore leading interpretable and reliable predictions of crossing intention.
INTRODUCTION
VRUs are complex participants for an Automated Vehicle (AV) to perceive. The AV should not only be able to detect VRUs, but also understand their underlying intentions and predict their future actions. In addition, several surrounding factors, including incoming traffic and traffic light status, influence VRU behavior. A pedestrian may decide to stop go at a particular moment based on these conditions. Consider situation where pedestrian standing near the boundary of the curb or walking towards the curb on a traffic light junction looking forward to cross the driving lane of the ego-vchicle. The information about the green traffic light status for the vehicle might help to predict that the pedestrian will keep standing stop at the curb, i.e., the intention of the pedestrian is not to cross. For this reason, is necessary to consider surrounding factors like traffic light status in addition to behavioral cues make an accurate prediction.
Object-based context cues such as pedestrian location over a period of time can provide rich infor- mation about VRU motion. But, it is challenging perceive features like human interactions with the ego-vehicle that can determine its maneuvering. Humans exhibit highly variable motion patterns, and even the same gesture or activity may differ subtly among individuals based on geographic lo- cations. In such a case, it is helpful to divide the task into a smaller sequence of tasks that can be solved independently. In the VRU case, we can learn a model infer an sppearance-invarrant rep- resentation. The articulated pose of VRUs is one such representation, commonly used in literature for action recognition (Duan et al. 2022). gesture recognition (Mitra Acharya, 2007). emotion recognition (Shi et al. 2020) and intention prediction (Kotseruba et al. [021)). These object-based features along with surrounding information about the pedestrian can be combined over temporal domain to generate reliable predictor for VRU actions in the future. In this paper, explore this approach and attempt to predict pedestrian crossing intention for a future time horizon of 1-2s by observing them for time horizon of 0.5ss
The handling of VRUs is saitty- so it is important to be aware of the uncertainty of the model that predict their behavior well. Despite much emphasis safety, deep learning models are often deployed as black box that do not offer reliability and interpretability. As a result, they do not indicate how system will behave under unknown circumstances. To interpret how the

---

Under review as a conference paper at ICLR 2023
model behaves in such situations, we intend predict the uncertainty of each prediction to know the confidence of our model.
2 RELATED WORK
2.1 ARCHITECTURES FOR PEDESTRIAN CROSSING INTENTION PREDICTION
To build a safer AV system for urban roads, it is important to estimate the crossing intention of pedestrian, i.e whether a pedestrian intends to cross/not-crow the road in front of ego-vehicle for a predefined time horizon. Pedestrian crossing intention prediction is mostly treated where the goal is to classify between two classes for pedestrian intention, i.e. Crossing (C) or Not Crossing (NC). One of the early works in this direction, as proposed by Rasouli et al. (017))
binary task
to predict crossing action at given frame using a static representation of the traffic scene and encod- ing pedestrian looking and walking actions using CNNs. Razali et al. (202I) propose a multitask architecture to estimate the intention and pose of a pedestrian simultaneously using RGB images as input. A multi-task architecture with an encoder-decoder based intention and action prediction to predict pedestrian crossing intent and forecast future behavior of the pedestrians is presented by Yao jet al. (202T). The authors also propose an attentive relation network to extract important features from traffic objects and scenes to improve the performance of the intention and action detection framework. [oreezo et al. (202Tb) and Lorenzo et al.] (2021a) use vision transformers to encode the non-visual features. They experiment with different types of video encoders and finally fuse the features from the two branches to predict pedestrian crossing intent. Kotseruba et al. (202T) present pedestrian action prediction model along with common evaluation criterion. In this paper, the authors evaluate different architectures for pedestrian action recognition, namely static (crossing prediction made using only last frame in the observation sequence), Recurrent Neural Network (RNN) models, 3D convolution and optical flow based models. They propose a novel architec- ture based on 3D convolution and multiple RNNs and experiment with different input features like bounding box, local-context, human-poss keypoints and ego- speed. We base our experi- ments on this architecture and perform ablation studies get insights on drouputs and uncertainties. Yang et al. (2022) fuse different phenomena such as sequences of RGB imagery, semantic segmen- tation masks, and ego-vehicle speed in an optimum way using attention mechanisms and a stack of recurrent neural networks. AAhaaji et al. (2022) present a framework based on multiple variations of Transformer models to predict the pedestrian street-crossing decision, based on the dynamics of its initiated trajectory, using only bounding boxes input.
2.2 FEATURES USED FOR PREDICTING PEDESTRIAN CROSSING INTENTION
Body language is generally modeled as head orientation, body orientation, posture and gesture which is often used to estimate the pedestrian intention in future. Yang & Ni (209)) use two vision cues to estimate a pedestrian's crossing intention. They propose looking/not-looking classifier using a 2D convolutional CNN to capture the eye contact between pedestrian and the ego-vehicle. They also come up with a C/NC classifier based on 3D CNN to model the pedestrian's 's early crossing action. Roth et al. (222I) propose a method to estimate vehicle-pedestrian path prediction that takes into account the awareness of the driver and the pedestrian towards each other. They extend Dynamic Bayesian Network (DBN) method by Kooij al. (2014) where they perform path pre- diction for an individual pedestrian, to the mutual vehicle-pedestrian case. Their results indicate that driver-attention-awarr models improve collision risk estimation compared driver-agnostic models. Human-pose is an intermediate representation which is very useful to determine various human behaviors. Quintero et al. (2017) propose a method to recognize pedestrian intentions such as standing, walking, stopping and starting based on a Hidden Markov Model (HMM). The authors use 3D positions and displacements of 11 skeleton points. They also propose a single frame skeleton estimation algorithm based on point clouds extracted from a stereo pair. |Fng & López (2018) 8) use CNN-based pedestrian detection, tracking and pose estimation to predict C/NC action for pedestri- ans. They use classifier to predict C/NC using human--pse features. The authors extend their work, to recognize the intention of the cyclists along with the pedestrians (Fang & López 2019). Minguez et al. (2018) present method to predict the future path, pose and intentions of the pedestrians up to a time horizon of 1s. The authors use balanced Gaussian process dynamic models (BGPDM) to learn 3D time-related information extracted from the skeleton points.
2

---

Under review as a conference paper at ICLR 2023
2.3 MODELING UNCERTAINTY
To ensure the safety of the AV, it is critical to estimate the uncertainty of model predictions. Gal (2016) show that uncertainty of a deep neural network model, i.c,, the epistemic uncertainty or model uncertainty. can be approximated by dropout training. Extending the work of Kendall & Gal (2077) the authors present a Bayesian deep learning framework to estimate both the epistemic and aleatoric uncertainties together. Recently, [juuric al. (020) presented high definition map based approach to predict the future trajectories of the traffic agents, taking the inherent uncertainty of the predictions into account. Peng et al. (202I) propose a method fine-tune the object detection performance of a deep neural network by calibrating the confidence of the algorithm using model uncertainty. To estimate the model uncertainty, the authors use Monte Carlo Dropout (MCD) method. Model uncertainty can also be used for collecting meaningful data by using active learning as proposed by Kaushal et al. (2019), i.e., high uncertainty samples can be collected and the model can be fine-tuned on these samples to make it more robust through an iterative process.
Our contributions:
We demonstrate that adding traffic light status additional input improves crossing inten- tion prediction at traffic light junctions by evaluating it on three state-of-the-ar baselines (Kotseruba et al. (2021). Yang et al. (2022). Achaji al. (022)).
Introducing uncertainty in crossing intention prediction to produce reliable and inter- pretable results. This can be used by downstream tasks like sensor fusion and trajectory planning.
Analyze attention weights distribution and uncertainty for models by sampling different training distributions on PIE dataset.
PROPOSED APPROACH
Solution Formulation: The model takes the following observations as input: 2D bounding- box bt-m: bt.mm, b, defined by top-left and bottom-right image coordinates, pose of the pedestrian Pt-m: Pt-m:Pttmmt.-... Pt. speed of the ego-vehicle St-m- $t-m+1, ...., St. local-context Ct-m, Ct-m+1, ct where is the cropped RGB image of the scene around the pedestrian and traf- fie light status tl tl, 144m+t1........ ..... tl, where is the time-to-event and is the observation length. Our goal is to predict a crossing action A = {C. NC). and epistemic uncertainty (U) of the model.
The network architecture of our proposed method depicted in Figure I is inspired by PCPA (Kotseruba et al. 2021). It consists of parallel RNN branches to compute non-visual features such as bounding-box, human-pose and ego-vehicle speed. We add traffic light status as an additional contextual feature. To compute visual features or RGB input which is referred to as local-context, the image crop of the enlarged bounding box (factor 1.5). is processed by 3D convolutions. Each RNN encoder produces a vector (h1, /22, h3, fim)) of hidden states where h = f(z,, h4-1).
Attention mechanism is used to combine features from different time frames and modalities as im- plemented in PCPA (Kotseruba et al., 202T). Temporal attention is used to weigh input for each time step and focus on important temporal events. It learned for m 16 hidden temporal states for each of the RNN branches. The attention mechanism applied here is the Luong's 's multiplicative attention (Luong et al. 2015). The goal is to get an attention weight vector a with length equal to observation length m. Further, modality attention is learned for weighing all the different input modalities in a similar manner. Here, the attention mechanism is applied to all the final vectors from 3D convolution branch and temporal attention outputs i.c. human-pose, bounding-box, ego-vehicle speed and traffic light status.
Epistemic or model uncertainty which denotes under-representioo of samples in training distribution, is computed during inference by running a Monte Carlo sampling over the network for N number of times (Gal 2016).
P= ply Y Ix, Dirain)
3

---

Under review as a conference paper at ICLR 2023
Box max-
Encoders
Temporal atlention
Modality atlention
Weights fullow probability distribution
Posee mx.
Speed
(1x256)
Traffic light
RNN
mmx
hidden states (mn256)
FC
Local context Bix 2242x223
3D
Flatlen
FC
modality outputs (1x256)
C/NC
Epistemic uncertalnty
Architecture diagram the proposed approach inspired by PCPA (Kotseruba al. 2021 RNN: Recurrent Neural Network, FC: Fully Connected Layer, ATT: Attention block. Our contribu- tions are highlighted in green borders. We incorporate the traffic light status as an additional input. Dropouts at temporal and modality attention blocks are added which are also active during run-time for performing Monte Carlo simulation to give probabilistic output for each simulation run.
Figure 1:
H Dtrain] =
Σ
Plog P
where H represents the predictive entropy and is the measure of model uncertainty, Y represents the output classes C/NC and P represents the final probability vector for C and NC class which is then mapped to crossing action {C. NC) for the given observation horizon. It is obtained by averaging all output probability vectors from N forward passes.
EXPERIMENTS
4.1 DATASET
In this work, we use Pedestrian Intention Estimation (PIE) by Rasouli et al. (2099) which large public benchmark dataset for studying pedestrian behavior. It contains 6 hours of continuous footages recorded in Canada under clear weather conditions. provides annotations for all pedes- trians sufficiently close the road who may may not attempt cross in front of the ego-vehicle. We follow the data split defined by Rasouli et al. (2019), i.e., videos from set01, set02 and set04 are used for training (TRAIN split), set05 and set06 for validation (VAL split) and set03 for testing (TEST split). The number of pedestrian tracks in PIE is 880, 243 and 719 in TRAIN, VAL and TEST splits, respectively.
We use following explicit features that are annotated in PIE dataset: bounding- box coordinates, ego- vehicle speed, and attributes for the scene e.g. local-context and traffic light. In addition, human- pose information here is generated by OpenPose (Cao et al.) 2017). Thus, we have 18 skeleton point coordinates which are concatenated into a 36D feature vector. PIE dataset has annotations for
Figure 2: Pedestrian crossing parallel to the direction of motion of eeo-ve (A) and perpendicular to the direction of motion of ego-vehicle (B). The arrows red denote the direction of motion of pedestrian and ego-vehicle.
4

---

Under review as a conference paper at ICLR 2023
traffic light as entity, with traffic light status being undefined, red, yellow green. We associate the traffic light status to the corresponding pedestrians in that frame using frame-ids. This traffic light status mapped to the relevant pedestrians using semi-automatic labeling technique as follows: We first assign body-orientations for all pedestrians in the PIE dataset. This was achieved by adding body-orientation predictor to the VRU Pose-SSI) (Kumar al., (02TT) model which was trained on body-orientation labels in TDUP dataset (Wang et al.) 2021). ). Based on the values of body-orientation, label the pedestrians to be crossing parallel or perpendicular to the direction of motion of ego-vehicle as shown in Figure We then associate the traffic light status in the scene, for perpendicular crossings and not-crossing pedestrians, as the traffic light in the scene is valid only for these pedestrians. This is because the parallel crossings are not regulated by the front traffic lights which visible to the ego-vehicle. For pedestrians crossing parallel to the direction of motion of the ego-vehicle, we encode the traffic light status to be undefined during training and inference.
4.2 COMPARISION RESULTS ON DIFFERENT ARCHITECTURES
Metries: We report the results using standard binary classification metries: accuracy, Area Under Curve (AUC) and Fi-score (F1) defined as F1 (x(preiision xrenal)))))))))) rreaal)
sprecision++rcaul
To evaluate crossing predictions when adding traffic light status to inputs, we experiment with three different architectures: PCPA (Kotseruba al. (02T), Feature-Fusion (Yang et al., 2022) and TED model (Achaji et al. (002). The process of dataset preparation remains same across these architec- tures. We report consistent improvement across all metries when incorporating traffic light status as additional input. There is improvement of 5% (CIPTLUmt F1 0.82), 6% (Feature Fusion, F1 = 0.83) and 6% (TED, Fl 0.83) in the Fl-score respectively as shown in Table [T Since, the dataset is not balanced, i.e., there are more not-crossing pedestrians compared pedestrians (1322 NC vs. 512 C), we use Fi-score for performance comparision of the models. We strongly believe that adding traffic light status any architecture will improve its performance, as the model learns better representations to predict pedestrian behavior for stop-and-go transitions at traffic light junctions. Although, we compare these three architectures, we now focus our experi- ments and analysis of attention-weights distribution and uncertainty only on PCPA as it allows us
crossing
Table 1: Accuracy (Acc), Area Under Curve (AUC), and F1-score comparison with baseline models (1., 7. and 9.). All metrics reported here are on the TEST set of PIE dataset which is described under the dataset sub-section in Section: TL status refers to presence or absence of traffic light status in the inputs and MC Dropout refers to adding Monte Carlo Dropouts to the model. Here, modal / temp signifies dropout layers are applied inside the modality attention block or temporal attention block respectively as shown in Figure [ We extend PCPA to CIPU: Crossing Intention Prediction with Uncertainty, CIPTL: Crossing Intention Prediction with Traffic Light, CIPTLU: Crossing Intention Prediction with Traffic Light and Uncertainty. The subscripts / temp) where the dropout is applied. The models in boldface represent the best setting for that architecture. Inputs to Feature Fusion same that of PCPA wheres inputs to TED constitutes only bounding- boxes other than traffic light status.
and denotes the blocks (modal
S.No.
Model
TL status
MC Dropout
Acc
AUC
Fl
1.
PCPA Kotseruba et al. 2021)
no
no modal
0.87
0.86
0.77
2.
CIPTL
yes
0.89
0.86
0.81
3.
CIPU
no
0.86
0.85
0.77
4.
CIPTLU m
yes
modal
0.89
0.87
0.81
5.
CIPTLU,
yes temp 0.90
0.87
0.81
6.
CIPTLU ml
yes modal, temp
0.90
0.87
0.82
7.
Feature Fusion (Yang et al.
2022)
yes no 0.87
no
0.84
0.78
8.
Feature Fusion
0.91
0.87
0.83
9.
TED (Achaji et al.
2022
0.86
0.86
0.78
10.
TED
yes no
0.89
0.89
0.83
5

---

Under review as a conference paper at ICLR 2023
to experiment with several inputs and also uses attention mechanism which gives deeper insights on representations learned by the model.
4.3 EXPERIMENTS ON PCPA
Inputs, model architecture and training: By adding traffic light status input to PCPA and adding
MCD on temporal and modality attention blocks, we report a improvement of 5% (F1 = 0.82) in the Fl-score as shown Table | (CIPTLUmb) Following Kotseruba al. (202I), we use RNN based model with 256 hidden units for each non-visual input and a conv3D network for encoding RGB input. The number of observation frames (m) is set 16 (= 0.5 and the time-to-event is 30 60 frames (1. Dropout of 0.5 is added before temporal and modality attention. 1.2 regularization of 0.001 is added to the final dense layer. The batch-size is set and model is trained with Adam optimizer. It is trained for 60 epochs with 5 x 10 learning rate.
Effeet of Monte Carlo Dropout: As mentioned in Section 3 Monte Carlo sampling is performed
N number of times during inference where N = 10. Even for higher values of N, we observe that the distribution of uncertainty remains same. We experiment with dropout layer at several places like inside the temporal attention block and modality attention block shown in the Table Applying dropout for both temporal and modality attention gives the best results (F1 = 0.82) for PCPA model. Further, we add dropout before the last FC layer but it neither improves nor degrades the performance (F1 = 0.81). We also perform experiments with different values of dropouts, namely 0.25, 0.50 and 0.75 where we observe that dropout value of 0.50 gives the best results whereas the Fl-score either degrades or remains constant for other values.
Ablation study model uncertainty: Results of experiments analyze the varation of samples with high uncertainty values w.r.t. time horizon are shown in Figure 3 Since pedestrian behavior becomes more complex when predicting for longer horizons, the uncertainty increases as the time horizon increases.
Variation high uncertainty samples (U > 0.66) w.r.t time harizon
% of ww..ttlliy sampplos
30
25
15
10
1-2
2-3
3-1
6-5
5-6
6.7
7-8
Time barizon (in s)
Figure 3: Percentage of samples with high uncertainty increases as the training and prediction hori- zon boundary is increased.
We perform experiments to analyze the effect of under-represented samples on uncertainty. In- order achieve this, create modified TRAIN splits by removing type of distribution from the PIE TRAIN split. We perform following three experiments by creating under-repressenaatios with-respect-to bounding-box, ego-vehicle speed and traffic light status: For bounding-box (Figure 4 A), we remove all the samples from the TRAIN split with pedestrian height less than 100 pixels get the modified TRAIN split for bounding- to train a new model. Uncertainties are then filtered for the samples with pedestrian height less than 100 pixels in the TEST split. Similarly, we perform experiments by removing all samples with speed equal to zero (Figure 4 B) and removing all samples with green-light status in the TRAIN split (Figure 4 C). When we compare the uncertainties on same samples from the original model, observe that the uncertainties are clearly shifted towards larger values for the under-represented samples. This demonstrates the effectiveness of model uncertainty in estimating out- ouittf-distribution samples in the training distribution.
experiment. We use this modified TRAIN split
Model uncertainty and misclassification rate: correlation with misclassification rate (ratio of number of incorrect predictions to total number of predictions) for crossing action prediction. This remains true for both the models with and without traffic light status (see Figure 5). This property of uncertainty can be utilized for ignoring samples
We observe that increase in uncertainty has a strong

---

Under review as a conference paper at ICLR 2023
Uncertainty samples with box height 100 px. (TRAIN)
Uncertainty samples with speed 0 (TRAIN)
Uncertainty samples with
green-light status (TRAIN)
No. of samples
No. of campless
No. of samplees
250
OLA 6.12
10.1K 6.34
01.44 8J
01.% 8
8.54
01 Ma
0000 8.73
(0 6,64
(00% 6.
00.18 6:24
00.. 8.3
(L4L 6-N
(LSA) B
00.06 6:73
(M. 6
0L172 6
00.3 RAN
8.34
0154, BA
0 9M
(LUA) 6:73
10.34
(0
Model uncertuinty
Model uncertaiaty
Model uncertainty
Uncertainty on samples
Uncertainty samples
Uncertainty on samples
bex beight < 100 (modified TRAIN)
speed = (modified TRAIN)
green-light status (modified TRAIN)
No. of samples
No. of samples
201
No. of samples.
201
Jn 9.001
(MUK 0.21
ILIK 0.44
n0 0:4
(1SS) 941
(0LA 8.11
(90% 6.
(018 6:24
(4tt 6-8
(1.4 B4
(1A (1A-AA
(8066 6.73
6MH
0..33 0J
6336 GAM
0.48 8.34
(1.5) M
(UA 9AH
(RA6) 6.72
(0,
0..44
Model uncertainty
Model uncertainty
Model uncortainty
(A)
(C)
Figure 4: Ablation study uncertainty with inputs. The output uncertainties are from the model CIPTLUmt trained on datasets mentioned in the figure. i.e., TRAIN and modified TRAIN which represents model trained on TRAIN split and reduced TRAIN splits respectively. The samples represented in the graphs are all the samples with mentioned settings on box height, speed and traffic light status from the TEST split. (A), (B) and (C) correspond uncertainty ablation experiments with bounding-boxt ego-vehicle speed and traffic light status respectively.
Model uncortainty without TT.
Model uncertainty with 11.
(900)
8
200
64.00
R.6-210
24444.44
444644
4446440
Rase-77
Model smcer lainty
M Model uncertainty
samples
Number of samples in bin
Mislassifiaation Rate
Figure 5: Misclassification rate of model uncertainty without TL status (A) and with TL status (B). The misclassification rate increases with model uncertainty for both models. The uncertainty values recorded in model with traffic light shift towards lesser values, which shows the model is more certain when using traffic light status.
with high uncertainty and spawning crossing action for pedestrians only with reasonable uncertainty for downstream tasks like sensor fusion and trajectory planning.
Analysis of attention weights: We compare the attention weights for the inputs of the model and found that local-context features still dominate for few samples in the model trained with traffic light status. There has not been much change in the attention weights of the earlier inputs like local- context, bounding box, human-pose and ego-vehicle speed except that they are pushed towards lower values to incorporate the weight given to traffic light status. But most importantly, there are no cases where traffic light status dominates as feature as shown in Figure 6 This distribution is important, as it shows that adding traffic light status as input does not shift the weights entirely towards traffic light. Domination of traffic light in attention weight distribution would have meant that the model ignores all other cues which might be crucial in scenarios where the pedestrians are violating the traffic light rules. We observe that PIE dataset does not contain cases where the pedestrian violates traffic light. Such scenarios play an important role in modelling rare pedestrian behavior (jaywalking on signalized junctions) which can lead to accidents.

---

Under review as a conference paper at ICLR 2023
Modality attention weights without TL
Modality attention weights with TI
3080
3900
2900
2580
2000
1900
1000
1000
300
(04.0 (01-4.2)00 212--00-2---1110 11182/4220 I B
[6-2.] [8.1-400 (IZALI (LS-I ja G-Ss) (is-tu (1o-41)) aus-a.m (ea+ (1o-La))
Attention weights
Attention weights
local box
bbox
pose
spred
local, box
hbox pose
speed
(A)
Figure 6: Attention weights distribution in model trained without (A) and with (B) traffic light status. Local-context still dominates the weights distribution for some samples. Additionally traffic light status does not dominate the weights distribution in any of the samples.
Qualitative results: Some examples of low and high uncertainties are shown in the Figure Many low uncertainty samples belong to the category of most common scenarios like NC scenarios where the pedestrian is standing near the curb (Figure 7 A) and the car is at moderate speed (greater than 30 kmph). It is observed that uncertainty is high for samples with shorter pedestrian height (children) (Figure 7 B), occlusion (Figure 7 C) and car taking a turn (Figure 7 D) as these category of samples under-represennte in PIE dataset.
comparision of success and failure cases on PIE TEST split for the models trained without and with traffic light status (left and right snippets repectively) is shown in the Figure g The success cases occur when the traffic light status is either green or red (Figure g A and B). On the other hand, failure cases are samples with yellow light status (Figure 8 C). This is because the PIE dataset con- tains low amount of samples with yellow light as compared to red and green light (as yellow light is transition light between red and green). The predictions become more stable across the 0.5 predic- tion window (Figure 8 D) which is attributed to improvement in F1-score. Also, the uncertainties for samples with traffic light (only for red and green light, as yellow light under-represented in PIE dataset) have drastically reduced for model trained with traffic light status (compare U values for samples left and right snippet). This shows the confidence of the model has increased when adding traffic light status. This is also evident for the model uncertainty graphs shown in Figure 5 as the overall uncertainty distribution shifts towards lower values in the model trained with traffic light status.
GT=
U-0.60
GT
Liso CO
Figure 7: Examples for low (A) and high uncertainty (B), (C), (D) samples from PIE dataset. GT: Ground Truth (C/NC) and epistemic uncertainty (U) are marked on top and bottom of the snippets respectively. Snippets encapsulated red indicates wrong model prediction and green encapsulta- tion indicates correct model prediction.
8

---

Under review as a conference paper at ICLR 2023
GT NC
[A]
U=0.65
GT
U=0.69
GT=C
LIHOUS
U10061
GTT M.C.
[D]
T=12
T-265
Figure 8: Comparision of predictions from the model without traffic light status (CIPU,, left) and with traffic light status as input (CIPTLUm- right). Refer to Figure caption for explanation to border color and abbreviations. Snippets with colored circle top (green/ red/ yellow) indicates the traffic light status for that sample. Snippets in (A), (B) and (C) belong to the same subsample of the observation window (16 frames x 0.5 whereas in (D), the snippets from subsequent subsamples with an overlap of 10 frames (/= 0.3
CONCLUSION AND FUTURE WORK
We demonstrate that adding traffic light status input improves crossing intention predictions the model learns representations for traffic light status as an additional contextual feature that influence pedestrian behavior. We prove this by showing improvement when incorporating traffic light status across three baseline models for crossing prediction. We also demonstrate that a model becomes more interpretable and robust when it predicts uncertainty for the crossing intention prediction, it provides a way to assess the reliability of the predictions by estimating a factor by which the sample represented in the model. Further, we demonstrate that traffic light status does not dominate completely over other inputs by analyzing the attention weights distribution for the inputs. We also show that uncertainty of pedestrian intention prediction directly proportional to the length of prediction horizon.
We also observe that the PIE dataset does not cover a variety of traffic light scenarios. As pedestrians are likely to break rules on the road, one of the most important scenario that is missing in the dataset the jaywalking cases traffic light junctions. With uncertainty, our proposed model can deal with jaywalking to a limited degree. We argue, that preparing the model explicitly for these scenarios can ensure more accurate predictions at traffic light junctions. Therefore, there need for dataset that has sufficient samples for jaywalking with and without traffic light junctions. We aim to prepare such dataset as one of our future works. Further, we want to optimize the model to avoid repetitive computations at runtime to perform MC sampling for uncertainty predictions. This work provides us with the necessary framework and evidence to further research uncertainty estimations and modelling surounding factors like traffic light status for pedestrian crossing intention prediction.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Lina Achaji, Julien Moreau, Thibault Fouqueray, Francois Aioun, and François Charpillet. Is at- tention to bounding boxes all you need for pedestrian action prediction? 2022 IEEE Intelligent Vehicles Symposium (IV), pp. 895-902, 2022.
The Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7291 2017.
Nemanja Djuric, Vladan Radosavljevic, Henggang Cui, Thi Nguyen, Fang- Chig--hhihhh Chou, Tsung- Han Lin, Nitin Singh, and Jeff Schneider. Uncertainty-awee aware short-term motion prediction traffic actors for autonomous driving. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2095-2104, 2020.
Haodong Duan, Yue Zhao, Kai Chen, Dahua Lin, and Bo Dai. Revisiting skeleton-based action recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 2969-2978, 2022.
Zhijie Fang and Antonio M López. Is the pedestrian going cross? answering by 2d pose estima- tion. 2018 IEEE Intelligent Vehicles Symposium (/V), pp. 1271-1276, 2018.
Zhijie Fang and Antonio M López. Intention recognition of pedestrians and cyclists by 2d pose estimation. JEEE Transactions Intelligent Transportation Systems, 21(11):4773-478. 2019, 2019.
Yarin Gal. Uncertainty in deep learning. Phl) thesis, University of Cambridge, 2016.
Vishal Kaushal, Rishabh Iyer, Suraj Kothawade, Rohan Mahadev, Khoshrav Doctor, and Ganesh Ramakrishnan. Learning from less data: A unified data subset selection and active learning framework for computer vision. 2019 IEEE Winter Conference on Applications of Computer Vision ( (WACV), pp. 1289-1299, 2019.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in Neural Information Processing Systems, 30, 2017.
Julian Francisco Pieter Kooij, Nicolas Schneider, Fabian Flohr, and Dariu M Gavrila. Context-based pedestrian path prediction. pp. 618-633, 2014.
Juliía Kotseruba, Amir Rasouli, and John K Tsotsos. Benchmark for evaluating pedestrian action prediction. Proceedings the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1258-1268. 2021.
Chandan Kumar, Jayanth Ramesh, Bodhisattwa Chakraborty, Renjith Raman, Christoph Weinrich, Anurag Mundhada, Arjun Jain, and Fabian B Flohr. Vru pose-ssd: Multiperson pose estimation for automated driving. Proceedings of the AAAI Conference on Artificial Intelligence, 35(17): 15331-15338, 2021.
Javier Lorenzo, Ignacio Parra Alonso, Rubén Izquierdo, Augusto Luis Ballardini, Álvaro Hernández Saz, David Fernández Llorca, and Miguel Ángel Sotelo. Capformer: Pedestrian crossing action prediction using transformer. Sensors, 21(17):5694, 2021a.
Javier Lorenzo, Ignacio Parra, and MA Sotelo. Intformer: Predicting pedestrian intention with the aid of the transformer architecture. arXiv preprint arXiv:2105.086. 2021b.
Minh-Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention- based neural machine translation. arXiv preprint
2015.
Raúl Quintero Mínguez, Ignacio Parra Alonso, David Fernández--lloraa, and Miguel Angel Sotelo. Pedestrian path, pose, and intention prediction through gaussian process dynamical models and pedestrian activity recognition. IEEE Transactions on Intelligent Transportation Systems, 20(5): 1803-1814 14, 2018
Sushmita Mitra and Tinku Acharya. Gesture recognition: A survey. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 37(3):111-324 2007.
10

---

Under review as a conference paper at ICLR 2023
Liang Peng, Hong Wang, and Jun Li. Uncertainty evaluation of object detection algorithms for autonomous vehicles. Automotive Innovation, 4(3):241 252,, 2021.
Raúl Quintero, Ignacio Parra, Javier Lorenzo, D Fernández-Llorea, and MA Sotelo. Pedestrian intention recognition by means of hidden markov model and body language. 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), pp. 1-7, 2017.
Amir Rasouli, Juliia Kotseruba, and John K Tsotsos. Are they going to cross? benchmark dataset and baseline for pedestrian erosswalk behavior. Proceedings of the IEEE International Confer- ence on Computer Vision Workshops, pp. 206-213, 2017.
Amir Rasouli, Iuliia Kotseruba, Toni Kunic, and John K Tsotsos. Pie: A large-scale dataset and mod- els for pedestrian intention estimation and trajectory prediction. Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 62626666226222 6271, 2019.
Haziq Razali, Taylor Mordan, and Alexandre Alahi. Pedestrian intention prediction: A convolutional bottom-up multi-task approach. Transportation Research part C: Emerging Technologies, 130: 103259, 2021.
Markus Roth, Jork Stapel, Riender Happee, and Dariu M Gavrila. Driver and pedestrian mutual awareness for path prediction and collision risk estimation. IEEE Transactions Intelligent Vehicles, 2021.
Jiaqi Shi, Chaoran Liu, Carlos Toshinori Ishi, and Hiroshi Ishiguro. Skeleton-based emotion recog- nition based two-stream self-attention enhanced spatial-temporal graph convolutional network. Sensors, 21(1):205, 2020.
Sijia Wang, Diange Yang. Baofeng Wang, Zijie Guo, Rishabh Verma, Jayanth Ramesh, Christoph Weinrich, Ulrich Kreßel, and Fabian Flohr. Urbanpose: A new benehmark for vru pose esti- mation in urban traffic scenes. 2021 IEEE Intelligent Vehicles Symposium (/V), pp. 1537-1544, 2021.
Biao Yang and Rongrong Ni. Vision- based recognition of pedestrian crossing intention in an ur- ban environment. 2019 IEEE 9th Annual International Conference CYBER Technology in Automation, Control, and Intelligent Systems (CYBER), pp. 992-995, 20129.
Dongfang Yang, Haolin Zhang, Ekim Yurtsever, Keith A Redmill, and Ümit Özgüner. Predicting pedestrian crossing intention with feature fusion and ppatio-tepoorl attention. IEEE Transac- fions on Intelligent Vehicles, 7(2):221 2022.
Yu Yao, Ella Atkins, Matthew Johnson Roberson, Ram Vasudevan, and Xiaoxiao Du Coupling intent and action for pedestrian crossing behavior prediction. Proceedings of the International Joint Conference on Artificial Intelligence, pp. 1238-122 1244, 2021.


Reviews:
[{"review_id": "hqB0XXN9sb", "reviewer": "Reviewer_DUfQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a method for predicting pedestrian crossing intentions. The model is an extension of the PCPA model proposed in Kotseruba et al. On top of the PCPA model, they 1) added the traffic light inputs and 2) used the dropout method to predict the epistemic uncertainty.\n\nThe authors evaluated the proposed model on the PIE dataset, and the model is shown to improve the prediction accuracy over the PCPA baseline. Moreover, the predicted epistemic uncertainties are shown to correlate with the misclassification rate.", "summary_of_the_review": "The novelty of this paper is rather low, so I am giving it a weak reject.", "strength_and_weaknesses": "--- Strengths\n\n- Pedestrian crossing intention prediction is an important problem. This paper presents a model that improves over the state-of-the-art baseline.\n\n\n--- Weakness\n\n- Novelty is the biggest issue of this paper. The network architecture proposed in this paper is almost exactly the same as the PCPA model in Kotseruba et al. The only difference is the addition of the traffic light inputs, which is a trivial change.\n\n- The improvement over the baseline is relative minor.\n\n- Although the evaluation result shows the proposed model improves the accuracy of pedestrian crossing prediction, but I doubt the real impact of this work. When the light on the ego lane is red, the ego vehicle will need to stop anyway, regardless of whether the pedestrian will cross the street or not. With the additional traffic light inputs, the model will unsurprisingly become better at predicting the crossing intention for those pedestrians crossing perpendicular to the ego vehicle, but the predictions for those pedestrians rarely matter. What truly matters is to predict whether a pedestrian will cross the street when the ego vehicle is moving (e.g., jaywalking).\n\n- The proposed model predicts the epistemic uncertainty, which can be addressed by adding more training data. What matters more is the aleatoric uncertainty.\n\n- The PIE dataset has only 880 training samples, which is way too small for this task. With a larger dataset, epistemic uncertainty will not matter that much anymore.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "QwlT9enYi9E", "reviewer": "Reviewer_ZTh8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The work introduced the usage of traffic signals into the problem of predicting the crossing intention. The work has a good amount of ablation study. ", "summary_of_the_review": "The work need more results on other datasets. \nThere is no significant model design introduced. ", "strength_and_weaknesses": "The problem of Vulnerable Road User is not common in literature. \nThe sole contribution of the work is in introducing the usage of traffic light status through attention into previous models that is used to solve this problem.\n\nThe model was evaluated on a single dataset ignoring other datasets like JAAD[1]. \nThe introduced model is the same as used in [1] with exception to adding the traffic status as the work indicates.\nIncremental performance where AUC is increased by 1-3% same for the accuracy. There is no std reported for the results to check if these numbers are due to different random seeds or not.\nThe analysis of method and different components section 4.3 is quite good. \n\n[1] \"Benchmark for Evaluating Pedestrian Action Prediction\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PS9DS5FqDO", "reviewer": "Reviewer_MM9u", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors explore the benefits of using traffic light (TL) information for predicting the probability of a pedestrian crossing the street. They use existing methods and architectures to which the TL inputs are added, and show the benefits of this new information. In addition, they explore the epistemic uncertainty of their model, and correlate it with the model performance (such as misclassification rate).", "summary_of_the_review": "The work has very limited novelty. The authors use existing methods (such as the architectures that they consider), and apply a new TL input without any deeper considerations or insights. In addition, they explore using dropout to estimate epistemic uncertainty which is a well-known method and no novelty can be claimed there. In fact, the authors only focus on epistemic and disregard aleatoric uncertainty, which would help and add some more novelty to the work. \nDetailed comments follow:\n- The authors say \"even the same gesture or activity may differ subtly\" to say that new features are needed to capture interactions, yet a few sentences later they say that this motivates them to use pose features which doesn't really help in that regard.\n- The architecture that is used is not well explained, and the authors should make their work more self-contained. E.g., they don't clarify well how are attentions implemented and what attends over what. They do provide a reference, however given already low novelty and the fact that this attention is explored quite a lot in the experiments, this important part of their methodology should be better explained in the text.\n- In relation to what are equations (1) and (2) added? It is unclear, they seem to be suddenly added to the text.\n- Traffic light info and its relevance should be discussed much more, given that that is the main topic of the work.\n- As explained at the end of Section 4.1, a lot of TL data is actually ignored (when it comes to pedestrians walking in parallel). More statistics about the data should be added to better quantify and understand this aspect.\n- How is MCD actually used in the work? Do you run multiple times and then average? This is not explained.\n- The authors point out that rare appearance of jaywalkers in the used data is affecting their results, however they do not dig deeper into this aspect which would be an interesting addition to the work. They leave it for future work which is unfortunate.\n- In general, the experimental results are not adding a lot of value, as many are already well known and discussed before (such as a large part of uncertainty results). I did find Fig 4 interesting however.", "strength_and_weaknesses": "Strengths:\n- Relevant and timely work\n- Promising results using the new TL data\n\nWeaknesses:\n- Very limited novelty\n- The paper can be written better, it is not fully self-contained", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject