The Past Matters while Driving
By: Till Hartmann
Tags: Autonomous Cars, Autonomous Vehicles, Recurrent Neural Network, Self Driving Cars, Pedestrians

A car drives down the street. A pedestrian waits at a crosswalk. The pedestrian briefly looks up, makes eye contact with the driver and waves the car on, while the pedestrian continues to engage in a text conversation on her cell phone. The attentive driver watching the scene unfold is likely to conclude that they can safely continue to drive by the pedestrian because humans easily process scenes like this one; realizing that the pedestrian is aware of the car’s approach, and does not intend to cross the street. This seemingly trivial situation, which is frequently and easily “solved” by human drivers, poses a daunting challenge for autonomous vehicles.

Human drivers intuitively integrate the images of a pedestrian over time to understand the state of mind of pedestrians.

Hierarchical processing of information. Humans process visual scenes like the above relevant to driving, or any other visual task, through activity in a part of the brain known as visual cortex. Light beams reflected from the visual scene hit the retina in the back of the eye, where they are transduced into electrical impulses. These impulses travel via the thalamus — a relay station of sorts — to the back of our heads, where the visual cortex is located. The brain first detects edges and corners (basic building blocks of images), followed by detection and identification of more elaborate objects like pedestrians and the direction of their movements. The whole scene is analyzed, the direction of the car is estimated by calculating flow fields, and then finally the brain integrates all of the information together to form a cohesive perception of the world. Often, this information flow has been described as one that happens in a hierarchical or “feed forward” manner. That is, the images are processed at each consecutive stage independently, resulting in progressively more abstract representations. The understanding of hierarchical processing of visual scenes led to the development of convolutional neural networks (CNNs). The information flow of CNNs is typically feed forward, and they are standard building blocks in the perception stack of self-driving cars.

Visual information is processed in the human brain using an abundance of recurrent connections.

Memory through recurrency at every stage. The human brain doesn’t actually employ a simple feed forward hierarchy. Instead, the majority of connections between neurons are recurrent and feedback connections. Recurrent connections are connections between neurons at the same hierarchical level; feedback connections transmit information from a higher to a lower hierarchical level. While the purpose of these connections is still not fully understood — evidence suggests that they may enable contextual modulation in early stages of processing. This fundamentally differs from how most CNNs work. In our brains, nearly every single processing stage integrates context from the past with new information from the present, adding built-in memory capacity throughout the system. Just like seeing the pedestrian with the cell phone in their hand — a human driver who has been driving for — say 10 years — has seen that situation thousands of times. The memory of those situations allows humans to create and maintain mental models of pedestrians. It is feasible that similar structures built into machine learning models can lead to performance increases on complex problems encountered by self driving cars.

Judging pedestrian State of Mind. At Perceptive Automata, we believe that a critical step for AI in autonomous vehicles is to mimic the natural human ability to judge pedestrians’ state of mind in a given situation. As humans, we intuitively understand that a single glance doesn’t provide enough information about a pedestrian’s intent. Often subconsciously, we factor in details about posture, their movement history, and awareness history that allow us to quickly understand what the pedestrian is currently doing and predict what they are going to do in the future. Drivers can typically recall a pedestrian’s behavior over a minimum of several seconds, which they intuitively use to build a mental model of the state of mind of the pedestrian. Someone walking home at a swift commuter pace and showing little sign of slowing as they approach a crosswalk, as they think about their delicious meal, indicates to any experienced driver that they should stop. At Perceptive Automata, intent to cross and awareness of the vehicle are two key factors that we are using to create a Human State of Mind signal for self-driving cars.

Integrating the past. In the brain, recurrency allows contextual integration even at the earliest stages of information processing. This is different from the typical autonomous vehicle (AV) software stack. For example, detection of pedestrians happens one frame at a time. The CNN that performs the detection has to start from scratch every time it processes an image. The detection module rarely integrates over time, rather it “perceives” a snapshot; integrating the information and predicting the future is performed later. Again, this differs from how humans process information. After we initially detect a pedestrian, the brain’s processed signals aren’t forgotten, but are continuously updated with incoming new information including that of the surrounding context. Importantly, this happens not at a late or abstract processing stage, but even within early perception stages (early visual cortices). Recurrency or other forms of short term memory, added to perception modules, should improve the inference quality of human behavior. At Perceptive Automata, we work on machine learning algorithms that take inspiration from the best known system, the human brain, to infer the state of mind of pedestrians to be able to predict their behavior.

If we want autonomous vehicles to be accepted on our roads, they need to drive like humans — preferably not humans in the driver seat for the first time! We need algorithms and machine learning modules that can not only understand pedestrians in the moment but also predict the future by remembering the past. These algorithms need to integrate and apply that information to the current context. At Perceptive Automata, we are building technology inspired by the human brain, to hopefully enable self-driving cars to not only emulate human driving, but also become better than a human driver. When the AV remembers that the pedestrian waved the AV on, or stood on the side of the road texting and not crossing, it will drive by the pedestrian without braking, allowing passengers to sit back and relax during the smooth ride.