Interpreting complex scenes using a hierarchy of prototypical scene models

To drive safely, a good driver observes her surroundings, anticipates the actions of other traffic participants and then decides for a maneuver. But if a driver is inattentive or overloaded, she may fail to include some relevant information. This can then lead to wrong decisions and potentially result in an accident. In order to assist a driver in her decision making, Advanced Driver Assistance Systems (ADAS) are becoming more and more popular in commercial cars. The quality of these existing systems compared to an experienced driver is relatively low, because they purely rely on physical observation and thus react only shortly before an accident. To fully avoid a collision, a driver needs more time to react, therefore the driver should receive an early warning. For an earlier warning of the driver, behaviors of other traffic participants would have to be predicted. We classify existing research in this area with respect to two aspects: quality and scope. Quality means the ability to warn a driver early before a dangerous situation. Scope means the diversity of scenes in which the approach can work. In general we see two tendencies, methods targeting for broad scope but having low quality and those targeting for narrow scope but high quality. Our goal is to have a system with high quality and wide scope. To achieve this, we propose a generic framework, called Context Model Tree (CMT), that combines multiple high quality classifiers to predict if an entity is coming into the way of the ego-vehicle for many scenarios. This framework is a tree structure in which context based models are ordered according to their context specificity, from the generic ones in the top nodes to the most specific ones in the leaves. We have designed a set of activation rules to activate the nodes fitting to the current situation, using sensory information like GPS, digital maps or vision. To show that a combination of general and specific classifiers is a solution to improve quality and scope, this thesis introduces the generic concept of our system followed by a concrete implementation for predicting if an entity is coming into the way of the ego-vehicle when changing lane for highway scenarios. On the highway, a driver usually changes lane for a reason. Our models use complex features based on contextual information and relations between entities. On the highway, one of the most influential indicators to predict if a vehicle is going to change lane is a slow predecessor. A CMT for highway contains in the top node a model that uses such general indicators. Two models to predict lane changes at entrance and giveway lanes are placed as sub-nodes. These models make use of the specific information inherent to these contexts. We will provide a comparison of the quality of the three models separately and the combination of the models using a CMT and show that, in general, the CMT performs better in terms of prediction time horizon and prediction errors. In order to show the flexibility and adaptability of the CMT, we also present an extension of the framework for pedestrian crossing prediction in inner-city scenarios. In inner-city, a pedestrian who wants to cross a road without having the priority to do so and decide not to is usually influenced by its surroundings, for example a vehicle approaching too fast and not having enough time to cross. A CMT for inner-city contains in the top node a model that uses such general indicators to predict crossing behaviors at an early time for any road, in particular roads where pedestrians do not have the priority to cross. However, there are specific locations such as zebra crossings, where based on expert driving experience, one would expect that a prediction can be done even earlier. Therefore, we have developed an additional specific model fitted to the context of zebra crossings. This model makes use of the specific information inherent to this context. The experiments show that this model produces both, better and earlier predictions in this specific context. Because our goal is to build a generic behavior prediction system, we finally apply the framework of the CMT to combine the two models. We demonstrate that this multi-model system is well suited to provide early predictions for realistic data, including both, generic inner-city situations and zebra crossings. This work could therefore be a step towards better advanced Driver Assistance Systems (ADAS), through the generation of earlier warnings to increase the reaction time of a driver.