Recognition of driving objects in real time with computer vision and deep neural networks

Traffic is one of the key elements nowadays that affect our lives more or less in a every day basis. Traffic is present when we go to work, is on week ends, on holidays, even if we go shopping in our neighborhood, traffic is present. Every year, we can see on TV how after a bank holiday (United Kingdom public holiday), the traffic incidents are a figure that all TV news are obliged to report. If we see the accident causes, the ones caused by mechanical failures are always a minimal part, being human causes the majority of times. In our society, the tasks where technology help us to complete those with 100% success are very frequent. We tune our TVs for digital broadcasting, robots complete thousands of mechanical tasks, we work with computers that do not crash for months or years, even weather forecasting is more accurate than ever. All those aspects in our lives are successfully carried out on a daily basis. Nowadays, in traffic and road transport, we are starting a new era where driving a vehicle can be assisted partially or totally, parking our car can be done automatically, or even detecting a child in the middle of the road can be automatically done instead of leaving those tasks to the prone-to-fail human. The same features that today amaze us (as in the past did the TV broadcast in colour), in the future, those safety features will be a common thing in our cars. With more and more vehicles in the roads, cars, motorbikes, bicycles, more people in our cities and the necessity to be in a constant move, our society needs a zero-car-accidents conception, as we have now the technology to achieve it. Computer Vision is the computer science field that since the 80s has been obsessed with emulating the way human see and perceive their environment and react to it in an intelligent way. One decade ago, detecting complex objects in a scene as a human was impossible. All we could do was to detect the edges of an object, to threshold pixel values, detect motion, but nothing as the human capability to detect objects and identify their location. The advance in GPUs technology and the development of neural networks in the computer vision community has made those impossible tasks possible. GPUs now being a commodity item in our lives, the increase of amount and speed of RAM and the new and open models developed by experts in neural networks, make the task of detecting a child in the middle of a road a reality. In particular, detections with 99.79% probability are now possible, and the 100% probability goal is becoming a closer reality. In this thesis we have approached one of the key safety features in systems for traffic analysis, that is monitoring pedestrian crossing. After researching the state-of-the-art in pedestrian movement detection, we have presented a novel strategy for such detection. By locating a fixed camera in a place where pedestrians move, we are able to detect the movement of those and their direction. We have achieved that task by using a mix of old and new methodologies. Having a fixed camera, allow us to remove the background of the scene, only leaving the moving pedestrians. Once we have this information, we have created a dataset of moving people and trained a CNN able to detect in which direction the pedestrian is moving. Another work that we present in this thesis is a traffic dataset and the research with state-of.the-art CNN models to detect objects in traffic environments. Crucial objects like cars, people, bikes, motorbikes, traffic signals, etc. have been grouped in a novel dataset to feed state-of-the-art CNNs and we carried out an analysis about their ability to detect and to locate those objects from the car point of view. Moreover, with the help of tracking techniques, we improved efficiency and robustness of the proposed method, creating a system capable of performing real-time object detection (urban objects). In this thesis, we also present a traffic sign dataset, which comprises 45 different traffic signs. This dataset has been used for training a traffic sign classifier that is used a second step of our urban object detector. Moreover, a very basic but important aspect in safety driving is to keep the vehicle within the allowed space in the road (within the limits of the road). SLAM techniques have been used in the past for such tasks, but we present an end-to-end approach, where a CNN model learns to keep the vehicle within the limits of the road, correcting the angle of the vehicle steering wheel. Finally, we show an implementation of the presented systems running on a custom-built electric car. A series of experiments were carried out on a real-life traffic environment for evaluating the steering angle prediction system and the urban object detector. A mechanical system was implemented on the car to enable automatic steering wheel control.