Building Intelligent Routing Models
By: Sanja Bogdanović Dinić
Tags: Data Preprocessing, Clustering, Routing, Trajectory Segmentation, Data Science

The significant expansion of crowdsourcing applications in the last decade, along with the vast availability of mobile phones and similar personal sensory devices, has inevitably generated a considerable amount of location data. The ability to transform locations into business value has triggered the interest of our data-driven economy in spatial data analysis. Geodata science has tapped its way into the emerging field of data-related solutions, expanding the reasoning behind such solutions to an additional dimension — a spatial dimension. But does location data simply add space to the existing challenges, or is it actually doing much more? Should space be observed as a single dimension, or is it much more complex, and how? These, and similar, intriguing questions are buzzing around the global data community, making known problems unknown again, and making known solutions obsolete.

There’s nothing more inspiring than a good challenge to dig into! Let’s take a common trajectory mining problem and go through data cleaning, trajectory discovery and routing in a quest for answers to some of these questions.

Problem Set

A spatial trajectory is a sequence of chronologically ordered geographical coordinates generated by a moving object. A moving object could be any object that changes its location over time and is equipped with a tracking device: people, vehicles, vessels, animals, even natural phenomena, such as hurricanes, tracked via satellite/aerial imagery. Generating such a dataset is the easy part. If the question were, “What can be done with it?“, the answer would be very broad, and it would generally depend on the specific usage requirements and the domain of one’s interest. However, if the question were, “How to deal with it?“, the discussion could steer towards the common trajectory mining challenges.

The focus of this research is on How, rather than What, and the effort made is concentrated on recognizing and solving specific challenges using a real dataset of geo locations gathered from taxi vehicles and buses in Aracaju, Brazil.

Dig into the Dataset

Similar to any other data science project, spatial data science projects start with dataset exploration. What is contained in the dataset, how is data structured or unstructured, what are potential anomalies and how they need to be transformed in order to result in a business value, are important questions that have to be addressed early on. So let’s look at our dataset.

The dataset contains geo locations of buses and taxi vehicles gathered via the Android app, and comes organized in two data tables: one holds a list of trajectories with additional features such as speed, distance, rating, car or bus indicators, while the other one contains timestamped geo locations for each trajectory. The raw dataset contains 163 taxi and bus trajectories and a total of 18 107 locations. The table below gives a statistical view of these trajectories.

Overview of trajectories by type in terms of total number, average number of locations and average distance traveled

It is noticeable that the dataset is well balanced in terms of the number of examples per trajectory type (taxi or bus), while taxi trajectories dominate in terms of the average number of locations, and average distance traveled. The figure below shows the distribution of these trajectories on a map of Aracaju.

Distribution of taxi (red) and bus (green) trajectories on a map of Aracaju

The green lines represent buses, while the red ones show taxi vehicles. As expected, the taxi trajectories cover a wider area than the buses, which are usually limited to specific roads through the city. The visual exploration of a spatial dataset is the number one tool that should be used throughout the entire process as it could reveal the specificities in data behavior and point to either potential data problems or important features.

Data Cleaning

Not a single data science project can be done without extensive dataset preprocessing. Real data is noisy, error-prone, incomplete and holds a certain amount of uncertainty. Moreover, it also comes in structures which are not fitted to specific business needs.

Looking at our dataset, and keeping in mind the final goal that needs to be achieved, it is noticeable that there are anomalous trajectories that do not carry useful information for road discovery. The examples can be seen in the following pictures.

Examples of anomalous trajectories

The picture on the left gives an example of an anomaly that needs to be completely removed from the dataset. Such examples are a result of tracker inaccuracies over longer time periods and other known or unknown factors. As depicted, certain parts of the trajectory cross residential areas, which do not match real roads, and if kept for further processing they can only introduce a larger margin of error to the final solution. The picture on the right demonstrates anomalies that could be corrected. These are also caused by tracker inaccuracies but are localized to a single point or a few close points in a trajectory. If successfully repaired, such trajectories could further be used in road discovery and later routing.

Various techniques can be applied to anomaly discovery and correction. Some propose map matching prior to any further processing, meaning that the trajectories are first being matched to a real road network and then processed further. This could be a good approach as it would enable easier recognition of road segments and potential anomalies; however, when dealing with large datasets, it could be time-consuming and probably inefficient. Other approaches are either based on using Kalman or semantic filtering.

The combination of semantic and Kalman filtering gave the best results for our dataset. The heuristic for detecting anomalies was considered to be focused either on distance or speed. For every two consecutive points in a trajectory, the distance between the points and the average speed were calculated. The goal was to identify larger scale anomalies, so, the threshold for distance was set to 300 meters and for speed to 120 km/h. Once again, better results, in terms of the number of true anomalies detected, were achieved when considering only distance. A trajectory was completely discarded from the dataset if there was any segment with the distance larger than the threshold. The following are some of the discarded anomalous trajectories.

Examples of discarded anomalous trajectories

Although, the first thought would be to discard the entire trajectory, splitting it into a number of sub-trajectories based on the detected anomalies should definitely be considered. This could help preserve parts of the trajectory which are valid and useful for further road discovery. For the purpose of this experiment, the decision was made to simply discard them entirely.

The next step was to detect complex trajectories, which are those that contain cycles. The approach undertaken here was to try to correct such trajectories by simply removing cycles without breaking their continuity. Those that could not be repaired were completely discarded. While this step helped detect and remove/repair anomalies that were not detected with previous semantic-based filtering, it also resulted in losing some of the valid trajectories. The lost ones were those going through city blocks and making valid cycles. A bit of a trade-off was made here because more was gained than lost with this approach, and the decision was reached that for the purpose of experimenting this step could be kept. However, when it comes to applying this approach to a real project, this step would either had to be revised or moved to a later phase where it could be applied to known segments, rather than the entire trajectory. Here are some of the examples of the detected and the removed complexities:

Examples of removed complex trajectories

And here is an example of the lost valid trajectories.

Example of lost valid trajectories falsely considered complex

The last cleaning step was the smoothing of our trajectories by applying the Kalman smoothing algorithm. This approach helped detect and smooth out the localized peaks in trajectories, as depicted in the following images.