# Big Data for Traffic Estimation and Prediction: A Survey of Data and Tools
## 4 Challenges and Future Directions  
While there are many successful application cases of big data for traffic estimation and prediction tasks, challenges still exist. For now, the available data sources presented in this study are still limited and can be used in statistical production only in a very limited way. There are still severe difficulties in the data generating process, which requires joint cooperation among academia, industry and government. Legislation is another matter to consider, since many companies do not want to share and would rather sell the data or use their monopoly, e.g., with mobile data in their countries.  
Data richness varies greatly for different transportation modes and the problems of data sparsity, high missing data ratios, data noise, and lack of data still exist. Data quality, privacy, and policies have not been fully considered in previous studies. For example, crowd-sourced data have the problems of low data quality, noise removal difficulty, and privacy concerns. Some efforts have been made to address these challenges, e.g., sparse Bayesian learning was used for traffic state estimation with under-sampled data [99]. Existing data for traffic estimation and prediction tasks are heterogeneous in the spatial and temporal ranges. Cross-scale data fusion by integrating various sources is still challenging in the transportation domain.  
Another challenge is the lack of \"real\" big data in the transportation domain, especially open data. While we reviewed many open data sources in this study, some of them have a data volume that is difficult to define as \"big\". The time range of some available datasets is not long enough for training effective deep-learning models. This challenge is partially caused by the high-cost and time-consuming process of collecting some types of data.  
\begin{table}
\begin{tabular}{p{85.4 pt} p{113.8 pt} p{113.8 pt} p{113.8 pt}} \hline \hline
**Tool** & **Main Purpose** & **Advantage** & **Disadvantage** \\ \hline Apache Hadoop & Distributed computing & Mature and reliable. & Difficult to use. \\ \hline Apache Pig & Distributed computing & High-level interface. & The need to learn Pig Latin language. \\ \hline Apache Maout & Distributed machine learning & Support for machine learning algorithms. & Performance bottleneck of default models. \\ \hline Apache Spark & Distributed computing & In-memory cluster computing. Easy to use. & No automatic optimization process. \\ \hline Apache Kafka & Distributed messaging & Low latency. High throughput. & Reduced performance. \\ \hline Apache Flink & Distributed computing & High efficiency. Easy to use. & Immature and lack of API support. \\ \hline Apache Storm & Distributed computing & Fast and fault tolerant. & Difficult to learn and use. \\ \hline Apache HDFS & Database & Fault tolerant. Integrated with Apache Hadoop. & Difficult to use. \\ \hline Apache HBase & Database & Real-time querying. Suitable for sparse data. Low-latency operation. & No SQL-like interface. \\ \hline MongoDB & Database & Document-oriented database. & Transactions are not supported. Limited data size. High memory usage. \\ \hline Apache Hive & Database & SQL-like interface. & Not a full database. No real-time querying. \\ \hline \hline \end{tabular}
\end{table}
Table 10: A comparison of big data tools.  
The other reason is the concern of location privacy leakage, which prohibits the collection of fine-grained data. For existing big datasets, e.g., GPS trajectories, the existing big data tools discussed in Section 3 are not yet fully exploited. With the popularity of using graph data for traffic prediction, e.g., transportation networks in graph neural networks, the existing graph processing tools cannot fully meet the computing requirements; thus, new tools are needed.  
To address these challenges, some future directions are pointed out in this section.