A distributed approach on early trending topics prediction on social network services

It is indisputable that we have entered the Era of Big Data. Nowadays, the explosion and profusion of available data in a wide range of application domains is phenomenal. According to IBM, 2.5 billion GB of data are created every day. Ninety percent of the currently existing data, has been created in the past two years and the amount of data is expected to increase exponentially. One of the key factors that lead to the boost of data volume is the widespread adoption of Web 2.0 tools and technologies . Web 2.0 applications may allow users to interact and collaborate with each other in a social media dialogue, as creators of user-generated content in a virtual community. Among the applications of Web 2.0, Social Media are of particular interest mainly because they are extremely popular among Web users. Users of social media, through their vigorous activity, create massive volumes of data. This growing data collection, has attracted the interest of both the research community and industry, as performing mining and analysis on social media data may reveals several useful conclusions. The process of extracting and analyzing data from social networks is described by the term “Social Media Analytics”. This process which have various aspects, is in general particularly tedious as the data generated from social networks are characterized by very large volume and lack of structure. This thesis is deepen into the area of “Social Network Analytics”. In particular, the problem of predicting, in real time, of popular topics (treding topics), appearing in social media is going to be studied. The popular topics of social networks are, by their nature, of utmost importance as they give immediate information about what individuals and the society are interested in, at a specific time period. Furthermore, popular topics, through the high visibility they obtain could affect public opinion, about what is considered trend or even their views on various social issues. In this thesis the problem of popular topics prediction is approached from the perspective of time series real time classification. These time series represent the occurrence frequency of the under study topics in the messages published by social media users, in the axis of time. The time series classification is implemented through a supervised learning method based on a latent source model. In order to perform and evaluate the aforementioned classification method, a distributed framework is going to be implemented so as to predict real time trends in Twitter. The main contribution of this thesis is that trend prediction is performed in real time fashion and thus several tools and techniques of the field of Big Data are used. In particular, the architecture of the implemented framework is inspired of the so called Lambda architecture. The main characteristic of Lambda architecture is that it enables stream and batch analysis to be performed simultaneously in the a common dataset. In this experimental scenario stream analysis is used in order to create, track and classify the topics’ time series. Whereas the batch analysis is used so as to create as well as to perform incremental improvement of the initial training set.