GitHub PullRequestReviewCommentEvent at 2014-02-19 17:16:02
Repository: euroscipy_proceedings
Actor: NelleV
Payload: {"comment":{"url":"https://api.github.com/repos/euroscipy/euroscipy_proceedings/pulls/comments/9873305","id":9873305,"diff_hunk":"@@ -0,0 +1,774 @@\n+:author: Antònia Tugores\n+:email: antonia@ifisc.uib-csic.es\n+:institution: Institute for Cross-Disciplinary Physics and Complex Systems, IFISC (UIB-CSIC)\n+:author: Pere Colet\n+:email: pere@ifisc.uib-csic.es\n+:institution: Institute for Cross-Disciplinary Physics and Complex Systems, IFISC (UIB-CSIC)\n+\n+\n+-----------------------------------------------------------------\n+Mining online social networks with Python to study urban mobility\n+-----------------------------------------------------------------\n+\n+.. class:: abstract\n+\n+On-line  social networks have grown quickly over the last few years and\n+nowadays many people use them frequently.  Furthermore the emergence of\n+smartphones allows to access these networks any time from any physical location.\n+Among the social networks, Twitter offers a particularly large set of data publicly available.\n+Here we discuss the procedure to mine this data and store it in\n+distributed databases using Python scripts. We also illustrate how geolocated tweets can be used\n+to study the mobility of people in urban areas.\n+\n+\n+.. class:: keywords\n+\n+big data, noSQL, data acquisition, online social networks\n+\n+\n+Introduction\n+============\n+\n+Although online social networks appeared already in the later nineties, their growth\n+has taken place mainly in the last decade, in parallel to that of devices providing\n+mobile internet connections. Out of these online social networks, some of them\n+consider the data to be private while others emphasize in distributing the\n+information publicly. Out of the last ones, Twitter is, by far, the larger.\n+Tweets, intended as a compact way to share an opinion to the world, also contain\n+several kinds of information on the user and way it has been send. In particular,\n+it may include the geolocation at the time it is send. This information can be used\n+to study the mobility behaviour of people in urban areas, including attitudes and\n+lifestyle, which are particularly important for e.g. developing demand management\n+concepts for influencing mobility decisions.\n+\n+The objective of this paper is to discuss efficient ways to retrieve and store\n+large amounts of data from social networks, such as Twitter. Since the fraction\n+of tweets that can be downloaded from Twitter is limited, besides a random sample,\n+we select the users which are located in the cities under study. We use Python\n+scripts to interact with Twitter API, in particular with Tweepy, and to decide\n+which are the users to be selected. To improve efficiency data is stored in a\n+distributed way using MongoDB noSQL data base. Finally we perform searches on the\n+database using the Python MongoDB driver to extract the relevant information to\n+study human mobility patterns. The paper is organized as follows, we first provide\n+a brief discussion on SQL and noSQL databases. The next section explains how data\n+is acquired and how users are selected. Then we compare the performance of a SQL\n+database with an distributed noSQL one. Finally, some preliminary mobility\n+results are shown and we give some concluding remarks.\n+\n+\n+Data storage\n+============\n+\n+Storage of data in plain files is not practical when dealing with thousands of\n+millions of tweets, even if each document is as small as a tweet. It is much more\n+convenient to store them in a database, which also allows for multi user-access\n+to the information. High availability, performance, scalability and recoverability,\n+together with good drivers, easy administration and preferably open source are\n+some of the desirable characteristics of a database to store large amounts of data\n+for a long period of time.\n+\n+\n+SQL vs noSQL\n+------------\n+\n+Traditional databases rely on collections of tables with data records, which\n+are formally described and organized according to a relational model. This is known\n+as Relational Database Management System, RMDBS) as introduced by E.F. Codd [Codd]_.\n+Relational databases are widely adopted and tested. As a consequence, Atomicity,\n+Consistency, Isolation and Durability (also known as ACID [ACID]_) is supported\n+as well as crash recovery, and master-slave consistency, which explains why they have been actively used in production since\n+the seventies. There are different tools to access relational databases but,\n+by far Structured Query Language (SQL) is the most common.\n+Therefore the term SQL is in many instances used as synonym to relational database.\n+On a relational database, data can not be inserted in an arbitrary way. One has\n+first to parse the data prior to insertion, which for fast data archival may\n+constitute a bottleneck. Typically a SQL database is stored in a pair of master-slave\n+servers. Since insertion and queries are performed on the same computer, performance\n+depends on the hardware of this computer and scalability involves improving\n+the existing hardware.\n+\n+On the contrary, noSQL databases make use of less constrained consistency models\n+to gain simplicity of design, horizontal scalability and finer control over\n+availability. NoSQL databases are schema free and are capable of distributing\n+data among different computers. Insert and search throughput is related to the\n+number of computers in which the data is stored.\n+This gives them the ability to horizontally scale up, while relying on relatively\n+simple APIs. Therefore scalability is achieved by adding computers to the cluster\n+rather than improving the existing hardware.\n+Crash recovery and high availability are obtained by data replication, a configuration\n+similar to the well-known master-slave SQL configuration. Moreover, queries can be\n+distributed among the different computers that hold the data in order to\n+improve the performance.  All these make noSQL database management systems quite\n+useful when working with large\n+datasets or when the nature of the data does not require a relational model.\n+\n+However, ACID which is guaranteed in SQL database transactions, is typically not\n+directly provided by noSQL databases. In particular, consistency is guaranteed\n+only after a period of time sufficient to propagate the changes through the\n+system. Thus, the management of noSQL databases is usually quite more cumbersome\n+than that of SQL ones.\n+\n+Finally we would like to note than in order to improve the search performance on\n+both relational and non relational databases, it is important to index the data\n+fields which are used more frequently for classification.\n+\n+\n+\n+\n+Documents format and storage\n+----------------------------\n+\n+Typically we download more than 15 million statuses (tweets) per day, and this\n+number grows over the time. However, the volume of tweets is not constant in time.\n+Political or social singular events can cause traffic spikes, thus the database must\n+have the ability to handle large and steadily increasing volume of data while\n+providing enough flexibility to deal with unexpected traffic peaks. In addition,\n+the format of the information contained in tweets can, and does, change over the time.\n+\n+The data we retrieve from Twitter is either encoded as JavaScript Object Notation\n+(JSON), [JSON]_ or it can be easily\n+converted to JSON format by using jsonpickle package [jsonp]_. JSON format\n+is used to transmit human readable data through a network connection as a\n+alternative to XML. The attributes of JSON encoded objects are unsorted and the\n+format of the attributes is not fixed. JSON flexibility allows for new fields to\n+be added or old fields to be\n+removed, and also to change the format of the fields (integer, string, datetime, ... ).\n+\n+Scalability and read/write performance drive us to consider noSQL databases to store documents.\n+CouchDB [CouchDB]_, and MongoDB [MongoDB]_, have been considered in the study.\n+Both databases are quite similar. On both, replication and failover security is\n+achieved by replicating data on different servers. The advantages of\n+CouchDB include that it accepts JSON data, ACID performance and allowing the\n+use of map/reduce operations for query paralelization. However, queries have to\n+be predefined in advance and users can only perform predefined queries or\n+combinations of them. This is an inconvenient when the users of the database have\n+different needs and objectives. Besides the language for the queries is quite\n+specific and different from standard SQL commands.\n+\n+On the other hand, MongoDB also accepts JSON data which is internally stored as\n+a BSON [BSON]_ object, which is a JSON-like document in a binary format. MongoDB\n+does not require the queries to be predefined and furthermore it supports SQL-like\n+commands including aggregation of the results. Besides it also supports map/reduce\n+queries. Furthermore it provides a large variety of indexes including geo indices,\n+which is particular relevant for the kind of data queries we are interested in\n+since it allows queries to be executed in real time. Even that MongoDB lacks real\n+ACID transactions, it is more suitable when using large amounts of data. Horizontal\n+scalability is much clear in MongoDB and the possibility to allow users to run\n+their own queries without requiring predefinitions made MongoDB more suitable to our\n+needs. In what follows the only noSQL database that is considered is MongoDB.\n+\n+\n+MongoDB configuration\n+---------------------\n+\n+MongoDB minimal configuration involves two computers, one server with all the data and\n+one client to which the user application connects to. This minimal configuration does not ensure failover recovery.\n+To ensure this, one needs additional computers forming a cluster. This cluster is\n+called Replica Set and these groups consist of a minimum of three computers one\n+of them designated as the primary and the others as secondaries. To ensure automatic\n+failover recovery, the members of the replica set run a daemon that replicates the data.\n+The primary member receives all the write/insert connections while secondaries replicate from the primary asynchronously\n+with a delay of milliseconds and can receive read orders. Even that data replication\n+uses much more space that the one really needed, it ensures high availability and increases read capacity. Apart from that,\n+a good practice, is to configure one of the secondary computers of each replica\n+set with a predefined replication delay time and use it as backup. The standard\n+MongoDB configuration hides the backup computer from clients so that it can not\n+be used for searches to prevent searching in a non up to date data set.\n+\n+The configuration with a single Replica Set is suitable when one single computer can store all\n+the data and the read/write performance using a single computer is enough.\n+In MongoDB the way to scale up the database is sharding: the collection of data is\n+partitioned by using a key defined by the database administrator\n+and the different chunks or portions are stored in different replica sets or shards.\n+Sharding automatically balances data and load across the shards and increases write\n+and read capacity. In addition to that, it provides a clear pathway to grow.\n+When a database collection becomes too large for the existing configuration, a\n+new shard (horizontal scalability) can be added and MongoDB\n+automatically distributes collection data to the new group of servers.\n+\n+\n+.. figure:: mongo_struct.eps\n+\n+   Schematic MongoDB configuration. :label:`fig:config`\n+\n+In addition to shards or replica sets, in a sharded cluster there are\n+configuration servers (CS) that store metadata relating replica sets with data portions and\n+that route reads and writes from mongo clients (CL) to the appropriate replica set. Notice\n+that client applications connect to mongo client (CL) which returns the answer\n+to the queries. The structure behind is hidden to the client.\n+\n+In our case (Fig.  :ref:`fig:config`), we configured a sharded cluster with\n+six replica sets formed by three members each. As recommended in production environments,\n+we are currently using three configuration servers.\n+Each of the replica sets has two eligible primary members and the third one\n+is a delayed copy by 72 hours. This gives us failover security because\n+if primary server daemon crashes or stops, the secondary one becomes primary.\n+The third member helps us to recover from human errors\n+such as inadvertently deleted databases or botched application upgrades.\n+Finally, the shard key used\n+is the tweet identifier and we added indices by user\n+identifier and latitude/longitude to speed up usual queries.\n+\n+To improve writing performance we took into account several MongoDB\n+features when customizing the operating system in the servers that form\n+the replica sets.\n+\n+\n+Data acquisition\n+================\n+\n+Even that Twitter provides mechanisms to retrieve only a small fraction of total\n+amount of tweets (about 1\\% randomly distributed), this constitutes a large amount\n+of data distributed all over the world. About 12\\% of the retrieved tweets have\n+geolocated data. And only a small part of these are located in the cities we are\n+focusing in, such as London or Bacelona. As a consequence, the number of tweets\n+that can be used to study the human mobility in these cities is limited. To solve\n+this issue, we select a set of users from the random sample which have tweets\n+geolocated in the metropolitan areas and we download their timeline.\n+\n+\n+Twitter APIs\n+------------\n+\n+Twitter data access can be achieved through two ad hoc APIs that represent\n+different Twitter features: Stream and Representational State Transfer\n+(REST). The '*Stream API*' is focused in data mining providing the real time sample\n+of the tweets. The '*REST API*' enables\n+developers to access some of the core primitives of Twitter including timelines,\n+status updates, and user information.\n+\n+Although possible, directly access the Twitter APIs is not trivial. Therefore it\n+is recommended to use a library [Twilib]_. There are libraries for many computer\n+languages although here we focus on Python. The code readability, the smooth\n+learning curve, the quick development or the dynamic typing makes Python a suitable\n+language to be used by\n+software engineers and scientists. Among the libraries for Python we use the\n+*tweepy* [Tweepy]_, for its simplicity and flexibility. Besides with a package\n+we can access both Stream and REST APIs. Furthermore, it is open source (MIT License).\n+\n+The data we receive from streaming is JSON encoded while the data we gather\n+from other APIs is converted to JSON format by using jsonpickle package [jsonp]_.\n+\n+\n+Random sample\n+-------------\n+\n+Connecting to the streaming API requires having a Twitter account and keeping open\n+a persistent HTTP connection to one of the public endpoints. Streaming API do\n+not support requests. Randomly sampled tweets are provided automatically\n+(subject to the limitation of about 1\\% of the total tweets). This API limits\n+each account to just one standing connection. In fact, connecting to a public\n+stream more than once with the same credentials causes the oldest connection to\n+be disconnected. Besides, IPs of clients that make excessive connection attempts\n+run the risk of being automatically banned.\n+\n+As the process that opens the connection receives raw tweets, it has to perform\n+all parsing, filtering, and aggregation needed before storing the result.\n+\n+A particularity of the *Streaming API* is that messages are not\n+delivered in the same precise order as they were generated. In\n+particular, messages can be slightly shifted in time and it is also\n+possible that deleted messages are received before the original tweet.\n+This is not critical for the case considered here because we are\n+interested in slower time scales (from minutes to hours) and therefore\n+we do not need to have an exact timing and order of the messages.\n+\n+In our case, connections to Twitter API are achieved by using *tweepy*.\n+It allows the implementation of a listener that activates when a tweet arrives\n+and so that it can be processed. Prior to the connection to the API it is\n+necessary to provide the user\n+authentication and to instantiate the listener.\n+\n+A example of the code to simply print the tweets to the standard output is:\n+\n+|\n+\n+.. code-block:: python\n+\n+  from tweepy import Stream, OAuthHandler\n+  from tweepy.streaming import StreamListener\n+\n+  class BasicListener(StreamListener):\n+    \"\"\"\n+      A listener handles tweets are the received from\n+      the stream.\n+    \"\"\"\n+    def on_data(self, data):\n+        # print received tweet to stdout\n+        print data\n+        return True\n+\n+    def on_error(self, status):\n+        # print error when data is not correctly\n+        # received\n+        print \"Error: \" + status\n+\n+  if __name__ == '__main__':\n+\n+    # authentication\n+    auth = OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)\n+    auth.set_access_token(ACCESS_KEY, ACCESS_SECRET)\n+\n+    # listener instance\n+    listen = BasicListener()\n+\n+    # open connection\n+    stream = Stream(auth, listen, gzip=True)\n+\n+    # start receiving data\n+    stream.sample()\n+\n+\n+In our case, we store the tweets we download in the same format they are received, JSON. To make data\n+search and manipulation easier we use the tweet id as one of the indices of the\n+database and it is necessary to have it in the highest level of the document.\n+\n+Deleted tweets have a different structure than regular ones. In particular they are identified by the key, *deleted*,\n+in the highest document level. The only contain information on the user and the tweetID of the erased tweet. We also store these tweets.","path":"papers/antonia_tugores/tugores.rst","position":327,"original_position":327,"commit_id":"8a06c6ff9601c3af95d0dab4b960c570c6872fa3","original_commit_id":"8a06c6ff9601c3af95d0dab4b960c570c6872fa3","user":{"login":"NelleV","id":184798,"avatar_url":"https://gravatar.com/avatar/18f719a71cd957dc6f79655448cacad1?d=https%3A%2F%2Fidenticons.github.com%2Fd6ed59be986bf08d804dffa1fc99669c.png&r=x","gravatar_id":"18f719a71cd957dc6f79655448cacad1","url":"https://api.github.com/users/NelleV","html_url":"https://github.com/NelleV","followers_url":"https://api.github.com/users/NelleV/followers","following_url":"https://api.github.com/users/NelleV/following{/other_user}","gists_url":"https://api.github.com/users/NelleV/gists{/gist_id}","starred_url":"https://api.github.com/users/NelleV/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/NelleV/subscriptions","organizations_url":"https://api.github.com/users/NelleV/orgs","repos_url":"https://api.github.com/users/NelleV/repos","events_url":"https://api.github.com/users/NelleV/events{/privacy}","received_events_url":"https://api.github.com/users/NelleV/received_events","type":"User","site_admin":false},"body":"«The only contain» => «They only contain»","created_at":"2014-02-19T17:16:02Z","updated_at":"2014-02-19T17:16:02Z","html_url":"https://github.com/euroscipy/euroscipy_proceedings/pull/13#discussion_r9873305","pull_request_url":"https://api.github.com/repos/euroscipy/euroscipy_proceedings/pulls/13","_links":{"self":{"href":"https://api.github.com/repos/euroscipy/euroscipy_proceedings/pulls/comments/9873305"},"html":{"href":"https://github.com/euroscipy/euroscipy_proceedings/pull/13#discussion_r9873305"},"pull_request":{"href":"https://api.github.com/repos/euroscipy/euroscipy_proceedings/pulls/13"}}}}