Social media, particularly Twitter, is increasingly used to improve resilience during extreme weather events/emergency management situations, including f loods: by communicating potential risks and their impacts, and informing agencies and responder s. In this paper, we developed a prototype national-scale Twitter data mining pipeline for imp roved stakeholder situational awareness during flooding events across Great Britain, by retrieving relevant social geodata, grounded in environmental data sources (flood warnings and river levels). Wit h potential users we identified and addressed three research questions to develop this application, who se components constitute a modular architecture for real-time dashboards. First, polling national f lood warning and river level Web data sources to obtain at-risk locations. Secondly, real-time retri eval of geotagged tweets, proximate to at-risk area s. Thirdly, filtering flood-relevant tweets with natur al language processing and machine learning libraries, using word embeddings of tweets. We demo nstrated the national-scale social geodata pipeline using over 420,000 georeferenced tweets ob tained between 20-29th June 2016. Highlights • Prototype real-time social geodata pipeline for flo od events and demonstration dataset • National-scale flood warnings/river levels set ‘at- risk areas’ in Twitter API queries • Monitoring multiple locations (without keywords) re trieved current, geotagged tweets • Novel application of word embeddings in flooding co ntext identified relevant tweets • Pipeline extracts tweets to visualise using open-so urce libraries (SciKit Learn/Gensim) Keywords Flood management; Twitter; volunteered geographic i nformation; natural language processing; word embeddings; social geodata. MANUSCRIPT ACCEPTED ACCEPTED MANUSCRIPT2 Software/data availability Name of software: Various scripts as they apply to pipeline steps. Description: all scripts for various steps of the p ipeline are described and specified in supplied Appendix A1.2.1, Table A.2. Developer: J. L. P. Barker Contact: lukebarker@gmail.com Year first available: 2017 Hardware required: Intel i3 or mid-performance PC w ith multicore processor and SSD main drive, 8Gb memory recommended. Software required: Python and library dependencies specified in Appendix A1.2.1, (viii) environment.yml Software availability: All source code can be found at GitHub public repositories https://github.com/battez/analysis/releases/tag/v0. 5-alpha ; https://github.com/battez/tweepy_now and https://github.com/battez/twystream/releases/tag/v0 .5-beta . Configuration file for python environment, with all necessary dependencies: https://drive.google.com/open?id=0B057bbdoYJDLYjlIO UFLMkJ0TDg Cost: Free. Software and source code are released u nder the New Berkeley Software Distribution (BSD) License, which allows for liberal reuse of th e software and code. Name of data set: 1 - Unlabelled Tweets Development Dataset Description: Unlabelled June dataset 420,218 geotag ged tweets and identifiers, in CSV format: flat file of Tweet identifiers for lookup via the Twitte r API. Or can search individually via Twitter website: e.g. https://twitter.com/statuses/74590764 4641193984 Developer: J. L. P. Barker Availability: Free, download from: https://drive.google.com/open?id=0B057bbdoYJDLT0REZ FFWNDQyMXc Name of data set: 2 - Labelled Tweets used in train ing classifier. Description: Labelled subset of June dataset 4502 t weet identifiers in CSV format: (Note: t_class=1 Relevant class): Developer: J. L. P. Barker Availability: Free, download from: https://drive.google.com/open?id=0B057bbdoYJDLVjg1Y Td5d0JsdkU 1. Introduction MANUSCRIPT ACCEPTED ACCEPTED MANUSCRIPT3 Social media is increasingly being used during natu ral catastrophes and emergency management situations, including flooding events, to improve t wo-way communication and understanding of potential risks and their impacts (Kryvasheyeu et a l, 2016). Stakeholders can inspect actionable information on real-time dashboards that integrate different sources and provide an information channel for communities (Mukkamala and Beck, 2016). Effective data mining of social media to improve crisis response is sought by the humanitari an sector (Middleton et al, 2014) but the volume of data proves difficult to identify relevant informat ion for decision-making (Vieweg et al, 2014). In part, this is due to dealing with large amounts of data with varying quality/information content (see Meier 2015 for a summary). Where spatial size and s ignificance of natural catastrophes can influence the quality of social geodata (Middleton et al, 201 4). Employing software (including machine learning) to alleviate information overload or data reliability issues is crucial to increasing adopti on by managers, and sensemaking requires geotagging an d map visualisations (Rao et al, 2017; Li et al, 2017). Indeed, stakeholders want place-based, geota gged, crowd-sourced or “volunteered geographic information” (VGI) about flooding events (Juhász et al, 2016). There is a need to automate identification and extr action of spatial information about flood events from social media e.g. tweets (Smith et al, 2015; E ilander et al, 2016; Arthur et al, 2018). Accuratel y identifying localised flood extent is difficult, es pecially during pluvial and fluvial flooding in urb an areas, and social media can help with gathering and disseminating information (Smith et al, 2015; de Albuquerque et al, 2015; Middleton et al, 2014; Her fort et al, 2014). Internationally, increases in intense localised convective rainfall events have b een observed (e.g. Zheng et al, 2015). Recently, extraction of pluvial flood relevant VGI by deep le arning from user generated texts and photos has been demonstrated (Feng and Sester 2018). The chara cterisation of catchment response during local flash flood events were shown to particularly benef it from combined authoritative sources and community data (VGI and citizen science) (Starkey e t al, 2017). Analysis of social geodata during floods in Sao Paulo, Brazil showed geographical rel ations were useful for identifying tweets relevant to flooding (de Assis et al, 2016). Other flood rel ated examples include: georeferenced tweets aiding real-time flood extent observations and agency resp onse in Jakarta, Indonesia (Eilander et al, 2016), and during floods on the River Elbe, in Germany (He rfort et al, 2014). Generally, research has focussed on using social me dia data in a stand-alone manner for specific events and locations, rarely attempting integration with real-time environmental sources - which are continually improving and becoming accessible via p ublic application programming interfaces (APIs) (Table A.1). One exception was the integration of e nvironmental sensor data with catchment polygons to prioritise tweets in Sao Paulo (de Assis et al, 2016). Existing web applications- that harness soci al media during emergency events- using machine learni ng techniques have generally been developed MANUSCRIPT ACCEPTED ACCEPTED MANUSCRIPT4 for specific incidents to be searched for - typical ly, one-off catchment to regional scale situations (Spielhofer et al, 2016) - rather than supporting n ational level surveillance monitoring of natural hazard events and their social impacts, e.g. only a t a city-scale (Eilander et al, 2016). In this paper, we demonstrate and test a prototype machine learning social geodata pipeline, based on real-time flood warnings and river level informatio n, and natural language processing of tweets during flooding events, with the aim to improve situationa l awareness for flood risk managers and other stakeholders. This involved novel automated selecti on and analysis of large volumes of geotagged and relevant social media data, and recent advances in vector-based natural language processing (NLP). Developing this prototype application required thre e research questions to be addressed. 1) How to programmatically identify at a national-scal e (across Great Britain) areas at-risk of flooding based on real-time flood warnings and rive r level information ? The first research question was how to identify tim ely shortlists of locations at-risk of flooding using available national-scale, real-time flood warnings (England/Wales) and river level (Scotland) information sources (from hundreds of potential locations). Previous studies highlighted that sensor data (e.g. river level gaug es) could support prioritising social media messages during flood events (de Albuquerque et al, 2015). Internationally, there are an increasing number of near real-time environmental d ata sources, providing information on at- risk areas, river levels and rainfall (Table A.1). 2) How to automate the spatiotemporal retrieval of rea l-time social geodata using Twitter APIs? The second research question was to investigate the viability of automated retrieval of social geodata (i.e. tweets geotagged by their author) in a timely and continuous way, based on multiple locations prioritised by potential risk of flooding. Previous studies researched cross- referencing tweets with prioritised locations for m apping flooding (Middleton et al, 2014), and location-based queries during floods, using geo referencing tweets (Laylavi et al, 2016). A recent application, GeoViewer, queried a location ( as defined by a user), via the Twitter Streaming API, to output maps of tweets in real-tim e (Tsou et al, 2017). 3) How to automatically identify flood-relevant tweets ? The third research question was whether we could ac hieve automatic identification of individual tweets for flooding relevance. Herfort e t al. (2014) noted such an automated task was difficult to achieve. We defined “relevance” as social geodata that contributed to situational awareness for managing in-progress floo ding events. Previous studies in crisis contexts generally used initial keyword filtering, with subsequent georeferencing: in flooding MANUSCRIPT ACCEPTED ACCEPTED MANUSCRIPT5 events ( FloodTags app1 of Jongman et al, 2015), hurricanes (Kryvasheyeu e t al, 2016) and landslides (Musaev and Hou, 2016). Our research aim was to develop and test a prototyp e integrated real-time solution based on these three research questions for stakeholder map-based dashboard visualisations, during a series of flooding events across England and Scotland between June 20-29th, 2016; that was also applicable to other countries and a range of weathe r-driven emergency situations. 2. Methods Our prototype was partly inspired by several discus sions with SEPA flood risk managers (April-June 2016), during these discussions we revised the aim of our research to demonstrate a prototype pipeline linking real-time river level information and socia l media to aid national level flood management. During a visit to the SEPA flood forecasting centre (Perth, Scotland), we were shown flood risk dashboards and learned of their usefulness for two- way communication about floods via community flood observation reporting tools (Report-a-Flood a nd Floodline2). They said there was a need to automate the use of social media, especially tweets , in real-time map-based dashboards to better understand flooding situations across Scotland. The se discussions along with similar needs identified in recent literature (see introduction) provided co nfidence in the potential usefulness of our approac h and enabled refinement of our three research questi ons. To achieve our aim and answer these three research questions, we developed a prototype pipeline (Section 2.1.1, Fig.1 and Table 1), and tested its operation during a series of flooding events across England and Scotland, between June 20-29th, 2016 (S ection 2.1.2). In Section 2.2, we present a systematic description of the pipeline steps and ma jor design options and decisions. We accessed the online data sources (APIs) of two British environme ntal agencies: the Scottish Environmental Protection Agency (SEPA) and the Environment Agency (EA). Similar data sources exist for other countries (Table A.1), and other kinds of weather e vents e.g. wildfires, hurricanes, snowstorms in North America. 2.1 Pipeline structure and its application during a ctual flooding 2.1.1 Pipeline structure and summary of steps 1 https://www.floodtags.com Last accessed: 20th Dec. 2017. 2 https://www.sepa.org.uk/environment/water/flooding / Last accessed 30th May 2018. MANUSCRIPT ACCEPTED ACCEPTED MANUSCRIPT6 Fig. 1. Steps in the pipeline application architecture for retrieval of social geodata during flooding: White background indicates steps for spatiotemporal data retrieval ( 1-5); Grey background summarises filtering steps to predict r elevant tweet data to output/visualisation (6-8). Table 1 Details of the prototype pipeline steps and the res earch questions addressed by each step. Research question Step Description 1) How to programmatically identify at a national- scale (across Great Britain) areas at-risk of flooding, based on real- time flood warnings and river level information ? 1 Reset retrieval task at regular periods, i.e. eve ry three hours a script ran to update the latest at-risk areas from online sources (See Section 2.2.1 for details of period duration). 2 Obtained at-risk areas from national-scale enviro nment data sources, namely via SEPA river gauge levels for Sco tland and EA flood warnings. SEPA flat files were polled to retr ieve all the latest river levels; these were compared to their average level for an indication of flood risk associated with each gauge ’s surrounding area. The top ranked EA flood warning and alert are as of risk were extracted from results returned by the EA API3 (see Sections 2.2.1 and 3.2 for details). 3 Prioritised at-risk areas were collated to a comb ined shortlist, up to a maximum of 25, as per Twitter Streaming API limit s. Areas were 3 https://environment.data.gov.uk/flood-monitoring/do c/reference#flood-warnings Last accessed: 30th May 2018. MANUSCRIPT ACCEPTED ACCEPTED MANUSCRIPT7 converted to bounding boxes, as required by this AP I. 2) How to automate the spatiotemporal retrieval of real-time social geodata using Twitter APIs? 4 A Streaming API location filter query monitored g eotagged tweets with the at-risk shortlist from Step 3. (See Table A.2. (i). Queried areas were updated throughout: see log Table A.2, (vi)). 5 Matching social geodata were returned in real-tim e, totalling 420,218 tweets for 20-29th June 2016. Tweets matche d when their geotag intersected an at-risk area in the location filter at the time the tweet was posted. Tweets returned by the API we re archived to a MongoDB NoSQL database4, which formed a development dataset. The Streaming API uses a heuristic to dete rmine whether a given tweet falls within a bounding box: ● If the Coordinates field is populated, the values t here will be tested against the bounding box. ● If Coordinates is empty but Place is populated, the region defined in Place is checked for intersection agains t the location’s bounding box. Any overlap will match5. More detail on this process is provided in Appendix 1.2. 3. 3) How to automatically identify flood-relevant tweets? 6 The tweet text was pre-processed, providing input s to a Paragraph Vector model (via Gensim “Doc2Vec” implementation) which embedded the tweets and words as numeric feature ve ctors. This model could convert (or infer) incoming, unseen twe ets as document vectors, for this step to pass to the next step. 7 Using a tweet’s vector, identified if it was of r elevance. The classifier was developed by an annotator manually l abelled a subset of 4,502 tweets from June 23rd as either relevant or irrelevant . These were split (randomly, and at 9:1 proportion ) into training and test sets to fit a SciKit Learn Logist ic Regression classifier. The class label for new, unseen, inferr ed tweet vectors could then be predicted by the model. 8 This step outputted relevant tweets to a dashboar d, for viewing and exploration by potential stakeholders. We prototype d several visualisations of social geodata on maps at their c oordinate 4 https://www.mongodb.com/what-is-mongodb Last access ed: 20th Dec. 2017. 5 https://developer.twitter.com/en/docs/tweets/filter -realtime/guides/basic-stream-parameters.html#locat ions Last accessed: 30th May 2018. MANUSCRIPT ACCEPTED ACCEPTED MANUSCRIPT8 locations (see Figs. 3, 4 and 5). 2.1.2 Applying and testing pipeline during national scale flooding events There were widespread flooding events during June 2 016, due to above average rainfall in Great Britain (147% of the 1971-2000 average), and above average river flows for most of England, Wales and North East Scotland6. The initial pipeline stages (Steps 1-5, see Fig. 1 and Table 1), were assembled and then tested in early June 2016 with f looding events across England and North East Scotland, and in northern and central France in lat e May 20167. Then, these same steps (i.e. Steps 1- 5), were applied nationally across Great Britain, r unning continuously, between 20-29th June 20168. Finally, Steps 6-8 were carried out using the data collected (Fig. 1 and Table 1). 2.2. Description of pipeline steps and major design options and decisions In this section we describe our research addressing the three research questions; we systematically describe a series of steps and major design options and decisions that enabled production of a prototype machine learning social g eodata pipeline, based on real-time flood warnings and river level information. 2.2.1 Steps 1-3: How to programmatically identify a reas across Great Britain at-risk of flooding based on real-time flood warnings and rive r level information? All steps (see Fig. 1 and Table 1) were controlled by a scheduled pipeline task: a shell script in Pyt hon that ran on a local computer server, the script was reset every three hours (Step 1) in order to updat e areas at-risk of flooding from the latest national, environmental data sources. A period of three hour s was chosen as an intended trade-off between trackin g the latest at-risk area forecasts (API updates varied between 15 minutes and several hours), and t o allow capturing reaction from those areas on Twitter. In a production application, this could be tuned to user requirements and to reflect the natu re of pluvial and fluvial flood events: optimising thi s was not within the scope of this study. Python wa s chosen as the server-side language since it is wide ly used on servers especially for data science applications; alternative options at the time inclu ded PHP and Ruby. Additionally, it provided the widest range of libraries to construct our pipeline , including those for accessing Twitter APIs and machine learning. This script, and all the project code, were version -controlled using Github- as Git is 6 http://www.hydoutuk.net/archive/july-2016/further- information-july-2016/ Last accessed: 27th Dec. 201 7. 7 https://www.ecologique-solidaire.gouv.fr/preventio n-des-inondations Last accessed: 25th Jan. 2018. 8 We consulted: http://www.metoffice.gov.uk/public/w eather/forecast and see month of June historic summ ary at http://www.metoffice.gov.uk/climate/uk/summaries /2016/june Both last accessed: 27th Dec. 2017. MANUSCRIPT ACCEPTED ACCEPTED MANUSCRIPT9 a widely used version control system, and Github is a popular repository hosting service for free and open source projects (see Table A.2). Our choice of APIs for real-time flood-related information was limited to those provided by SEPA in Scotland, and the EA in England and Wales. To establish national-scale coverage, the script jointly checked their respective APIs (see Appendix Table A.1): Scottish (i.e. SEPA) river level data feeds and Eng land & Wales (i.e. EA) flood risk areas (Step 2). a) the script first cycled through a set of SEPA web a ddresses for river gauge readings, parsing these flat files using Python web scraping code and obtained all the latest river levels (351 gauges in total). Then each reading was compared wi th its respective cached average level, so as to identify the highest relative levels present. The gauges for these levels then provided (by using their coordinates as a centre-point for 16 km wide square boxes) the locations at-risk across Scotland. This size was arbitrarily chosen t o cover the areas nearby to each gauge; future research could explore the influence of this value and ideally tailor it to the morphological characteristics of the rivers. b) the script then queried the EA’s public web API for its summary of latest flood risk warning and alert areas. These locations were ordered by risk severity, logg ed to a text file, and then converted to rectangula r bounding box coordinates (as per Twitter Streaming API requirements, Step 3). Further investigations would optimise this process, e.g. to establish the ideal spatial range for each box, but that was not the focus of our research. All the SEPA gauge locations were represented by simple polygon boxes in a spatial JavaScript Object Notation (JSON) format9. Generating these areas would be improved by accounting for e.g. topography and flood defences. The EA API flood areas10 vary in size and can comprise multiple catchments. These were converted from complex polygon shapes to simpler containing bounding boxes (see Appendix A1.2.2 and A1.2.3, Figs. A1 and A2). To generate a shortlist of locations for this task’ s duration i.e. three hours: we combined the 14 mos t at-risk SEPA areas with up to 10 at-risk EA areas ( warning levels 1-4)11. We divided the proportion of SEPA and EA areas arbitrarily (it was expedient to have more Scottish locations due to the use of river levels): future work would explore and improv e this proportion. The maximum of 24 at-risk areas was selected at each time point to respect th e Twitter API. The current at-risk areas were recalculated once the scheduled task was reset, Fig . 2 illustrates at-risk areas for the period (and F ig. 6 shows those from the 23rd June in Southeast England). See Figs A.1 and A.2 for further details of how 9 GeoJSON was used - an open standard format for geog raphic features. 10 https://environment.data.gov.uk/flood-monitoring/d oc/reference#flood-areas Last accessed: 20th Dec. 2 017. 11 For details of EA severity, please see https://env ironment.data.gov.uk/flood-monitoring/doc/reference Last accessed: 20th Dec. 2017. MANUSCRIPT ACCEPTED ACCEPTED MANUSCRIPT10 the bounding boxes for England and Wales encompass their associated at-risk flood area, as provided by the EA. 2.2.2 Steps 4-5: How to automate the spatiotemporal retrieval of real-time social geodata using Twitter APIs? To tackle the second research question required two main design choices: how to access the Twitter APIs, and what Twitter API to use. We chose the Pyt hon Twython library12, which is a wrapper for the Twitter APIs, as the rest of our pipeline was i n Python. We could have used the Tweepy library13, as both provided the same functionality and were we ll documented. In relation to Twitter APIs there were two options: the REST (now called ‘Search’) an d the Streaming. REST Search methods were more suited to singular, specific, queries for twee ts over the past seven days, whereas the Streaming API provided a real-time stream of tweets. To compl ete this pipeline we chose the Streaming API as stakeholders had highlighted their need for real-ti me information on flooding events, we discuss additional reasons for our use of the Streaming API in the context of other studies in section 3.1. Experimentation with both Twitter Search14 and Streaming APIs to retrieve flood-related tweet s from Great Britain (and France) was done during May and early June 2016. An example from this experimentation (Fig. A.2) demonstrates a monitored at-risk location intersecting with a geotagged tweet. A web-based console15 was used to test Twitter API queries and view resu lts. A visualisation tool- GeoJSONLint16 was used to inspect the polygons. We monitored solely georeferenced tweets, filtered by the shortlisted areas, which retrieved tweets whose geotag intersected: a second Python script (s ee (i) in Table A.2) queried the Twitter Streaming API for the prioritised list of locations (Step 4) - a maximum of up to 25 location bounding boxes were permitted at a time17. These locations were monitored for any newly post ed tweets whose geotag intersected a bounding box, which constituted a “ma tch”: location queries used the /statuses/filter endpoint of Twitter’s Streaming API, and tweets’ Place or GPS coordinates (as according to a user’s choice for the tweet18) were matched against, with priority given by the API to coordinates when 12 Twython - Twitter API wrapper library in Python: ht tps://github.com/ryanmcgrath/twython Last accessed: 16th Nov. 2018. 13 Tweepy- Twitter API wrapper library in Python: http s://github.com/tweepy/tweepy Last accessed: 16th Nov. 2018. 14 https://developer.twitter.com/en/docs/tweets/searc h/overview Last accessed: 20th Dec. 2017. 15 Website to experiment with public APIs https://api gee.com/console/twitter Last accessed: 20th Dec. 20 17. 16 GeoJSON is a JSON-based, open standard format for geographic features; “linting” helps ensure valid f ormat http://geojsonlint.com Last accessed: 20th Dec. 201 7. 17 https://developer.twitter.com/en/docs/tweets/filte r-realtime/overview ; e.g. a filter endpoint of the form https://stream.twitter.com/1.1/statuses/filter.json ?locations=-74,40,-73,41 Last accessed: 20th Dec. 2 017. 18 See Adding your location to a Tweet | Twitter Help Center. https://support.twitter.com/articles/12223 6 [Accessed 20 Mar. 2017]. MANUSCRIPT ACCEPTED ACCEPTED MANUSCRIPT11 possible (Appendix 1.2.3 Fig. A2 and Appendix B for their specifics). The script then stored the retrieved tweets19 (Step 5) in a MongoDB database. MongoDB was chosen as it was a widely used open-source document database, ideal for storing tw eets in JSON format, with helpful geospatial indexing and querying and full-text searches. Fig. 2 All 163 unique at-risk areas, as sourced from two Br itish national agencies’ data APIs, and queried via Twitter Streaming API 20-29th June 2016. Catchment polygons from EA risk areas were converted to containing bou nding boxes - i.e. rectangular polygons, as required by Twitter A PI20. Box shading gives corresponding date from which an area was first monitored, and size indicates actual bounding box f or each location. Fig. 6 shows a subset of these bo xes in more detail. This pipeline application was in active operation b etween 20-29th June, it consisted of Steps 1-5 and stored tweets to our database, providing a real-wor ld dataset, which we then explored and used for development and testing of the remaining pipeline s teps. Additional database queries were run on the collected tweets to facilitate time/date queries, a nd convert tweet geo