Repository: bokeh/bokeh

Issue:
Twitter scrapping map demo using CLI tool 
References: #1585 

Description from @pzwang : "twitter-scraping tool that plops geotagged tweets matching some hashtags as scatter points on a map plot"

The main task for this would be figuring out how to allow users to spell (on the command line) a mapping from the data returned from some REST endpoint, to what gets shoved into a Bokeh `ColumnDataSource`. @pzwang your input on what you'd like to be able to write at the command prompt to make this happen would help drive this. 



Discussion:
Ah, @pzwang, we had very similar ideas about this find of example :-)

I had many different ideas about examples of scatter (or bubble) charts with an overlayed map plot (though more about virus evolution or crowd distribution over time then twitter hashtags, but the concept is very similar).

Right now `charts` interface misses some features we need to make this happen:
- map plot and bubble plot support
- Overlaying multiple charts on the same figure

What we have in place already:
- scatter chart
- CLI accepts file inputs and supports a `--sync_with_source` parameter that forces the CLI to read the input file source every n seconds and update the `sources` on `bokeh-server` if the input file changes. This is not a super nice design but very common use case (using files to transfer data between different apps). Another couple of inputs I've being thinking about that I find interesting are reading from rest services or from a socket. But those would probably need more engineering to let user specify how to consume this data..

On the specific scrapper example, would this be an external process feeding the CLI with input file stored on disk? From a DB? 

I'd love to hear more thoughts, opinions and examples of what we could/should be supporting. :-)

Instead of reading the file every N seconds, I think perhaps this is a perfect use case for generators? It is trivial to write a generator in python the behaves like `tail -f` and then you can easily chain up an entire data processing pipeline to that. This is pretty much the entire content of Chapter 8 of Beazley's class that we teach. I will point you at the materials on FD. 

FWIW I think, of the three, I think this demo is the "stretch goal"

@pzwang @bryevdv to get this done we also need to figure out some important details like how CLI tool interacts with scrapping tool, if we build this scrapping too [IIRC]  this needs to authenticate to twitter and we need to understand how to deal with the auth key and secret... With that in mind we won't get this in time for 0.7.1 .. 
Would a simulation script that writes simulated geolocated tweeds to a file consumed by the CLI be an acceptable goal for 0.7.1? We could improve it for 0.8 later..

You're absolutely right - the data download portion of that demo will be significant to do "for real", and introduce a lot of code and complexity that is unrelated to the purpose of demonstrating bokeh's features.  It would be sufficient and simpler to have an example gzipp-ed file of JSON tweets (maybe even downloaded from Twitter or some other source), and a Python one-liner to read out a tweet every second.

```
python -c 'import gzip,time,sys; for line in gzip.open("sampletweets.gz"): (sys.stdout.write(line),time.sleep(1))'
```

Opinions about how users could specify that they want a GMapPlot? First thing I can think of is a new CLI option:  `--gmap`  that accepts "lat,lon,zoom,map_type" and tells how to build the GMap. Show usage would be:

``` Bash
bokeh --gmap "97.3,36.2,10,hybrid"
```

I'm not sure if I would prefer to have this fully expanded or something like this:

```
$ bokeh --gmap=97.3:36.2 --zoom=10 --map=hybrid"
```

+1 on that suggestion

I like it more too. But what about `map_zoom`? `zoom` could be exchanged with the zoom tools..

```
$ bokeh --map=97.3,36.2 --map_zoom=10 --map_layer=hybrid"
```

later possibility of `--map_type=geojs`, e.g.

Note: personally prefer comma for separating lat, lon. 

+1 (using `map_layer` here overlaps with `GMapOptions` attr `map_type` but I think it's the best option and is an implementation details not know to most of users)


Resolving patch:
diff --git a/examples/cli/sample_data/create_tweets_file.py b/examples/cli/sample_data/create_tweets_file.py
new file mode 100644
--- /dev/null
+++ b/examples/cli/sample_data/create_tweets_file.py
@@ -0,0 +1,38 @@
+import gzip,time,sys, os, random
+here = os.path.dirname(os.path.realpath(__file__))
+filepath = os.path.join(here, 'sim_tweets.csv')
+
+header = 'text,lat,lon\n'
+with file(filepath, 'w') as f:
+    f.write(header)
+
+tweets = []
+lons = []
+lats = []
+# get some saved simulated tweet as base
+for i, line in enumerate(gzip.open(os.path.join(here, "simplified_geo_tweets.gz"))):
+    if i > 0:
+        line = line.strip()
+        tweet = line.split(',')
+        if len(tweet) == 3:
+
+            tweets.append(tweet)
+            lons.append(float(tweet[1]))
+            lats.append(float(tweet[2]))
+            with file(filepath, 'a') as f:
+                f.write("%s\n" % line)
+            time.sleep(0.1)
+
+delta = 1
+lons = [min(lons)-delta, max(lons)+delta]
+lats = [min(lats)-delta, max(lats)+delta]
+# start infinite retweets from random locations insite the original tweets area
+while True:
+    for tweet in tweets:
+        tweet[1] = random.uniform(*lons)
+        tweet[2] = random.uniform(*lats)
+        line = ','.join(str(x) for x in tweet)
+
+        with file(filepath, 'a') as f:
+            f.write("%s\n" % line)
+        time.sleep(0.1)
\ No newline at end of file


Test patch: