mapbox/tilemill — Issue #922: Intelligent url cache refreshing
Author: springmeyer
Created: 2011-11-15 23:10:08+00:00
State: open

TileMill's support of URL based datasources like remote geojson, kml, and csv opens up big possibilities for visualization of live data streams.

Currently however we cache data once and require the user to "flush" the cache in the layer UI to re-trigger downloading data.

We can think of cool ways to make this more powerful based on the general idea of [TTL](http://en.wikipedia.org/wiki/Time_to_live).

For starters we could simply allow the user to request that the data be re-cached on every save event (for very small,live data this could work well). But we could also start parsing by `Expires` and `Cache-Control` headers to learn if remote resources should be re-cached, and at what interval or what time.

An extreme, but fun example is rendering plain test csv out of piratepad like: http://piratepad.net/lUa5VImYua. Piratepad supports hitting a download link directly for the plain text which can be used in TileMill as a datasource: http://piratepad.net/ep/pad/export/lUa5VImYua/latest?format=txt

Curl shows valuable headers (including `Pragma: no-cache`) indicating this is live data:

```

$ curl -I http://piratepad.net/ep/pad/export/lUa5VImYua/latest?format=txt
HTTP/1.1 200 OK
Server: nginx/0.8.52
Date: Tue, 15 Nov 2011 23:01:55 GMT
Content-Type: text/plain; charset=utf-8
Connection: keep-alive
Set-Cookie: ES=6a9e0fee3d472674; domain=.piratepad.net; path=/
P3P: CP="IDC DSP COR CURa ADMa OUR IND PHY ONL COM STA"
Set-Cookie: ET=6fbe3fb4b0038f20; expires=Wed, 01-Jan-3000 08:00:00 GMT; domain=.piratepad.net; path=/
Expires: Sat, 18 Jun 1983 07:07:07 GMT
Last-Modified: Tue, 15 Nov 2011 23:01:55 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: ES=25f48695d19d20b5; domain=.piratepad.net; path=/
Content-Disposition: attachment; filename="lUa5VImYua-latest.txt"
Set-Cookie: ES=bd1d5d166e44b100; domain=.piratepad.net; path=/
Set-Cookie: ES=27b230f5f23895a5; domain=.piratepad.net; path=/
Content-Length: 26
```