Stopped script trying to read non-files (e.g. directories)
Minor change to try to catch ValueError caused when decoding raw json data
Added new 'firehose' functionality: the '-s' flag now makes the listener just listen to the twitter firehose sample (1% of all tweets). Also changed the way it handles data that aren't tweets: now it checks for 'delete' messages and ignores these. There are a lot so it wasn't possible to write them all out. It will print a message every so often saying how many of these it has ignored.
Compressing json files now done in a separate thread so that the script doesn't halt and potentially miss incoming tweets.
Small change to stop script trying to parse tweets that can't first be recognised as valid json
Last fix didn't work, program still crashed. Will try returning True rather than False in on_data()
Nicks merge