Robinlovelace/tweepy — pull request #7 by nickmalleson (2014-08-04 19:49:45+00:00)
Title: Finished json2csv

Hi Robin, I've added a script to convert json files to csv. At the moment it just converts, but I intend to extend it to any analysis we want, running on the grid (distributed across cores). Otherwise the data are becoming a bit unmanageable! As it stands the multi-thread version should run pretty quickly (it runs in multiple threads by default)

It will accept command-line arguments giving the fields to add to the csv, but it also has some defaults. E.g.:

python json2csv.py -o tweets.csv data/*

will take all data in 'data' dir and write to tweets.csv. It will parse normal json text files, and the gzipped ones as well (assuming they end in '.gz'),

Could you see if it works with your data as it stands? It seems OK on my ~500k tweet collections, but before I change anything I want to see if it breaks down on a fuller set. Particularly if I need to make it more memory efficient. It also seems to handle some of the unicode characters that have been bothering you, at least on OS X (again this might have different behaviour on your OS).