MapofLife/MOL — Issue #27: loader.py should be able to continue a partial upload
Author: gaurav
Created: 2012-01-05 23:55:02+00:00
State: open

With hundreds of shapefiles and thousands of features to upload, it is no longer feasible for loader.py to upload everything in one go - any error at any point in the process currently requires the entire upload to be repeated.

I propose a simple verification step, which will count the number of features in a particular file. If the count on CartoDB agrees with the count in the file, the data will be assumed to be accurate, and that file will not be reuploaded. If the count disagrees, all features for that file on CartoDB will be deleted and the entire file will be reuploaded. If the file is not found on the server, it will be deleted.

A command line option for loader.py will force it to delete all data for a particular provider/collection combination, thus forcing all data to be uploaded again.