added some useful stderr and logging messages.  Also improved memory consumption by dropping some unneeded columns midway through processing.