optimizations and tests for external_id/chunking code