significant improvements to efficiency of chunking algorithm