There is a compromise here. We'd like to be running less Python code which implies bigger chunks. But smaller chunks are more cache friendly. 8MB is an empirically determined value which sits vaguely near the quick end of things. If you make it much smaller it can rapidly get a lot slower. But if you make it bigger it gradually gets slower.