Increasing the amount of VAS space I needed worked.
It unfortunately looks like the prefetcher is doing some kind of magic.