The main problem is that your results aren't comparable from point to point as they are with a single held-out test set. So if the only thing we're trying to illustrate is the out-of-core scaling, you should definitely be holding out some instances.