> Ok, so if I understand correctly, what you propose is to:
> 1) held out a fixed number of instances before learning at all
> 2) use the entire held-out data each N steps to estimate accuracy

This is a standard setting in online learning.