Ok, so if I understand correctly, what you propose is to:
1) held out a fixed number of instances before learning at all
2) use the entire held-out data each N steps to estimate accuracy

It seems quite reasonable and I understand now (at least!) what you mean by "not comparable from point to point" :)

I'm going to figure out now how to do the hold out in not so clumsy way.

Stay tuned !