I know this is a hack / not an ideal work-around, but I would prefer if we can do this only every x iterations or something like that.

Doing it on every iteration is even less ideal, because it can slow things down.