Short answer is as soon as pudding thickens. It seems you're whisking for hours when really it's only few minutes and the consistency doesn't change, then at some point everything comes together and pudding becomes thicker and it gets harder to whisk. This is when you remove it from the heat, otherwise either milk will burn at the bottom of the saucepan and not smell pleasant or eggs will curdle and pudding loses consistency.