Only do this calculation after a suspicious byte.