Remove some more checks from critical loops