Fix a threading issue
Remove some unneeded UNLIKELYs