No need to do a CAS here, we can just call decrement.