Make sure the count down latch is decremented