Count down latch on failure as well