fix deadlock for non fitness only - count down the latch also for readding parents