make the highest ancestor wait, and rm the semaphore