Remove the semaphore on daemon stop