Fix the semaphore unblock issue