Defer the semaphore to be sure to releease the lock