Fix race condition while removing object directory

Found this while running the same script several times concurrently.