CountDownLatch was the right choice in the end, otherwise we might deadlock on subsequent calls to get()