Require destinations when they are needed and do not fetch all of them in advance

Fixes #5266.