Fix eager load when relation is empty

Eager loaded documents cannot try to fetch from the db if it didnt find from the first
time.

[fixes #3379]