Fix out-of-tree builds rebuilding too much

This fixes out-of-tree builds rebuilding things that are already built.