Fix out-of-tree builds rebuilding too much

This fixes out-of-tree builds rebuilding things that are already built.  I believe this was missing from 3f3ee1aab336d2194ca79871810e6abdbdf5717b