Remove unneeded modules from non-build node

Since git failures are seemingly quite common,
remove as many clones as we can from the nodes
that don't need the full set.