Only touch target boxes and their parents in the downward pass.