Jesus never goes back to the issue of adultery. It was a thrust against the sealed door of her heart. But now his foot is in and he is willing to take the very issue she raised and use it to finish his saving work. How and whom are vastly more important than where . Worship is not merely an external act that you can accomplish by going to a place.
Worship is first and foremost an experience of the heart.
Prayer without heart is vain. Songs without heart are vain. Confession and creeds and liturgies and sermons that don't come from the heart are empty and worthless in God's eyes. How you worship is vastly more important than where. Then verse 22 introduces the question of whom you worship.