That is nothing to sneeze at. I remember asking a young man if he attended a church somewhere. Then I was politely dismissed. I have heard this excuse many times. And I thought it was time that I address it seriously.
Lets look at that response.
And lets hypothetically say that we can worship God anywhere. Another part of Worship is fellowship. The most important ingredient in worship specifically incorporated into our worship is the hearing of God's Word, hearing about our sinful nature that leads to death, and the wonderful news of God's forgiveness in Christ, and paying for our sin on the cross. Only in Church are we reminded the lifesaving news of the Gospel on a regular basis. That God loves sinners like you and me.