It might drive people away, etc. Other members show by their lack of attendance and involvement that they don't consider the church important. Such ideas are primarily caused by denominationalism. People see denominational division or hypocrisy in the church or churches not following the Bible, and they want no part of it. So they reject the whole idea of the church.
Denominations deliberately belittle church membership in order to justify their division.
So people conclude you don't need to be a member of any church at all, as long as you believe in Jesus. The premillennial concept says Jesus came to set up His kingdom but failed because people rejected Him. So He changed His plan and set up the church. This makes the church an unplanned stop-gap substitute, which God never really wanted to set up. That does not make it very important.