You cannot do one without the other. Unless you do both together, you cannot love either God or neighbour. God commanded these ancient Jews to worship him with sacrifices of animals and grain and all the other rituals of tabernacle and temple. No, it doesn't. If you think going to church is a ritual by which you worship God, you are mistaken.
Church actually belongs to the other category.
It's part of loving our neighbour. But there is a worship part of our love for God, just as for Israel in Leviticus nineteen. Israel came to God only on his terms. They worshipped only by the sacrificial system which God himself ordained. But that system was only a shadow of the reality to come in Christ.