Basically he says, ignore them and leave them to God. God knows those who are his. God knows you are his by faith in His son alone, apart from any man-made rulekeeping. Just ignore them. They are nothing in God's sight.
Legalism causes fear.
You are afraid of being seen to fail, of not living up to man-made standards, and so you are not willing to show what you are really like on the inside. Legalism is a facade, and it produces hypocrisy. That is what we see in verses 7 to 9. The essence of hypocrisy is that you think you are better than you really are, because you use human standards and not God's standard to judge yourself. You compare yourself, not to God, but to others.