That is unjust to them. The Church is the household and family of God. We are the New Covenant people of God. As circumcision brought infants into the Covenant of Abraham, Isaac and Jacob, so baptism brings infants into the New Covenant of Christ's Blood. Family membership is something given the child, that he grows up with and grows into.
Baptism is a life to be lived, a spirit to be filled with.
It rightly belongs to children, so we give it to them. With their parents and others, children learn to live out their baptism. And it should begin with babyhood. The parallel between circumcision and baptism, between entry into the old Israel and the new, is intensified when we read 1st Cor. Since baptism is the entry to the Church, and the Church is the first stage of the Kingdom, as it appears in this world, babies are baptized to put them in possession of what belongs to them.