There is no way that the situation in the homes of your various children could be identical. They each look after their babies in the best way they can, and if in one case that means bringing him over to your house frequently, you should not assume that the other family will do the same. If you persist in trying to persuade your son and his wife to share your grandson with you more often, you will only alienate them. The quarrel could develop into a full fledged feud which could be disastrous for everyone concerned. Make sure that whatever time you have with the baby is pleasant and doesn't include any arguing about anything.
Babies observe everything, and little children remember.
They form attitudes early. Love him and display your love for his parents every time you are together with them. If you feel an emptiness in your life that only time with that particular grandson will fill, I think you could look around the community and find other activities to fill the void. Volunteering at hospitals, community centres, senior centres, schools, church - are just a few ideas. Or how about art classes, or singing in a choir.