He has come back home to live. Now they ask for my son to give consent to an adoption. The notice states that if he doesn't give consent that they will seek termination of his parental rights. If the natural parent agrees, that pretty much wraps up the process unless the grandparents have a compelling reason why the adoption would be harmful to the child. If the natural parent refuses to give consent, then a much more complex procedure can be used to terminate parental rights.
Different states have different standards for what constitutes neglect or abandonment.
However, if there is proof that your son maintained a constant interest in his child, then it will be very difficult for the mother to have his parental rights terminated. Failure to pay child support by itself should not be a reason. I have full custody of my 12-year old son. His father and I are divorced, and have had no contact for 10 years. I have since remarried.