If you were never married and there has been nothing to establish paternity or legal custody on his behalf, then the decisions regarding your son are yours alone.  But remember that he can indeed go to court to establish all of the above at any time.  Good luck.