Under the laws of all states in this country so long as you have physical custody of your minor child you legally have the right to keep your husband away from the child so long as there are no court orders setting forth visitation, custody and the like. Given what you have written, I suggest that you consult with a family law attorney to assist you in your matter.