Not unless the child's father's rights were terminated in some way under the law.  He has the same rights as you do to the child and that means that if something were to happen to you he would have the first right to custody.  If he were found to be unfit or that allegation were made to challenge the right to custody then a court would decide.  Good luck.