Yes. Under the right circumstances (or quite frankly, the wrong circumstances) either parent can be denied joint legal or shared physical custody. It depends upon so many different considerations, but the common denominator is always what's in that child's best interest.
If a Judge or Magistrate determines that the father (or mother) is incapable of making decisions, he or she may deny that parent joint legal custody and give the other parent sole decision-making power. Or, if there is evidence of one parent acting inappropriately toward the child or others when the child is in their care, the other parent may be given sole physical custody. Again, the Court always looks at it from the child's perspective. It's not about the father. It's not about the mother. It is all about the child.