It is possible that the relationship with Vivienne happened before Uther was married to Ygraine, since Morgana is older than Arthur.
When Morgause summoned Ygraine's spirit to speak with Arthur, she vehemently claimed that Uther betrayed her by knowingly sacrificing her so Arthur could be born through magic.