Silverman and Cowell first met back in 2004, when Silverman was still married to her first husband, real estate mogul Andrew Silverman, who was a good friend of Cowell's. … Cowell spoke to the Mirror in 2014, where he confirmed that the pair had had an affair whilst Silverman was still married.