After being released from West Bromwich Albion as a teenager, Williams played for non-league side Hednesford Town before turning professional at Stockport County in 2003, later becoming their captain. In March 2008, he was loaned to Swansea City, and after helping them win League One title, signed for a then club record £400,000. He amassed 352 appearances for the Swans, scoring 14 goals. He aided them in their promotion to the Premier League and their victory in the 2013 Football League Cup Final.