The smallest in the four divisions of Premier League plus Nationwide is certainly Irthlingborough - home of Rushden and Diamonds. The biggest town/city never to have had a team in Football League Div 1 or Premier League is usually alleged to be Hull.