Elsom was chairman of Leicester City F.C. from 1998 until mid-2002. During his tenure as chairman the club finished in mid-table in the Premier League, took part in the 1999 and 2000 League Cup Finals, and competed in the UEFA Cup. He also gave them a new stadium, the 32,000-seat Walkers Stadium, which opened for the 2002-03 season as a replacement for 111-year-old Filbert Street. However, his chairmanship of the club came to an end after the 2001-02 season when the Foxes had been relegated from the Premier League and were more than £30million in debt.