He wound down his career after 2003, running a best of 47.03 seconds in 2004 and 47.81 in 2006 before retiring from the sport. Following the end of his active career, he moved into sports administration. He was elected as secretary general of the Bahamas Association of Athletic Associations (BAAA) in 2012. His appointment came at a time of much political in-fighting at the organisation, between the president Mike Sands and other members of the board. Oliver found himself excluded from the 2013 CARIFTA Games organising committee, alongside fellow BAAA executives Iram Lewis and Harrison Petty. The decision to host the 2014 IAAF World Relays event was also made without their input. Oliver agreed to a vote of no confidence at a BAAA general meeting in response.