He was born in Inveresk House on 12 November 1782, the fourth son of Isabella Blackburn and James Wedderburn Colville of Ochiltree House in Ayrshire. John Wedderburn of Ballindean, 6th Baronet of Blackness (1729–1803) was his father's elder brother. Both brothers spent decades buying and managing sugar plantations in the Caribbean, re-establishing the family fortune following the execution and attainder of their father the 5th baronet after the Jacobite uprising of 1745.