Drummond was a member of the family who owned and ran Drummonds Bank, and a great grandson of William Drummond, 4th Viscount Strathallan who died fighting for the Jacobite cause at the Battle of Culloden in 1746. His parents were Charles Drummond, a banker, and his wife, Frances Dorothy (herself the second daughter of the Reverend Edward Lockwood). He was one of nine children, of whom five survived to adulthood. His sister Charlotte did not marry and shared a house with him in Lower Grosvenor Street. His older brother Charles followed their father into the bank; younger brother Berkeley went into the army and younger brother Arthur into the church. Drummond joined the civil service in June 1814, becoming a clerk at the Treasury. He was later Private Secretary to a succession of British Prime Ministers: George Canning, Lord Goderich, the 1st Duke of Wellington, and Robert Peel.