Drummond first came back to Canada in 1809 on the staff of Governor General Sir James Henry Craig. After briefly serving as Commander-in-Chief of the Canadas in 1811, as a result of Craig being recalled to England, Drummond was reassigned to Ulster. Late in 1813, Drummond was recalled to North America to serve in Upper Canada as lieutenant governor, replacing Francis de Rottenburg. Rottenburg had proven an unpopular officer who was considered over-cautious, nervous about any sort of engagement, and reluctant to send reinforcements to vital areas. Successive lieutenant governors—Rottenburg and his predecessor, Roger Hale Sheaffe—had failed to make an impact in the North American war since the death of the successful Sir Isaac Brock at the Battle of Queenston Heights.