He clearly had some military aspirations, enlisting into the Royal Navy as a midshipman aged 14 and serving aboard HMS "Victory" as a lieutenant at 18. At 17 he was also listed as a founding Governor of the Foundling Hospital, a popular charity project to look after abandoned babies that had previously been championed by his mother. After an abortive stint as a captain in the Duke of Kingston's Regiment during the Jacobite Rebellion, he went on to marry Elizabeth Shaw, daughter and heiress of Charles Shaw of Besthorpe in Norfolk, on 28 March 1747 – they went on to have four children, two of whom lived to adulthood, including: