Educated at the High School, Edinburgh, where he was a contemporary of Lord Brougham, Frazer joined the Royal Military Academy, at Woolwich, on 16 August 1790 as a gentleman cadet. On 18 September 1793, he was gazetted a second lieutenant in the Royal Artillery, and at the end of the year joined the Army in Flanders, under command of the Duke of York. In early 1794, he was promoted to First Lieutenant, and attached with 2 field guns to the 3rd Foot Guards (Grenadiers), where he served until the army returned to England in May 1795. During that time, he was present at several engagements, including the battles of Tournay and Boxtel. Upon return to England, he was appointed to the Royal Horse Artillery, and on 16 July 1799 received the rank of Captain Lieutenant. From August to late November, he took part in the Duke of York's expedition in Holland.