In his first years in Parliament Robinson declined offers of junior ministerial posts, out of deference to his patron Hardwicke, who was an opponent of the Prime Minister, the Duke of Portland. However, the Foreign Secretary, George Canning, chose him as the secretary of Lord Pembroke's mission to Vienna, aimed at securing a new treaty of alliance between Britain and Austria. The mission was unsuccessful, but Robinson's reputation was not damaged, and, as his biographer E Royston Pike puts it, "as a good Tory [he was] given several small appointments in successive ministries." His political thinking was greatly influenced by Canning, but he became the protégé of Canning's rival Lord Castlereagh, who appointed him his under-secretary at the War Office in May 1809. When Castlereagh resigned from the government in October, unwilling to serve under the new Prime Minister, Spencer Perceval, Robinson resigned with him. In June 1810 he accepted office as a member of the Admiralty board. At the time of Perceval's assassination early in 1812, he was absent from parliament ostensibly on militia duties in Yorkshire.