James Walker was born in 1764, the son of James Walker of Innerdovat, Fife and his wife the novelist, Mary Leslie, the third daughter of Alexander Melville, 5th Earl of Leven. He entered the navy as a midshipman aboard the 32-gun on 18 December 1776, serving under Captain William Garnier. He went out to Jamaica in January 1777, but returned to British waters for service in the North Sea and then the English Channel with Sir Charles Hardy's fleet during the invasion crisis in 1779. While serving in the Channel in 1780, "Southampton" captured an 18-gun French privateer off Portland, with 80 men aboard her. Walker was sent to assist in removing the prisoners, and after doing so remained on board to help with the baling and pumping, as the privateer was in danger of sinking. Despite his efforts the privateer suddenly sank, nearly taking Walker down with her. He was in the water for ten minutes before being rescued.