Buller was born into a prominent West country family in 1764 and began his naval career twelve years later, serving with Lord Mulgrave during the American War of Independence. He initially saw action at the Battle of Ushant in 1778, before travelling to the East Indies with Sir Edward Hughes and participating in several of the engagements with the Bailli de Suffren. Appointed to his first command during his time off India, Buller narrowly survived a hurricane and a hazardous journey back to Britain. He commanded a sloop off the North American coast after the end of the war, and received his promotion to post-captain in 1790. Buller spent the early years of the French Revolutionary Wars escorting convoys and serving on the Cape of Good Hope, before returning to Britain and spending some time ashore, and some time at sea commanding ships in the English Channel.