He was born in Springfield, Essex in England, to Robert Livermore and Mary Cudworth. Livermore was a stonemason's apprentice as a youth. At the age of 17, he decided to go to sea, joining the crew of an English merchant ship. Arriving in Baltimore, Maryland, he enlisted in the United States Navy and traveled to South America. He subsequently was part of Lord Cochrane's crew in 1820 during the Peruvian War of Independence against Spain. After Peru, he signed on with an English trading ship bound for California.