9th district (1825–1833) True to his campaign pledge to serve only one term, Polk left office in 1849 and returned to Tennessee where he died three months after leaving the White House. A Kentucky slaveholder at a time when opponents of Texas annexation argued that it would give slavery more room to spread, Clay sought a nuanced position on the issue.