Garrard returned to Mount Lebanon, where he developed a reputation as a notable agriculturist. His son James oversaw the day-to-day operation of the farm and frequently won prizes for his innovations at local agricultural fairs. The Mount Lebanon estate was badly damaged by one of the New Madrid earthquakes in 1811, but Garrard insisted on repairing the damage as thoroughly as possible in order to reside there for the rest of his life. He imported fine livestock – including thoroughbred horses and cattle – to his farm and invested in several commercial enterprises, including several saltworks, which passed to his sons upon his death. He died on January 19, 1822, following several years of feeble health. He was buried on the grounds of his Mount Lebanon estate, and the state of Kentucky erected a monument over his grave site.