Gardner traveled with James Madison to Philadelphia in 1780, where Madison was serving in the Continental Congress. Madison was short on cash in 1782, but wanted to acquire philosophy books, including Thomas Hobbes's "Leviathan". In trying to acquire these books, he considered selling Gardner in order to pay for them. Laws in Pennsylvania would have made this impossible. That year, Gardner also attempted to runaway but was subsequently captured.