Church was an experienced duellist, and owned the Wogdon pistols used in the 1804 Burr–Hamilton duel. The weapons had already been used in an 1801 duel, in which Hamilton's son Philip was killed. Following the duel, the pistols were returned to Church, and reposed at his Belvidere estate until the late 19th century.