Tayloe had very close ties to Washington DC. His father built the Octagon House in 1799. The Octagon was designed by Dr. William Thornton, the first architect of the U.S. Capitol. In 1814, Colonel Tayloe offered the use of his home to President James Madison and his wife, Dolley, for a temporary "Executive Mansion" after the burning of the White House by the British. Madison used the circular room above the entrance as a study and there, on February 17, 1815, he signed the Treaty of Ghent which ended the War of 1812.