BENJAMIN OGLE 

TAYLOE

TAYLOE HOUSE
Benjamin Ogle Tayloe House , circa 1890.
 The Tayloe House is one of a small group of historic 

buildings located on Madison Place, on the east side of 

Lafayette Square in Washington, D.C. The Tayloe House 

was erected in 1828 by Benjamin Ogle Tayloe, second son of 

the prominent Washingtonian Colonel John Tayloe who built 

the famous Octagon House located on New York Avenue. 

 The fi rst and third fl oors of the yellow three-story brick 

house now serve as conference, reception and offi ce space 

for the United States Court of Appeals for the Federal Circuit 

and the Court of Federal Claims. The middle level contains 

a cafeteria and dining room serving breakfast and lunch. The 

Tayloe House Café is open to the public. 

 At the time of Benjamin Tayloe’s birth, George Washington 

was ending his term as President. Over the years Benjamin 

Tayloe became one of the most infl uential people in the 

nation’s capital as he developed close relationships with his 

neighbors, President Washington’s successors. 

 Tayloe received his formal education at Phillips Academy 

in Exeter, New Hampshire, and at Harvard College in 

Cambridge, Massachusetts. After graduation from Harvard 

in 1815, he studied law and accepted a position with the 

American Ambassador to Great Britain. After several years 

in London, he returned to the United States and purchased a 

large estate on the Potomac River in King George County, 

Virginia. He named it “Windsor.” 

 In November 1824, Tayloe married the former Julia 

Dickinson, daughter of Congressman John D. Dickinson 

who represented the Rensselaer District of New York from 

1819-1823 and 1827-1831. At Windsor, Tayloe became 

a country gentleman and engaged in breeding cattle and 

pursued his keen interest in horse racing. His wife, however, 

enjoyed an urban life and particularly the social activities 

and excitement of the capital city. 

 In 1827 Tayloe’s father gave the couple a lot in 

Washington, D.C. in the “United States Bank Square,” which 

would later become known as “East President’s Square” and 

included Madison Place. Tayloe began construction on his 

new home, which was completed in 1828. It was the second 

home to be erected on Madison Place; the fi rst was the 

Dolley Madison House. 
My dear Madam: 

 I comply with your request, and return 

your volume with a name which, I cannot have 

the vanity to believe, will in the slightest degree 

enhance its value. I am, with regard, Your most 

obedient servant, 

 Daniel Webster

 January 29, 1829

 After the house was constructed, Tayloe chose to lease the 

house and remain on his estate in Virginia. In 1829, Julia 

Tayloe persuaded her husband to abandon country life for 

a more active social life and the family moved into their 

home on Madison Place in November of that year. 

Mrs. Tayloe soon became one of the capital’s favorite 

hostesses. The house became a center of early nineteenth 

century Washington social life and was known as “The 

House on the Square.” Distinguished guests included 

John Quincy Adams, John C. Calhoun, Henry Clay, Dolley 

Madison, Chief Justice John Marshall, Daniel Webster, and 

Presidents Jackson, Van Buren, Harrison, Tyler and Fillmore. 

 The house contained a rare collection of fi ne paintings and 

other art objects obtained by the Tayloes during their world 

travels. One of the original works of art, a bust of Tayloe by 

Hiram Powers, is on display on the fi rst fl oor of the Tayloe 

House today. Also displayed on the fi rst fl oor are parts of the 

French china service, circa 1775, used by the Tayloes when 

entertaining guests. 

 The “House on the Square” hosted many social and 

political gatherings during the period of the 1830s and 

1840s. Julia Tayloe would ask her many visitors to write a 

verse in an album which she placed in a central place in the 

house. A neighbor on Lafayette Square who would serve 

his country as Congressman, Senator, Secretary of State and 

renowned attorney wrote:
 After moving to his new home, Tayloe soon became a close 

friend of Senator Henry Clay of Kentucky who owned the 

lot next door and who was said to have won money to pay 

for the lot in a high stakes poker game. Clay then traded the 

lot to Commodore John Rodgers of the Navy for a jackass 

that Rodgers had brought back to America from one of his 

many travels abroad. The “Rodgers House” was completed 

in 1830. 

Julia Dickinson Tayloe
 Julia Tayloe died in 1846. Three years later Tayloe 

married Phebe Warren, a close friend of his fi rst wife and, 

like her, a native of Troy, New York. The advent of the Civil 

War found the Tayloes in the midst of wartime activities with 

General McClellan’s headquarters next door in the Dolley 

Madison House while Secretary of State William H. Seward 

resided in the Rodgers House on the other side. 

Benjamin Ogle Tayloe
 The “House on the Square” once again became a center 

for both social and political gatherings. One close friend, 

General Winfi eld Scott, was a frequent guest and Tayloe 

recorded an incident on one of his visits:

 Benjamin Ogle Tayloe died while vacationing in Rome, 

Italy in 1868 at the age of 71. His body was returned to 

the United States and was buried in the Warren family 

chapel in Troy, New York. At the time of his death, Tayloe 

had substantial fi nancial holdings, particularly real estate 

in Virginia and Washington, D.C., in addition to cotton 

plantations in Alabama. His will provided that most of his 

property be divided among his children. The “House on 

the Square” plus one-fi fth of the income from his estate for 

life was given to his wife Phebe, with the provision that she 

bequeath the house to the children or their heirs. 

 Mrs. Tayloe remained in the house until her death in 1884. 

The “House on the Square” was sold by Tayloe’s children 

shortly after Phebe’s death. The fabulous art collection 

was bequeathed to the Corcoran Gallery of Art, with the 

provision that it be retained in the gallery as a separate group 

called “The Ogle Tayloe Collection.” It remained intact until 

the gallery moved to its new home in 1902, when most of the 

art was returned to the family. 

One Sunday in the summer of 1857, a terrifi c 

hailstorm hit the capital killing cattle, breaking 

the thick glass of the Capitol building, and 

passed through our windows like bullets. 

The General seemed greatly impressed and 

solemnized. Dinner was delayed. When the 

storm was over, we cooled our champagne with 

hailstones, bowl after bowl being brought in, of 

what the General called “celestial ice.” 
 Senator James Donald Cameron of Pennsylvania 

purchased the house from the Tayloe heirs in 1887. Senator 

Cameron made some changes to the interior of the house, 

including installation of a beautiful staircase which has been 

restored to its original splendor. In 1895, The Lafayette 

Square Opera House started to rise on the site of the former 

Rodgers House. Senator Cameron and his neighbors were 

outraged. Senator Cameron was quoted in the newspaper as 

having said that he would not have improved his property 

had he known a theater was going up within a “biscuit’s 

throw of his southern windows.” Cameron remained in the 

house until 1897, when he declined to run for reelection and 

returned to his other home near Harrisburg, Pennsylvania. 

 In 1897 the Tayloe House was leased to Vice President 

Garret A. Hobart who resided here until his death in 1899. 

Senator Marcus Alonzo Hanna of Ohio, who was the chief 

advisor and confi dant of President William McKinley, made 

the house his residence from 1900 to 1902, and it was during 

this period that the house became known as the “Little White 

House” because of frequent visits by President McKinley. In 

1903 Hanna moved across the street to the Arlington Hotel 

on Vermont Avenue, where the Veterans Administration now 

stands. Cameron continued to lease the house to others for 

many years. 


Benjamin Ogle Tayloe and Julia Dickinson Tayloe portraits courtesy of the 

Rensselaer County Historical Society. Photography by Shannon Chambers, 

Patricia McDermott and Robin Reid. 

 In late 1915, The Congressional Union for Women’s 

Suffrage, or The National Women’s Party, leased the house 

and it became the national headquarters for the movement to 

give women the right to vote. During that time Alice Paul, 

one of the most active suffragists, led peaceful marches 

to the White House. These women, known as the “Silent 

Sentinels,” helped immeasurably to achieve the vote for 

women in 1920. 

 Meanwhile, the Cosmos Club, which occupied the Dolley 

Madison House and the property immediately adjacent to the 

Tayloe House, began looking for additional space. Senator 

Cameron sold the property in 1917 to the Cosmos Club 

for $250,000 and the property was immediately made the 

“Ladies Annex.” 

 In 1940 the Cosmos Club sold the Tayloe House, along 

with other holdings of the Club on Madison Place, to 

the federal government for eventual construction of a 

government building or a courthouse. The Club was paid 

$1 million for its Madison Place property. World War II 

intervened and all government construction in Washington 

ceased. The Cosmos Club continued to rent the house until 

1952, when it moved to its new location on Massachusetts 

Avenue. 

 The National Science Foundation and then the National 

Aeronautics and Space Administration became the new 

occupants of the Tayloe House, Cosmos Club and Dolley 

Madison properties. In 1964, the Lafayette Square 

Restoration Project was championed by President 

John F. Kennedy and Mrs. Kennedy, utilizing the 

architectural plan of their friend John Carl Warnecke. 

When the project was completed in 1968, the “House on 

the Square” became part of the National Courts Building 

complex. 

 Fortunately, this restoration retained much of the original 

house constructed by Benjamin Ogle Tayloe in 1828 and it 

remains one of the most attractive and historically signifi cant 

buildings on Lafayette Square. 
And when our days’ sun shall go down in the west,

Mid the shadows of trials and joys to its rest,

May bright rays of hope gild that evening’s last sorrowWith a promise of blessed reunion the morrow.

In that city celestial, whose streets have been trod

By saints and by martyrs, the “city of God,”

In the palace called beautiful, gathered be there

All who met or who dwelt in “that House on the Square.”

 Penned by a family friend

 following the death of

 Benjamin Ogle Tayloe

TAYLOE HOUSE

United States Court of Appeals for the Federal Circuit 

Lafayette Square

Washington, DC 20439

www.cafc.uscourts.gov May 2010