Castle Williams Castle Williams is a circular fortification of red sandstone on the northwest point of Governors Island , part of a system of forts designed and constructed in the early 19th century to protect New York City from naval attack . It is a prominent landmark in New York Harbor . Together with Fort Jay ( Fort Columbus ) , it is managed by the National Park Service as part of Governors Island National Monument . Design and construction . Castle Williams was designed and erected between 1807 and 1811 under the direction of Lieutenant Colonel ( later Colonel ) Jonathan Williams , Chief Engineer of the Corps of Engineers and first Superintendent of the United States Military Academy at West Point , New York . The castle was one component of a defensive system for the inner harbor that included Fort Columbus ( later renamed Fort Jay ) and the South Battery on Governors Island ; Castle Clinton at the southern tip of Manhattan , Fort Wood on Liberty Island , and Fort Gibson on Ellis Island . Its pioneering design consisted of multiple levels of enclosed or fortified gun enplacements ( casemates ) . It established a prototype for American coastal fortification design for the rest of the 19th Century . The nearly circular fortification , 40 feet high and 210 feet in diameter , was constructed of sandstone walls 7 to 8 feet thick . Each of its four levels had 13 casements that could each hold 26 cannons of varied caliber . Before its completion , Colonel Henry Burbeck , commanding the defenses of New York , issued an Army order on November 24 , 1810 , that named the castle for its designer and builder : " In future the Stone Tower on this Island ( by the approbation of the Secretary of War ) will bear the name of CASTLE WILLIAMS , in honor of the commandant of the United States Corps of Engineers , who designed and erected it . " Civil War . During the Civil War , the casemates of Castle Williams were used either to house newly recruited Union troops , to serve as a barracks for the garrison ' s troops , and to imprison Confederate enlisted men and deserters from the Union Army . After 1865 it became a low - security military prison that was also used as quarters for recruits and transient troops . By the 1880s , the castle , with its pitted and crumbling walls , was considered to be an aging and obsolete fortification . Improvements that included the installation of central heating and plumbing were most likely made in 1895 when Castle Williams was designated one of the U . S . Army ' s ten military prisons . Prison . A commitment to preserve the forts of Governors Island was made in the early 20th century by Secretary of War Elihu Root when landfill operations doubled the size of Governors Island between 1901 and 1912 . The castle was fitted up as a model prison in 1903 . It was most likely wired for electricity when it became available on the island in 1904 . The angled gate walls were remodeled in 1912 - 13 to create a two - story guardhouse , using stones from two demolished magazines within the courtyard . Castle Williams became the Atlantic Branch of the Fort Leavenworth Disciplinary Barracks in 1915 , and the Eastern Branch of the United States Disciplinary Barracks in 1921 . Expansion of the plumbing system occurred in 1916 , and complete renovation of the plumbing , central heating , and electrical systems was carried out in the 1930s . The floors and roof were also reinforced with steel in the 1930s , and steel grating and solitary confinement cells were installed in selected casemates of the second and third tiers . Extensive renovations were carried out in 1947 - 48 that gave the courtyard an industrial appearance . Concrete balconies enclosed with steel sashes replaced existing wooden galleries , and a three - story brick addition enclosed a steel stair . Concrete floors and brick partitions were installed in the casemates of the second tier , and steel security sashes and doors replaced those made of wood . Castle Williams ceased operations as a military prison in 1965 when the U . S . Army closed its post at Fort Jay and moved Headquarters , First United States Army from Governors Island to Fort Meade , Maryland . Final years of use and preservation . The U . S . Coast Guard arrived on Governors Island in 1966 and initially considered demolishing the castle . Instead it became a community center that provided a nursery , meeting rooms for scouts and clubs , a woodworking shop , art studios , a photography laboratory , and a museum . With the relocation of those civilian functions to new locations on the island in the mid - 1970s , the castle ended its military career in a state of mild neglect as a storage facility and landscape shop for the Coast Guard . When the Governors Island Coast Guard base closed in 1997 , the General Services Administration stabilized the building with replacement windows and a new roof . In 2003 , Castle Williams and the neighboring fortification , Fort Jay , were transferred to the National Park Service under the administration of the Governors Island National Monument . It is listed on the National Register of Historic Places .