 In 1940, Hearst decided to give the monastery away. The government of Francisco Franco requested that it be returned to Spain, but Hearst refused. In August 1941, Herbert Fleishhacker, director of the M. H. de Young Memorial Museum, convinced Hearst to give the stones to the City of San Francisco in exchange for the city's payment of his $ 25,000 storage debt. Hearst stipulated the stones be used to construct a group of museum buildings adjoining the de Young Museum in Golden Gate Park. The city moved the crates from the warehouse to store them outdoors behind the museum and the Japanese Tea Garden, allotting only $ 5,000 for trucking and for building rough sheds and protective covers. The museum plan was estimated to cost $ 500,000, but that amount was not available. Morgan prepared several layouts for the city to approve, each with a different arrangement of the buildings than in Spain. However, in December 1941, the U.S. was drawn into World War II and the museum plans were shelved. Picking up the project in 1946, the city paid Morgan to construct a scale model of the complex of buildings which was to be the Museum of Medieval Arts, a West Coast version of The Cloisters in New York.
