Idlewild has been in ruins since it burned in April 2003 after vandals set it on fire. Idlewild again suffered damage from a smaller porch fire in August 2006. An earthquake on August 23, 2011 damaged the chimneys in the main house and leveled the two chimneys on the summer kitchen. The brick walls of the house have been braced with steel beams and remained basically unharmed by the quake. Considering, the house remains in stable condition.there are many theories to how the Downman house burned down the most likely being vandals. There are others such as homeless campers who set up a fire in the weakest fireplace.