It is very common for old letters, scrapbooks and photo albums to be tossed into the garbage or sold to a flea market dealer - especially after the last surviving relatives die. Most archives and some libraries hold photographic collections. Photographs are probably the best tool for identifying lost features on a house or building. Unfortunately, finding an early photograph of your house is more a matter of luck than certainty. Never overlook streetscape photos or family photos that may show your house in the background.
Aerial photographs can also be a useful.
City Hall Records Municipalities generally manage a great deal of information about the buildings within their jurisdictions. Look for Council minutes and committee reports. Planning, building and engineering departments often store maps, plans, blueprints, building specification documents, contracts and building and demolition permits. Fire department records often include old fire logs and other interesting material. If your lucky, your community will have transferred its early municipal records to the local archives.