Early in the siege of Boston that began after the battles of Lexington and Concord in April 1775, the house was occupied by troops that eventually became part of the Continental Army; one of the building's notable occupants during this time was Benedict Arnold, then in the Connecticut militia. After the Battle of Bunker Hill it was used as a hospital. When the British military evacuated Boston in March 1776, the Olivers, like many other Loyalists, traveled with them to Nova Scotia. Oliver eventually settled in Bristol, England, where he died in 1815.