When Vernon Smith and his older brother David bought The Brewster Fish House in 1982, it was a fish market that had previously been a gladiola farmstand and, based on two overhead hydraulic doors, part of a commercial garage.