Baron's Cove is about 0 mi from Sag Harbor Whaling Museum, one of Sag Harbor’s most popular landmarks.