It's worth it. It shows you the other, more authentic side of Key West and its varied history. Don't miss the houses were artists and writers lived and worked, for instance the Audubon House, and Hemingway's Spanish Colonial mansion, built in 1851 and set in a big tropical garden. It's easy to fall in love with the latter. More than forty still live at the estate.
Restaurants cover a variety of cuisines and the whole range from modest to expensive.
And the seafood, of course, is truly fresh. One must consider that space is at a premium on this small gem of an island, and many goods and supplies have to be brought in. Key West is the perfect spot for an extravagant weekend, but for a longer vacation, most people would probably combine a short stay in Key West with a longer stay on the mainland in Florida or at one of the many resorts and hotels hidden along the abundant beaches of the other Keys. These offer every watersport one can think of. The temperatures of the Keys are usually ten degrees lower than on the mainland, which makes them pleasant year around.