This is a terrific--if tiny!--place to stay! Views are *spectacular.* Rena is extremely nice and accommodating, with a welcoming bottle of wine and local cake etc., and the Caverne feels like a true "cave house," built into the wall of the caldera. The porter is amazing, carrying multiple suitcases up and down tiny, steep stairs, and he was very accommodating, as well. We were surprised and pleased to find we had daily cleaning services--it felt a bit more like a hotel than an airbnb, which was quite welcome, especially as we went through towels at an alarming rate because of beach ventures. The soap, shampoo and conditioner, and shower gel were also appreciated.
A couple of notes of caution: You should be warned that the beds are rather hard and the pillows are especially firm--we have definitely had more comfortable sleeping accommodations elsewhere in Greece. The air conditioner in the downstairs bedroom is also quite noisy, and the entire place really is very tiny--it's not a place designed to hang out inside. It's a great base of operations (and is wonderful to wake up to the beautiful views), but is not a home base to relax in. The balconies outside are nice, but during the August heat, were also not places we wanted to spend hours in, even when shaded. Oddly, mornings were often the hottest part of the day, so breakfast on the terrace was less charming than you might otherwise expect. Finally, you should know that the house is on the caldera, but is not a place to view the sunset from, because of the local geography -- you have to walk about 6-7 minutes to the other part of the caldera to do that. This was fine with us, so not a criticism; just make sure that if you're desperate to stay in a place that you can watch the sunset from your terrace, this isn't it. (Did I mention that the views are nonetheless gorgeous? :-))
Oia is a charming and beautiful town, although definitely overrun by tourists. We found some great places to eat (especially Candouni, in the town, and Ammoudi Fish Tavern in Ammoudi Bay), and appreciated the chance to take the kayaks out into Ammoudi Bay for the sunset (and for swimming there multiple times).
In sum, we would strongly recommend Caverne des Poetes. Many thanks to Rena and her staff for a terrific stay!