We stayed in the cave house for 2 nights and couldn't have been happier with it. The location is very central in oia. The porter that led us to our room was great and very informative. He explained everything about the room and helped us with a return taxi to the airport. We were met in our room shortly after check in by Maria (not the owner Maria, a different Maria) and she explained the island and all the activities we could find.
The room was well kept and extremely comfortable (especially after 10 days sailing) There was a nice breakfast spread and an excellent nescafe espresso maker. We had emailed Maria once during our stay and she was very quick to respond.
It is a little bit of a hike down to the house if you have some mobility/health issues that would be something to keep in mind but it is so worth it.