Great place to explore the Otago Peninsula with kids:
There is plenty of space in the house, things to play with and books he enjoyed for our 4-year-old, beds are very comfortable, there was even Tupperware so we could take food on our adventures, we loved the blue penguin tour (10 min from the house just like the Albatrosses) and some small hikes on the peninsula.
We went into Dunedin on a day trip, too.
PS: If you want to grab fish and chips in the shops nearby on a Sunday you have to be much earlier than 15 min before they close: There was such a queue they did not take orders, which gives me the feeling that they are good… ;-)