We had a fantastic visit to Port Fairy staying at Virginia and Dan's place. Very friendly hosts and clean, cosy accommodation. We took our bikes which allowed for easy venturing and access to beaches and Main Street, however Virginia and Dan's place is in walking distance to everything.
We would highly recommend staying with these guys, lovely people, lovely place!