You can walk to Kali Amman Temple while staying at Joe Fisherman Inn - you’ll be about 2 mi away.