Minamoto clan
Per local folklore, the island was one of the havens of the defeated Heike clan after they lost the Genpei War against the Minamoto clan.