Direct damage means breaking corals and kicking up sediments. Sediments will settle on corals and plants and will reduce the amount of sunlight and fresh water they get. Just being there is an impact. Chose the place for your vacation by ecological criteria. Some organisms are harmful when touched, and your touch is no benefit to marine creatures.
Fish are wild animals.
Like all wild animals, they should stay this way. It is very tempting to feed fish in order to get a closer look. On Vilamandhoo in the Maldives, every evening people started to feed. As soon the first crumb of bread hit the water, there was a lot of action going on. Only one species of snappers.