Oftentimes a hurricane will leave behind broken but living coral to recover in the clear calm aftermath of the storm. Sediment will be churned up and cloud the water during the storm, but usually a return to calm conditions will settle it back out of the water column. Continuous wind, although it doesn't cause the structural damage of a hurricane, poses its own threats to the coral reef. Wave action can break up branching corals. Wind also make human impacts more possible.
Boats are more likely to hit the reef when they are struggling against wind.
Diver and snorkelers, if they can get out to the reef, are more likely to accidentally touch coral if the visibility is poor, and more likely to stand on it or bump against it if they are struggling with waves. But probably the most severe impact of a windy season is something as innocuous as cloudy water. Sediment suspended in the water column doesn't get a chance to settle. Without sunlight, the natural symbiosis between the coral and its algae partner are thrown out of whack. Photosynthesis is interrupted.