The zooxanthellae would be quickly eaten if isolated on a rock surface. Coral polyps are genetic clones of their parents, indeed of every ancestor that ever lived in its coral rock community. They live in limestone homes constructed on the tops of the abandoned abodes of their ancestors. The polyps in the above photo were living on a large rock colony built very slowly over many years. The amount of annual growth depends on the success of the algae crop for that year.
Some years are good, some are bad.
The average year produces a growth of only a few centimeters of added height. The answer lies in its density of polyp growth rather than on its genus or family. Sometimes two members of the same family can be a star coral and a brain coral, respectively. Star coral polyps live far enough apart that they can make individual flower-shaped homes. Brain coral polyps grow so densely packed that their homes run into each other.