You would have to find out what they are, then find out if that substance is harmful to marine life. Well, if it is quartz, then that is silicon, and will dissolve into the water at around 12ppm, this your diatoms would love if silicon is the limiting nutrient. The rock that I use comes from Mount Gambier here in Australia, and appears to have no contaimination at all. I don't currently know of any, but I can soon find out for you. The source that I have used typcially supplies limestone blocks for housing.
Limestone is just ancient marine skeleton etc, i.e old reef.
You can see the old urchin shells, fish, etc through out it. So what ever is part of a coral skeleton has the possiblity of leaching out. I carved it, then washed it with fresh tap water to remove all the dust generated during the carving process. That is all you have to do. You could, but it would never be consistant, and a heap of dust would be generate.