Since you do not want any of these in your aquarium, removing them is important. You will also be able to look at the shape and size of all the rock and form yourself an idea as to how and where you are going to place all the live rock in the aquarium. This helps a great deal in shaping the look of your tank in advance. Once the live rock is in, it is not that easy to make large changes in how it is positioned. The exception to this is coralline algae, which should of course be left on the rock.
Coralline algae is very desirable in reef aquariums.
Note that your aquarium should be ready to receive the rock and sand you now have. The tank should not be full with water. When you add rock and sand, the water level will automatically rise. Fill the tank up to about half it height and have some reserve water. Have all your filtration in place and make sure that the protein skimmer is in place.