Do not actually rotate the astroid objects, only rotate the model. Else they are network replicated all the time.