Generally speaking, no he can not.  If the court awarded you temporary possession it is probably until a final determination is made or the court reverses the order upon his application.  Read the order.  It will govern the actions of both parties.  Good luck.