You modified the agreement between you without the help of the court.  Good on one hand and bad on the other.  Now you need to go back to court to enforce the original agreement.  Generally you can do this yourself in family court.    I would, though, take whatever documents you signed as to the temporary order to an attorney to review and to consult with.  It is very difficult in this type of forum to give any guidance without the ability to read the paperwork you have and see exactly what you signed.  Your son is a good age to testify as to what he wants and where he wants to live.  The court will listen to him as long as the reasons are not something like "my dad is too strict."  You get my point.  Good luck.