In June 1561 , when the town had 30,000 inhabitants , Philip II of Spain moved his court from Toledo to Madrid , installing it in the old castle .
In 1561 Felipe I moved the royal court from Toledo to Madrid .