No, there are no negative consequences for visiting realms out of order. However, some aspects of the story may not make as much sense if you do this.