Switched the activity_main.xml layout files to LinearLayouts and switched the order of the panels around. This has solved the issue of what order things were appearing in.