Refactor recursive structure code to be cleaner, and work on remote refs