There are very good technical climbers who don't move skillfully on rough terrain, and vice versa. You must become skilled at scrambling. There's a psychological component to this, helping you deal with heights and exposure over a long period of time. There are also specific movement skills for moving over rough terrain, whether walking or climbing, that must be learned, and many people don't adequately appreciate that. Skilled scramblers are much more efficient and faster in moving over the ground.
Skilled scramblers can make quick decisions and have fluidity of movement.
Skilled scramblers have the confidence to face out while descending, which is much faster and less energy consuming than facing in. Plain old physical fitness is obviously important. The ability to put out for long periods of time, is something that must be specifically trained for. You should train that way. In order to train for long days, you'll have to put in long days.