A younger horse is too young to break, and a mature one doesn't respond as fully to conditioning. An eight year old, overweight individual that has spent his life standing in a stall, stuffing himself on alfalfa and sweetfeed, is going to take a long time to get fit. Still, if you see a horse that you really like, don't dismiss him simply because of age. Endurance horses often compete well into their teens, so there might still be time to get a lot of enjoyment from him. There is no ideal size for an endurance horse.
Ponies can do very well, as can horses of sixteen hands or so.
Most good endurance horses run around fifteen hands, but since this is a common height for Arabians, this measurement probably has no special significance. The size of the horse should be appropriate for the size of the rider, however. No one could expect a pony to be competitive carrying a two hundred pound man. Small, lightweight riders obviously have an advantage in that they have a greater range of sizes from which to choose. An endurance horse should have grown up in a situation that allows him to express his equine nature.