Size is another. Myelinated axons are all large, and the myelination makes them larger. The space available to contain axons is not unlimited - nerves have to be able to fit inside the body parts they subserve. So the cost of making all axons large and myelinated to achieve uniform maximised speed is a smaller number of axons. Fewer axons means poorer control of effectors and poorer sensation.
Poor sensation can be just as big an evolutionary threat as low ground speed.
So while there is an evolutionary benefit to fast axons, there is also a benefit to compromises where some axons are slow and space efficient, allowing for more parallel paths, and richer information capabilities at a compromised speed. The conduction velocity vs. axon diameter graph that follows is instructive. The myelinated fibre diameter used is that including the myelin. It is most useful to examine the lower end of this curve. Myelination gives faster speeds only above about 1um diameter.