The rate at which the fuel is being burned is proportional to the luminosity. Less massive stars have somewhat less fuel but are burning it much more slowly. Thus they longer lifetimes than the Sun. More massive stars have somewhat more fuel but are burning it much faster. Thus they shorter lifetimes than the Sun.
Astronomers have a model of stars as giant nuclear reactors that burn hydrogen to make helium.
This model is backed up with detailed computer models based on physical principles. One piece of evidence for this picture is that the stellar models do a good job of predicting the existence of the main sequence and where it should be on the HR diagram. Another kind of evidence comes from looking at clusters of stars. Here is a sketch of what one sees for the HR diagram of a cluster of the type known as a globular cluster. The evidence from this diagram is that it is about 10 billion years old.