Well, no, it isn't fair, but there is a way out. A low expense ratio is simply the most important reason a fund does well. Management fees are annual fees charged by all funds, and you want to maker sure that management fees are as low as possible. The average fee, by the way, has actually been climbing in recent years. You also want to make sure that you aren't paying any sales charges .
Sales charges come in various stripes, also known as loads or commissions.
Avoid all of these. Some funds have back-end loads that are reduced the longer you hold the fund. Best to avoid all of these. If you have to buy an actively managed fund, buy the fund with no sales charges at all. 12b-1 fees are really remarkable.