Individual websites elements may be more intertwined than they look at the first sight. One metric you improve may result in many other metrics with deteriorated results - for that reason there is need to set proper main KPI. If useful, there is possible to also set less-important smaller KPIs which may indicate the state of the route for achievement of the main KPI. Then, after deployment of the more versions within A/B/../X test you track and evaluate changes of set KPIs between individual web versions.
Example:
You got an idea to increase the number of visited pages per session on your website. With a use of common sense, you realize that you could try to achieve that by a replacement of an ad unit displayed in exclusive space of your web for a section offering recommended articles. By the change, the number of pages per session will probably be increased and KPI filled, however, with a side effect in a form of affected web revenue which is usually a very important parameter. Thus, generated revenue should be another KPI parameter (maybe even main) for correct evaluation of the test.