And while removing a point that has a large residual may lead to a smaller residual variance for the new fitted linear function, it will not necessarily lead to a greater R-square value for the new fitted model, or to a smaller P value for the F test of overall fit. Such methods assume that you have identified all the reasonable candidate X variables, and only need to choose among them. One method is simply to perform all possible linear regressions, which may be feasible if the number of candidate X variables is small. If there are too many X variables to examine each possible regression, then stepwise regression is often used. At each step, the decision to add or drop an X variable is based on a test of whether that variable will or does make a statistically significant contribution to the model.
Stepwise regression identifies a single regression instead of several possible candidates.
The particular set of X variables suggested by a mechanical method, especially a stepwise multiple regression, may be very dependent on the specific data values observed for X and Y. Such models should always be validated. First, make sure that the model makes sense theoretically, and is comparable to any results from fitting to other data sets. Then, try the model out on new data to see if it still holds. One validation method is to divide the data set into two parts, using one to fit the equation, and the other to decide whether it is a reasonable model. Validation is vital when using a stepwise procedure.