Model Validation Using Mutated Training Labels: An Exploratory Study

1 INTRODUCTION . Out-of-sample validation ( such as test accuracy ) is arguably the most popular approach adopted by researchers and developers for empirically validating models in applied machine learning . It uses data different from the training data to approximate future unseen data . However , out-of-sample validation is widely acknowledged to have limitations : 1 ) the sample set may be too small to represent the data distribution ; 2 ) the accuracy can have a large variance across different runs ( Pham et al. , 2020 ) ; 3 ) the samples are typically randomly selected from the collected data , and may therefore have similar bias as in the training data , leading to an inflated validation score ( Piironen & Vehtari , 2017 ; Gronau & Wagenmakers , 2019 ) ; 4 ) excessive reuse of a fixed set of samples can lead to overfitting even if the samples are held out and not used in the training process ( Feldman et al. , 2019 ) . The Mutation Validation ( MV ) approach we explore is a new approach to validating machine learning models relying only upon training data . MV applies Mutation Testing and Metamorphic Testing —two software engineering techniques that validate code correctness . Mutation testing mutates the program by making synthetic changes ( e.g. , to change a + b into a − b or to remove a functional call ) , then re-executes the tests to monitor the behaviour changes and check the power of a test suite ( Papadakis et al. , 2019 ; Jia & Harman , 2010 ) . Metamorphic Testing detects program errors by checking metamorphic relations , which is the relationship between input changes and output changes ( Chen et al. , 2020 ; Segura et al. , 2016 ) . Combining these two techniques , MV mutates training data labels and retrains the model using the mutated data , then measures the training performance change . As shown by Figure 1 , the key intuition is that a learner , if fitting the given training data property , would be less likely to be ‘ fooled ’ by a small ratio of mutated labels . Consequently , the model trained with the mutated training data would “ detect ” the mutated labels and still exhibit high predictive performance on the original training data . By contrast , an overfitted learner violates the Occam ’ s Razor ( Hawkins , 2004 ) , and has extra capacity to fit incorrect noisy labels , and thus will yield a model that exhibits poor predictive performance on the original training data , but high performance on the mutated training data . Furthermore , an over-simple learner has poor learnability , and will have low performance on the original training data and the mutated data . There are a significant number of theories proposed to explain model validation and complexity , such as Rademacher complexity and VC dimension . Nevertheless , the prescriptive and descriptive value of these theories remains debated . Zhang et al . ( 2017 ) found that deep hypothesis spaces can be large enough to memorise random labels . They discussed the limitations of existing measurements in explaining the generalisation ability of large neural networks , and called for new measurements . original data mutated data good learner over-complex learner over-simple learner expected PV performance train acc . train acc . train acc . noise degree noise degree noise degree 00 ( 00 ^ 00 X X 0 \ °° { oo o µ + go °\ X O O X O o : :÷ : ÷÷÷ : ¥ : ÷ : : : ÷÷ : * : :÷ : * × I ^ 00 X X 0 / I ¥ : O . × 0 × 0 0 : i÷÷÷ : ÷ : : : :÷÷ : ÷ : xx × X X X ÷ ! .. →÷÷ ; ; t * ÷÷÷i '' : ! ÷÷÷a . | ° ° ° `` 5 } | ° ° ° ° ° 0=1-7 0 0 0.1 0.2 0.3 Out 0,2 0.3 0.1 0.2 0.3 Figure 1 : The intuition underpinning MV : A better learner is less affected by the mutated labels . In this present work , we do not compare MV with these theories . Rather , similar to Zhang et al . ( 2017 ) , we focus on empirical investigation . In Appendix A.1 , we provide the theoretical foundations underpinning the metamorphic relation that MV uses . We report on the performance of MV on 12 open datasets and 6 synthetic datasets with different known data distributions ( see Table 1 ) , using 8 widely-adopted classifiers ( including both classic learning classifiers and deep learning classifiers ) . We investigate the effectiveness and stability of MV serving as a complementary measure to the existing practical model validation methods for model selection and parameter tuning . The experimental results provide evidence to support the following conclusions . First , MV captures well the degree of match between decision boundaries and data patterns in model selection . The model recommendation hit rate for MV is 92 % , but is 53 % for cross validation , and 55 % for test accuracy . Second , MV is more responsive to changes in capacity than conventional validation methods . When cross validation ( CV ) accuracy and test accuracy could not distinguish among large-capacity hyperparameters , MV complements them in hyperparameter tuning . Third , MV is stable in model validation results . Its dropout rate tuning result does not change across five runs ; the variance is zero . For validation set and test set , the average variance is 0.003 and 0.007 , respectively . The paper also discusses the connections between MV and other noise injection work in the literature , as well as the usage scenarios of MV ( Section 5 ) . In summary , we make the following primary contributions : 1 ) An exploratory study on the performance of Mutation Validation . We explore the effectiveness and stability of MV to validate machine learning ( ML ) models as a complement to the currently used empirical model validation methods . MV requires neither validation nor test sets , but uses the training performance sensitivity to the mutated training labels . We study 18 datasets , 8 different learning algorithms , and 5 hyperparameter tuning tasks . 2 ) An application of software testing techniques in ML model validation . MV is the first approach that applies mutation testing and metamorphic testing —two widely studied software testing techniques —on model validation tasks . 2 MUTATION VALIDATION . 2.1 GENERAL INTUITION . Figure 1 illustrates the intuition that underpins MV . For a ‘ good ’ learner that fits the training data well ( first column of Figure 1 ) , the learner is less likely to be ‘ fooled ’ by a small number of mutated labels , and would keep predicting the original labels for the mutated samples . As a result , the model trained on the mutated data will still have high predictive accuracy on the original training data , but will have decreased predictive accuracy on the mutated data . An overfitted learner tends to fit noise in the training data ( second column of Figure 1 ) . With mutated data , the learner will yield a model that makes predictions following the incorrect labels , leading to a high training accuracy on the mutated data , but poor accuracy on the original data . An over-simple learner has poor learnability . The model it yields has poor performance with or without the mutated training data . As a result , the model trained on the mutated data has low accuracy with both the original labels and the mutated labels . 2.2 MUTATION VALIDATION . MV uses mutation testing and metamorphic testing , two software validation techniques , to validate ML models . Mutation testing creates mutants by injecting faults in a program , then re-executes the program to check whether a test suite detects those faults . Metamorphic relation specifies how a change in the input should result in a change in the output . It is used to detect errors in software when there are no reliable oracles . For a program f , its input x and x′ . Let f ( x ) and f ( x′ ) be the execution outputs of x and x′ against f . Let Ri be the relationship between x and x′ , Ro is the relationship between f ( x ) and f ( x′ ) . A metamorphic relation can be represented as : Ri ( x , x′ ) ⇒ Ro ( f ( x ) , f ( x′ ) ) . If the relation is violated , the program under test contains a bug . For example , when validating the sin mathematical function , one metamorphic relation that a correct program should hold is sin ( x+ π ) = − sin ( x ) . Now consider the scenario of ML model validation . If we treat a learner as the program under test , the training data as the input , the trained model ’ s behaviours as the output , our intuition introduced in Section 2.1 can be well captured by mutation testing and metamorphic relations . In particular , the input changes are introduced by mutating training data , each mutated data instance is called a mutant ; the output changes are defined in terms of performance changes of the trained models . Based on this , we propose Mutation Validation , a new machine learning model validation method that validates ML models based on the relationship between training data changes and training performance changes . A good learner is expected to “ detect ” the mutants and have a certain amount of training performance changes according to the number of input data changes . There are different metamorphic relations that can be explored to conduct MV , which may depend on the data mutation method , the training performance measurement ( e.g. , accuracy , precision , loss ) , and the calculation of performance changes . Let η be the mutation degree ( i.e. , the ratio of randomly mutated labels in the training data ) . S is the original training data , Sη is the mutated training data with mutation degree η ( η ≤ 0.5 ) , f ( S ) is the model output trained on S , f ( Sη ) is the model output trained on Sη , T̂S ( f ( S ) ) , T̂S ( f ( Sη ) ) are the accuracy of f ( S ) and f ( Sη ) based on the original training labels , respectively . T̂Sη ( fS ( η ) ) is the accuracy of f ( Sη ) based on the mutated labels . In this present work , as the first exploratory study on MV , we study the following MV measurement m to conduct model validation : m = ( 1− 2η ) T̂S ( f ( Sη ) ) + T̂S ( f ( S ) ) − T̂Sη ( f ( Sη ) ) + η . ( 1 ) The above measurement , although derived from theory ( Appendix A.1 ) , matches well with our intuition introduced in Section 2.1 . In particular , if the learner is less affected by the mutated labels , the predictive behaviours of the trained model with mutated labels should be closer to that of the model trained with the original labels . This leads to a larger T̂S ( f ( Sη ) ) and a larger difference between T̂S ( f ( S ) ) and T̂Sη ( f ( Sη ) , as long as the mutation degree η is fixed . The larger m is , the better the learner fits the training data . In the optimal case , the trained model on mutated data has a perfect training accuracy on the original training labels ( i.e. , T̂S ( f ( Sη ) ) = 1 ) , and detects all the mutants ( i.e , T̂S ( f ( S ) ) − T̂Sη ( f ( Sη ) ) = η ) . Thus , m = 1 . Mutation degree η ranges between 0 and 0.5 , but needs to be a fixed value . The theory inspiration of this metric , its metamorphic relation , as well as the influence of η on m are provided in our appendix .

Summary: The paper introduces a new method for model validation called Mutation Validation (MV). Briefly, MV mutates training data labels and trains the model on both original and mutated training data. It then uses both these models to get a score that is a function of (1) difference in training accuracy of models trained on original data and training accuracy of models trained on mutated data and (2) Accuracy of model trained on mutated data based on the original training data. The paper explains the larger the score the better the learner fits the training data and hence can be used for model validation.  The paper provides an empirical study to show the effectiveness of MV by comparing it with Cross Validation (CV). In this study they explore 8 different learning algorithms, 18 datasets and 5 types of hyper-parameter tuning tasks.