Model Validation Using Mutated Training Labels: An Exploratory Study

1 INTRODUCTION . Out-of-sample validation ( such as test accuracy ) is arguably the most popular approach adopted by researchers and developers for empirically validating models in applied machine learning . It uses data different from the training data to approximate future unseen data . However , out-of-sample validation is widely acknowledged to have limitations : 1 ) the sample set may be too small to represent the data distribution ; 2 ) the accuracy can have a large variance across different runs ( Pham et al. , 2020 ) ; 3 ) the samples are typically randomly selected from the collected data , and may therefore have similar bias as in the training data , leading to an inflated validation score ( Piironen & Vehtari , 2017 ; Gronau & Wagenmakers , 2019 ) ; 4 ) excessive reuse of a fixed set of samples can lead to overfitting even if the samples are held out and not used in the training process ( Feldman et al. , 2019 ) . The Mutation Validation ( MV ) approach we explore is a new approach to validating machine learning models relying only upon training data . MV applies Mutation Testing and Metamorphic Testing —two software engineering techniques that validate code correctness . Mutation testing mutates the program by making synthetic changes ( e.g. , to change a + b into a − b or to remove a functional call ) , then re-executes the tests to monitor the behaviour changes and check the power of a test suite ( Papadakis et al. , 2019 ; Jia & Harman , 2010 ) . Metamorphic Testing detects program errors by checking metamorphic relations , which is the relationship between input changes and output changes ( Chen et al. , 2020 ; Segura et al. , 2016 ) . Combining these two techniques , MV mutates training data labels and retrains the model using the mutated data , then measures the training performance change . As shown by Figure 1 , the key intuition is that a learner , if fitting the given training data property , would be less likely to be ‘ fooled ’ by a small ratio of mutated labels . Consequently , the model trained with the mutated training data would “ detect ” the mutated labels and still exhibit high predictive performance on the original training data . By contrast , an overfitted learner violates the Occam ’ s Razor ( Hawkins , 2004 ) , and has extra capacity to fit incorrect noisy labels , and thus will yield a model that exhibits poor predictive performance on the original training data , but high performance on the mutated training data . Furthermore , an over-simple learner has poor learnability , and will have low performance on the original training data and the mutated data . There are a significant number of theories proposed to explain model validation and complexity , such as Rademacher complexity and VC dimension . Nevertheless , the prescriptive and descriptive value of these theories remains debated . Zhang et al . ( 2017 ) found that deep hypothesis spaces can be large enough to memorise random labels . They discussed the limitations of existing measurements in explaining the generalisation ability of large neural networks , and called for new measurements . original data mutated data good learner over-complex learner over-simple learner expected PV performance train acc . train acc . train acc . noise degree noise degree noise degree 00 ( 00 ^ 00 X X 0 \ °° { oo o µ + go °\ X O O X O o : :÷ : ÷÷÷ : ¥ : ÷ : : : ÷÷ : * : :÷ : * × I ^ 00 X X 0 / I ¥ : O . × 0 × 0 0 : i÷÷÷ : ÷ : : : :÷÷ : ÷ : xx × X X X ÷ ! .. →÷÷ ; ; t * ÷÷÷i '' : ! ÷÷÷a . | ° ° ° `` 5 } | ° ° ° ° ° 0=1-7 0 0 0.1 0.2 0.3 Out 0,2 0.3 0.1 0.2 0.3 Figure 1 : The intuition underpinning MV : A better learner is less affected by the mutated labels . In this present work , we do not compare MV with these theories . Rather , similar to Zhang et al . ( 2017 ) , we focus on empirical investigation . In Appendix A.1 , we provide the theoretical foundations underpinning the metamorphic relation that MV uses . We report on the performance of MV on 12 open datasets and 6 synthetic datasets with different known data distributions ( see Table 1 ) , using 8 widely-adopted classifiers ( including both classic learning classifiers and deep learning classifiers ) . We investigate the effectiveness and stability of MV serving as a complementary measure to the existing practical model validation methods for model selection and parameter tuning . The experimental results provide evidence to support the following conclusions . First , MV captures well the degree of match between decision boundaries and data patterns in model selection . The model recommendation hit rate for MV is 92 % , but is 53 % for cross validation , and 55 % for test accuracy . Second , MV is more responsive to changes in capacity than conventional validation methods . When cross validation ( CV ) accuracy and test accuracy could not distinguish among large-capacity hyperparameters , MV complements them in hyperparameter tuning . Third , MV is stable in model validation results . Its dropout rate tuning result does not change across five runs ; the variance is zero . For validation set and test set , the average variance is 0.003 and 0.007 , respectively . The paper also discusses the connections between MV and other noise injection work in the literature , as well as the usage scenarios of MV ( Section 5 ) . In summary , we make the following primary contributions : 1 ) An exploratory study on the performance of Mutation Validation . We explore the effectiveness and stability of MV to validate machine learning ( ML ) models as a complement to the currently used empirical model validation methods . MV requires neither validation nor test sets , but uses the training performance sensitivity to the mutated training labels . We study 18 datasets , 8 different learning algorithms , and 5 hyperparameter tuning tasks . 2 ) An application of software testing techniques in ML model validation . MV is the first approach that applies mutation testing and metamorphic testing —two widely studied software testing techniques —on model validation tasks . 2 MUTATION VALIDATION . 2.1 GENERAL INTUITION . Figure 1 illustrates the intuition that underpins MV . For a ‘ good ’ learner that fits the training data well ( first column of Figure 1 ) , the learner is less likely to be ‘ fooled ’ by a small number of mutated labels , and would keep predicting the original labels for the mutated samples . As a result , the model trained on the mutated data will still have high predictive accuracy on the original training data , but will have decreased predictive accuracy on the mutated data . An overfitted learner tends to fit noise in the training data ( second column of Figure 1 ) . With mutated data , the learner will yield a model that makes predictions following the incorrect labels , leading to a high training accuracy on the mutated data , but poor accuracy on the original data . An over-simple learner has poor learnability . The model it yields has poor performance with or without the mutated training data . As a result , the model trained on the mutated data has low accuracy with both the original labels and the mutated labels . 2.2 MUTATION VALIDATION . MV uses mutation testing and metamorphic testing , two software validation techniques , to validate ML models . Mutation testing creates mutants by injecting faults in a program , then re-executes the program to check whether a test suite detects those faults . Metamorphic relation specifies how a change in the input should result in a change in the output . It is used to detect errors in software when there are no reliable oracles . For a program f , its input x and x′ . Let f ( x ) and f ( x′ ) be the execution outputs of x and x′ against f . Let Ri be the relationship between x and x′ , Ro is the relationship between f ( x ) and f ( x′ ) . A metamorphic relation can be represented as : Ri ( x , x′ ) ⇒ Ro ( f ( x ) , f ( x′ ) ) . If the relation is violated , the program under test contains a bug . For example , when validating the sin mathematical function , one metamorphic relation that a correct program should hold is sin ( x+ π ) = − sin ( x ) . Now consider the scenario of ML model validation . If we treat a learner as the program under test , the training data as the input , the trained model ’ s behaviours as the output , our intuition introduced in Section 2.1 can be well captured by mutation testing and metamorphic relations . In particular , the input changes are introduced by mutating training data , each mutated data instance is called a mutant ; the output changes are defined in terms of performance changes of the trained models . Based on this , we propose Mutation Validation , a new machine learning model validation method that validates ML models based on the relationship between training data changes and training performance changes . A good learner is expected to “ detect ” the mutants and have a certain amount of training performance changes according to the number of input data changes . There are different metamorphic relations that can be explored to conduct MV , which may depend on the data mutation method , the training performance measurement ( e.g. , accuracy , precision , loss ) , and the calculation of performance changes . Let η be the mutation degree ( i.e. , the ratio of randomly mutated labels in the training data ) . S is the original training data , Sη is the mutated training data with mutation degree η ( η ≤ 0.5 ) , f ( S ) is the model output trained on S , f ( Sη ) is the model output trained on Sη , T̂S ( f ( S ) ) , T̂S ( f ( Sη ) ) are the accuracy of f ( S ) and f ( Sη ) based on the original training labels , respectively . T̂Sη ( fS ( η ) ) is the accuracy of f ( Sη ) based on the mutated labels . In this present work , as the first exploratory study on MV , we study the following MV measurement m to conduct model validation : m = ( 1− 2η ) T̂S ( f ( Sη ) ) + T̂S ( f ( S ) ) − T̂Sη ( f ( Sη ) ) + η . ( 1 ) The above measurement , although derived from theory ( Appendix A.1 ) , matches well with our intuition introduced in Section 2.1 . In particular , if the learner is less affected by the mutated labels , the predictive behaviours of the trained model with mutated labels should be closer to that of the model trained with the original labels . This leads to a larger T̂S ( f ( Sη ) ) and a larger difference between T̂S ( f ( S ) ) and T̂Sη ( f ( Sη ) , as long as the mutation degree η is fixed . The larger m is , the better the learner fits the training data . In the optimal case , the trained model on mutated data has a perfect training accuracy on the original training labels ( i.e. , T̂S ( f ( Sη ) ) = 1 ) , and detects all the mutants ( i.e , T̂S ( f ( S ) ) − T̂Sη ( f ( Sη ) ) = η ) . Thus , m = 1 . Mutation degree η ranges between 0 and 0.5 , but needs to be a fixed value . The theory inspiration of this metric , its metamorphic relation , as well as the influence of η on m are provided in our appendix .

Summary: This paper introduces new approach for validation of ML models which is based on top of ideas imported from software development & QA (so-called mutation and metamorphic techniques). The core claimed advantage consist in the ability to avoid cross-validation and validation and test sets (the famous approach to avoid overfitting while learning a ML model). The work contributes also a huge empirical study with experiments for dozens of datasets and several models and hyperparameter tuning.