Three essays on outlier detection in auditing: applying machine learning to accounting data for full-population testing
Creators: name: Wei, Danyang
Publisher: name: No Publisher Supplied
Year: 2024

description: Full-population testing in auditing has drawn much attention today due to technological advancements. A branch of machine learning literature, unsupervised outlier detection, appears to be a suitable tool for auditors to discover abnormalities in large accounting data such as general ledger (GL). It identifies outliers that deviate from the majority of observations in a dataset without utilizing any prior knowledge. The algorithms learn normal patterns from the data and calculate an outlier score for each observation based on its deviation from the normal patterns. A large score indicates a higher probability of being an outlier. Applications in various domains, such as credit card fraud detection, prove useful insights can be generated through the identified outliers. However, the application of outlier detection in auditing has not been thoroughly explored in existing literature. Since unsupervised outlier detection is not guided by prior knowledge, investigation of the identified outliers provides a chance for auditors to discover new potential problems. However, “there ain’t no such thing as a free lunch” – the credibility of a detection algorithm is hard to evaluate without prior knowledge about the actual outliers. When various algorithm candidates flag different observations as outliers, it is almost impossible for auditors, who have limited investigation capacity, to manually examine all the outliers. Therefore, to establish outlier detection as a viable technique in auditing, it is imperative to effectively address the evaluation difficulty. Additionally, insights obtained through the investigation of outliers are valuable and have the potential to benefit future audits. From this perspective, auditors can employ supervised classifiers for outlier detection purposes. Specifically, investigated outliers from previous years can be collected to form a labeled data pool and utilized to train supervised classifiers for outlier prediction in the current year. As the investigated outliers accumulate year by year, the classifiers learn from more experience, which may improve their prediction accuracy. However, some factors, such as the quality of the initial training set, may negatively affect the prediction accuracy. Hence, more research is required to examine the impact of relevant factors in this specific auditing application scenario on overall prediction accuracy so that auditors can gain some insights into the reliability of such a methodology. To explore how to apply machine learning to accounting data for full-population testing, this dissertation 1) proposes a framework to identify various types of outliers in general ledger data and tests whether new insights are gained through the identified outliers, 2) studies properties of outlier score distributions and proposes an internal validation solution, and 3) studies whether investigated transactions in previous periods can be used to train supervised classifiers so that they can identify more real outliers in the current period. The first essay (Chapter 2) proposes a Multilevel Outlier Detection Framework (MODF) that identifies outliers at three levels: transaction level, account level, and combination-by-variable level, respectively, with each level representing a specific type of outliers in general ledger (GL). The framework is tested with both real and synthetic datasets to test its effectiveness in gaining new insights. For the real GL dataset, an external audit team from one of the world’s largest audit firms partnered with our research team to provide both the data and the feedback on the outlier detection results. The synthetic GL dataset used is from a case study created by Ernst Young Academic Resource Center (EYARC), a clean dataset with no fraud. To test whether outlier detection can capture errors that are not obvious to auditors, we create seeded errors in a way that the variable values fall in the same ranges as the original observations in the dataset, but the dependency structure between the variables is changed. Since the dependency structure is not directly observable, such seeded errors are difficult to identify by auditors’ judgment. The performance of the framework on the two datasets indicates that outlier detection can be utilized to gain new insights regarding the risk of misstatements in GL data. The second essay (Chapter 3) focuses on the main challenge of applying outlier detection in auditing – algorithm evaluation with no ground truth. In particular, the properties of outlier score distributions are studied for internal validation. When scores from an algorithm are normalized and ranked in descending order, how the scores are distributed between one and zero is referred to as a score distribution. The slope at an observation is the score difference between this observation and its immediate right neighbor. We find that if an algorithm can successfully identify real outliers, their score distributions always involve a shift in the pattern of neighboring score differences. It motivates us to develop a hypothesis that the shift signifies the transmission from the identified real outliers to the remaining observations, and the location of the shift has a positive relationship with algorithm performance. We conduct a proof of concept to test the hypothesis using six methods and their parameterizations on real and synthetic data. The results indicate that a positive relationship exists. We then propose an internal validation solution called the Index of Score Distribution Breakpoint (ISDB) to estimate the location of the shift in a given score distribution. The algorithm with the largest ISDB is selected. The evaluation results show that the algorithm selected by the ISDB methodology tends to perform well. The third essay (Chapter 4) explores whether auditors’ previous experience with a client can be utilized to train supervised learning algorithms for outlier detection. Specifically, as auditors investigate a sample of transactions in each audit period, those investigated transactions can be used to train supervised classifiers for future prediction. We study two main factors that can affect the prediction accuracy of supervised classifiers to provide some insights into the reliability of such a methodology: data similarity across different periods and quality of the initial training set. Particularly, we design an experiment to examine the impact of the data similarity across different periods, the class balance in the initial training set, the proportion of the initial training set in the first period, and the percentage of labeled real outliers among all the real outliers in the first period on prediction accuracy of supervised classifiers. The experiment results with seven supervised classifiers suggest that the data similarity and the percentage of labeled real outliers can significantly influence the overall prediction accuracy. This indicates that auditors should construct high-quality initial training set to ensure a high prediction accuracy in subsequent periods.Overall, this dissertation examines the potential of using machine learning to identify suspicious observations in auditing, which aims to efficiently uncover risks that auditors do not expect. The findings of this dissertation have important implications for audit practitioners and regulators.