Query: thesis and error uncertainty

title: Error compensation and uncertainty evaluation of CMMs based on kinematic error models and gaussian processes
abstract: This thesis was submitted for the degree of Masters of Philosophy and awarded by Brunel University London.

title: Uncertainty estimation in deep learning with application to spoken language assessment
abstract: Since convolutional neural networks (CNNs) achieved top performance on the ImageNet task in 2012, deep learning has become the preferred approach to addressing computer vision, natural language processing, speech recognition and bio informatics tasks. However, despite impressive performance, neural networks tend to make over confident predictions. Thus, it is necessary to investigate robust, interpretable and tractable estimates of uncertainty in a model's predictions in order to construct safer Machine Learning systems. This is crucial to applications where the cost of an error is high, such as in autonomous vehicle control, high stakes automatic proficiency assessment and in the medical, financial and legal fields. In the first part of this thesis uncertainty estimation via ensemble and single model approaches is discussed in detail and a new class of models for uncertainty estimation, called Prior Networks, is proposed. Prior Networks are able to emulate an ensemble of models using a single deterministic neural network, which allows sources of uncertainty to be determined within the same probabilistic framework as in ensemble based approaches, but with the computational simplicity and ease of training of single model approaches. Thus, Prior Networks combine the advantages of ensemble and single model approaches to estimating uncertainty. In this thesis Prior Networks are evaluated on a range classification datasets, where they are shown to outperform baseline approaches, such as Monte Carlo dropout, on the task of detecting out of distribution inputs. In the second part of this thesis deep learning and uncertainty estimation approaches are applied to the area of automatic assessment of non native spoken language proficiency. Specifically deep learning based graders and spoken response relevance assessment systems are constructed using data from the BULATS and LinguaSkill exams, provided by Cambridge English Language Assessment. Baseline approaches for uncertainty estimation discussed and evaluated in the first half of the thesis are then applied to these models and assessed on the task of rejecting predictions to be graded by human examiners and detecting misclassifications.

title: Master Thesis: Biologically Inspired Monocular Visual Odometry
abstract: Determining a robots location with high precision is essential for it to allow an autonomous operation. The reason why this problem is so challenging is the uncertainty with which a robot moves and senses. The error in robot motion estimation, can lead to complete localization failures. One reason for that is slippage on the ground, an error which cannot be estimated by wheel encoders. During its operation, the mechanical parts of a robot wear out as well and introduce additional inaccuracy to the robots motion. One way to improve the motion estimation is to use an additional exterior sensor, like a monocular camera, in order to estimate the robots motion between time steps.

title: Quantitative methods for improved error detection in dose guided radiotherapy
abstract: With the increasing complexity of radiotherapy, it has become increasingly important to verify that the desired radiation dose is delivered as planned. This thesis focuses on Dose Guided Radiotherapy (DGRT) using portal dosimetry with the Electronic Portal Imaging Device (EPID) as a method for treatment verification. Automatic error detection and consequently patient selection for adaptive radiotherapy using portal dosimetry remains difficult. This is due to the unknown uncertainty of the portal dosimetry methods, as well as the lack of objective triggers for adaptation and the use of simple decision protocols (often consisting of an action threshold on a single metric) that cannot take the multidimensionality of the EPID dose distributions into account. The overall aim of this thesis was to investigate and improve the error detection capabilities of portal dosimetry methods. The first part of the thesis concerns uncertainties and sensitivity of portal dosimetry methods. A framework for analyzing uncertainties of portal dosimetry methods was developed, and factors influencing the uncertainty, and the sensitivity and specificity of portal dosimetry were investigated and quantified. It is important to incorporate this quantitative knowledge in decision protocols for adaptive radiotherapy. The second part focuses on advanced methods for error detection and classification, applying Artificial Intelligence (AI) to portal dosimetry. Several AI methods and classification problems were investigated to show the potential of AIbased error detection for portal dosimetry. First, it was shown that it is possible to relate 2D portal dosimetry results to differences in dose volume histogram metrics by using Convolutional Neural Networks (CNNs) a relationship that is notoriously difficult to find with traditional action threshold based classification methods. Second, it was demonstrated that deep learning is a promising powerful tool for identifying types of treatment errors with portal dosimetry. Errors can be identified to a high level of detail, and this approach can provide additional information not currently available from portal dosimetry. Third, external validation of a machine learning model for classifying anatomical changes showed that it is difficult to use such a model directly on clinical data from a different institute due to differences in data acquisition and clinical interpretation. Overall, portal dosimetry for DGRT can greatly benefit from improved error detection methods. This thesis has contributed to this improvement, by providing a framework for uncertainty analysis of portal dosimetry methods, by examining the sensitivity and specificity for action threshold based classification methods for various portal dosimetry methods, and by investigating various advanced AI algorithms and error classification problems. References to author publications that relate specifically to the dissertation: 1. Baeza JA, Wolfs CJA, Nijsten SMJJG, Verhaegen F. Validation and uncertainty analysis of a pre treatment 2D dose prediction model.  Verhaegen F. Should dose from small fields be limited for dose verification procedures? Uncertainty vs. small field dose in VMAT treatments.

title: Adaptive algorithms for partial differential equations with parametric uncertainty
abstract: In this thesis, we focus on the design of efficient adaptive algorithms for the numerical approximation of solutions to elliptic partial differential equations (PDEs) with parametric inputs. Numerical discretisations are obtained using the stochastic Galerkin Finite Element Method (SGFEM) which generates approximations of the solution in tensor product spaces of finite element spaces and finite dimensional spaces of multivariate polynomials in the random parameters. Firstly, we propose an adaptive SGFEM algorithm which employs reliable and efficient hierarchical a posteriori energy error estimates of the solution to parametric PDEs. The main novelty of the algorithm is that a balance between spatial and parametric approximations is ensured by choosing the enhancement associated with dominant error reduction estimates. Next, we introduce a two level a posteriori estimate of the energy error in SGFEM approximations. We prove that this error estimate is reliable and efficient. Then we provide a rigorous convergence analysis of the adaptive algorithm driven by two level error estimates. Four different marking strategies for refinement of stochastic Galerkin approximations are proposed and, in particular, for two of them, we prove that the sequence of energy errors computed by associated algorithms converges linearly. Finally, we use duality techniques for the goal oriented error estimation in approximating linear quantities of interest derived from solutions to parametric PDEs. Adaptive enhancements in the proposed algorithm are guided by an innovative strategy that combines the error reduction estimates computed for spatial and parametric components of corresponding primal and dual solutions. The performance of all adaptive algorithms and the effectiveness of the error estimation strategies are illustrated by numerical experiments. The software used for all experiments in this work is available online.

title: Quantifying and reducing input modelling error in simulation
abstract: This thesis presents new methodology in the field of quantifying and reducing input modelling error in computer simulation. Input modelling error is the uncertainty in the output of a simulation that propagates from the errors in the input models used to drive it. When the input models are estimated from observations of the real world system input modelling error will always arise as only a finite number of observations can ever be collected. Input modelling error can be broken down into two components: variance, known in the literature as input uncertainty; and bias. In this thesis new methodology is contributed for the quantification of both of these sources of error. To date research into input modelling error has been focused on quantifying the input uncertainty (IU) variance. In this thesis current IU quantification techniques for simulation models with time homogeneous inputs are extended to simulation models with nonstationary input processes. Unlike the IU variance, the bias caused by input modelling has, until now, been virtually ignored. This thesis provides the first method for quantifying bias caused by input modelling. Also presented is a bias detection test for identifying, with controlled power, a bias due to input modelling of a size that would be concerning to a practitioner. The final contribution of this thesis is a spline based arrival process model. By utilising a highly flexible spline representation, the error in the input model is reduced; it is believed that this will also reduce the input modelling error that passes to the simulation output. The methods described in this thesis are not available in the current literature and can be used in a wide range of simulation contexts for quantifying input modelling error and modelling input processes.

title: Improved uncertainty analysis for tidal energy project development
abstract: High investment risk is a key barrier to the commercialisation of the nascent tidal energy sector. An increase in investor confidence can unlock funding for early arrays, the lessons from which can provide further de risking, leading to further investment. This thesis focussed on increasing investor confidence by improving the uncertainty analysis methods used to quantify the overall uncertainty in key investment decision metrics; energy yield, levelised cost of energy (LCOE) and internal rate of return (IRR) A Monte Carlo Analysis (MCA) framework for tidal energy annual yield uncertainty analysis was developed and compared to the currently recommended ISO GUM method. It was shown that key assumptions implicit in ISO GUM are inaccurate for most realistic projects. Crucially, the resultant error provides an overly optimistic view of a project's P90 energy yield. By modelling a range of realistic projects, it was shown that the ISO GUM P90 yield overestimate exceeds 2% for a maximum resource uncertainty between 4% and 11% depending on the project, with increasing uncertainty leading to larger errors. It is difficult to judge accurately where within that range a given case crosses the 2% error threshold, as it is a complex function of numerous project specific variables. This undermines confidence in ISO GUM results, even in cases where the method may be acceptable, because it is not possible to deduce the validity for a particular project a priori. MCA does not make the same assumptions and provides consistently

title: Discretization of processes at stopping times and Uncertainty quantification of stochastic approximation limits
abstract: This thesis consists of two parts which study two separate subjects. Chapters 1 4 are devoted to the problem of processes discretization at stopping times. In Chapter 1 we study the optimal discretization error of stochastic integrals, driven by a multidimensional continuous Brownian semimartingale. In this setting we establish a path wise lower bound for the renormalized quadratic variation of the error and we provide a sequence of discretization stopping times, which is asymptotically optimal. The latter is defined as hitting times of random ellipsoids by the semimartingale at hand. In comparison with previous available results, we allow a quite large class of semimartingales and we prove that the asymptotic lower bound is attainable. In Chapter 2 we study the model adaptive optimal discretization error of stochastic integrals. In Chapter 1 the construction of the optimal strategy involved the knowledge about the diffusion coefficient of the semimartingale under study. In this work we provide a model adaptive asymptotically optimal discretization strategy that does not require any prior knowledge about the model. In Chapter 3 we study the convergence in distribution of renormalized discretization errors of Ito processes for a concrete general class of random discretization grids given by stopping times. Previous works on the subject only treat the case of dimension 1. Moreover they either focus on particular cases of grids, or provide results under quite abstract assumptions with implicitly specified limit distribution. At the contrast we provide explicitly the limit distribution in a tractable form in terms of the underlying model. The results hold both for multidimensional processes and general multidimensional error terms. In Chapter 4 we study the problem of parametric inference for diffusions based on observations at random stopping times. We work in the asymptotic framework of high frequency data over a fixed horizon. Previous works on the subject consider only deterministic, strongly predictable or random, independent of the process, observation times, and do not cover our setting. Under mild assumptions we construct a consistent sequence of estimators, for a large class of stopping time observation grids. Further we carry out the asymptotic analysis of the estimation error and establish a Central Limit Theorem (CLT) with a mixed Gaussian limit. In addition, in the case of a 1 dimensional parameter, for any sequence of estimators verifying CLT conditions without bias, we prove a uniform a.s. lower bound on the asymptotic variance, and show that this bound is sharp. In Chapters 5 6 we study the problem of uncertainty quantification for stochastic approximation limits. In Chapter 5 we analyze the uncertainty quantification for the limit of a Stochastic Approximation (SA) algorithm. In our setup, this limit is defined as the zero of a function given by an expectation. The expectation is taken w.r.t. a random variable for which the model is assumed to depend on an uncertain parameter. We consider the SA limit as a function of this parameter. We introduce the so called Uncertainty for SA (USA) algorithm, an SA algorithm in increasing dimension for computing the basis coefficients of a chaos expansion of this function on an orthogonal basis of a suitable Hilbert space. The almost sure and Lp convergences of USA, in the Hilbert space, are established under mild, tractable conditions. In Chapter 6 we analyse the L2 convergence rate of the USA algorithm designed in Chapter 5.The analysis is non trivial due to infinite dimensionality of the procedure. Moreover, our setting is not covered by the previous works on infinite dimensional SA. The obtained rate depends non trivially on the model and the design parameters of the algorithm. Its knowledge enables optimization of the dimension growth speed in the USA algorithm, which is the key factor of its efficient performance.

title: Essays on volatility networks and uncertainty
abstract: This thesis empirically investigates different aspects of time varying volatility. Chapter 1 estimates a large TVP FAVAR and recovers a dynamic directed network of connections between European stock volatilities. We propose an ad hoc estimation methodology that is shown to outperform both standard approaches and competing models. Chapter 2 focuses on tracking dynamic connectedness between US sectoral volatilities using Generalized Forecast Error Variance Decompositions with a Bayesian model. As opposed to estimates obtained with rolling windows, we allow parameters to vary in a more flexible way. We show that there exists a stable relationship between the network structure and the volatility regimes in place at a given time. Chapter 3 estimates the unexpected time varying volatility component of fiscal budgets in Italy. We show that periods of higher unexpected fiscal volatility are likely to be recessionary. Expansionary policies are effective only when not accompanied by increases in uncertainty.

title: Measurement Error And Uncertainty Analysis By Excel
abstract: This thesis has focused on the measurement error and uncertainty analysis by excel conducted in the ABC Company Sdn Bhd. The objective of this study was to investigate the methods applies in error determination and measurement uncertainty. In order to analyze the data some of Statistical Process Control (SPC) tools were used such as histogram, cause and effect diagram and control chart. The result that gained is achieved to develop software to measure error and uncertainty. The errors in the production line are determined and proper tool is used to analyze the quality problem. Major defects were highlighted and analyzed. Root causes for the problems were determined and suggestions for improvement were suggested. After the improvement stage, suggestions for control the quality also were suggested.