Enabling efficient and effective data extraction and data labeling

History has left us with many wonderful visualizations. Historical visualizations capture valuable historical statistics. The original datasets that generate historical visualizations are useful to humanities and social science scholars but are often lost. This thesis starts with the problem of extracting such datasets from historical visualizations. It is impractical to expect purely automatic solutions for data extraction, as historical visualizations feature diverse designs, contain image defects, and the data extraction has low error tolerance. Meanwhile, manual extraction is laborious. It calls for efficient and effective solutions.
To this aim, this thesis proposes a machine-aided approach to data extraction from historical visualizations. We convert data extraction to iterative machine learning processes of carrying out classification tasks. Following the conversion, we propose a generic workflow that features active sampling and default labeling to reduce user effort in classification tasks. Based on the approach, we design a data extraction pipeline and develop an interactive system for data extraction from spatial visualizations. To extract data from diverse historical visualizations, a single data extraction pipeline may not suffice. We design a set of reusable data extraction workflows that can be assembled into various data extraction pipelines. We demonstrate the feasibility of the design through prototypes of two pipelines.
Data extraction can be seen as a particular data labeling task. Thus, we investigate the generalization of the machine assistance approach for data extraction into the scenario of data labeling. Through a coding of the literature, we identify a collection of common modules, which extends the set of machine assistance modules we have designed for data extraction. Based on the identified reusable implementations, we have developed an authoring toolkit that builds in common implementations of the modules. Using the toolkit, developers can visually configure data labeling workflows with no code or less code. We demonstrate using the toolkit to develop diverse systems for data labeling and data extraction.
Quality assurance is a laborious process in data extraction and data labeling to validate the correctness of the resulting data. It is thus critical to optimize user productivity in quality assurance interfaces. We propose using model-based evaluation for quality assurance interfaces in machine learning. We model user routines and operation time costs in quality assurance interfaces. The model-based approach facilitates efficient estimation of task completion time through simulations for quality assurance interfaces. We demonstrate using model-based evaluation in data extraction and aerial image classification scenarios.
This thesis builds up an interactive machine learning approach to data extraction and data labeling. We demonstrate that imperfect machine-learned models can still be used to assist end-users and improve their productivity in these tasks that are hard to automate.

Keywords: historical visualization
interactive machine learning
model-based evaluation
data extraction
data labeling
Advisor: Chen, M
Coecke, B