Big Data Veracity Assessment: Improving risk assessment by adding high veracity data to existing contents insurance models (2016)

Companies and scholars around the world are figuring out how to utilize the rapid growing availability of data for new insight and business value.In both mainstream media and academic literature, Big Data analytics is considered to have huge potential.New challenges are introduced through the nature of Big Data.The amount of data makes traditional storage and analysis techniques inadequate.The speed with which new data is created enables near real time analysis, but only with the appropriate infrastructure.Most data is not created with the purpose of analysis, but as a product of online activity or collected by the sensors we surround ourselves with.This data needs to be structured in such a way that correctly captures the meaning of the original data.Furthermore, the quality, useability, and reliability, also known as the veracity of the data, are often uncertain.According to literature reviewed in Chapter 3, four steps can be identified in the Big Data analytics chain of a company -data collection, data preparation, data analysis, and decision making.In each of these steps challenges arise due to the Big Data characteristics described.In this thesis the focus is on veracity in the data collection step, because this is where companies currently struggle to evaluate new data.In data collection knowing the veracity of the data is crucial, yet little research has been done into measurement techniques.Data analysts have to combine data from a variety of sources with different origins and estimating the veracity early can prevent time wasted on analysis and potentially bad decision making.This leads us to the main research questions of this thesis.Research Question -How to assess the veracity of Big Data for the use in insurance risk assessment models?Evaluating new data sources is a critical skill for contemporary data analysts.Researching how this is currently done and providing a solution firmly grounded in academic theory is both an academic contribution to Big Data research and a practical contribution to any organization looking to advance their Big Data analytics capabilities.This thesis is written from an insurance industry perspective, but most of these challenges apply to other industries as well.vThe ethical perspective cannot be overlooked when dealing with personal data on this scale.Some fear that Big Data can lead us into a dystopian future where our free will is lost to companies knowing exactly how to manipulate us and everyone is under permanent surveillance, but the tremendous positive potential of this technology is also widely acknowledged.How a company should navigate into the era of Big Data ethically is also discussed within this thesis.Methodology For the purpose of creating the veracity assessment framework the action design research methodology is chosen.This methodology combines aspects from both action research and design research.The key characteristics are that it entails development of a technological artifact in an organizational setting, solving a practical problem using academic theory.The academic contributions consist of the artifact itself and a reflection on the theories used to guide future research.The artifact is an example of exaptation, applying known solutions to new problems.The academic theory used is from the fields of data quality assessment and entity resolution.In these two fields metrics can be found for assessing data that allow data analysts to judge its merits.Several data quality assessment methodologies are evaluated for their applicability to veracity assessment.These metrics are then used to evaluate different types of external data sources.Different types of data sources are evaluated to test the framework for practicality and usefulness.Two cases are done where one data source is a structured dataset acquired from a data broker and the second data source is semi-structured data from a web based house trading platform.While being semi-structured, the data was converted to a structured dataset before analysis.The strategic advice for navigating into the era of Big Data ethically follows from a literature review combined with semi-structured interviews with key decision-makers in the insurance company.The internal data acquisition processes are described and analyzed to find whether ethical issues found in the literature are taking into account.Then the impact of advancements of data analytics in the insurance is evaluated and recommendations are given to company on how to deal with the potential consequences. ResultsSince no clear definition of veracity exists in Big Data literature, the first step in the research was creating this definition.The definition was made based on a collection of definitions found in academic literature with the focus on Big Data science.A distinction between Big Data science and other disciplines seems to be that the former focus mostly on truthfulness, while usefulness and data quality play a larger role in Big Data literature.In the context of the previous three V's -volume, variety, and velocity -the ability to trust and use the data for decision-making could be represented by veracity.Therefore the following definition was made. VeracityThe ability of the data to support a decision making process by being appropriate, useful, and of sufficient quality in the context in which it is analyzed.Collecting and preparing the data the company itself used for making their risk assessment models proved to be difficult due to a recent system migration, causing several iterations between preparation and collection to be necessary before the data was ready for analysis.Data acquired from data brokers was available within the company.Web-scraping as a form of data collection successfully yielded a dataset containing data of over 100.000 houses in The Netherlands.However, gathering data from social media posed new challenges of interpreting semi-structured data among other things that are outside the scope of this thesis.vi Definition Paper An indication of data integrity and the trust on this information to make decisions Lopez, del Rio, Benitez, and Herrera (2015) Veracity directly refers to inconsistency and data quality problems Saha and Srivastava (2014) Low veracity equals uncertainty Camacho (2014) Trustworthiness and accuracy of Big Data Lu, Zhu, Liu, Liu, and Shao (2014) We use Veracity interchangeably with trust, reliability and credibility Vlassov, Lozano, Rosell, and Franke (2015) Objectivity, trustworthiness, and credibility are the main dimensions of veracity Lukoianova and Rubin (2013) No definition Shyr and Spisic (2014) Table 1.1: Definitions of veracity Big Data.From the moment we started recording data until 2003, 5 billion gigabytes of data was gathered.In 2011, this amount of data was generated every two days, in 2013, every 10 minutes, and in 2015, every 10 seconds (Zwitter , 2014).This increasing amount of data means that a more complete digital image of our world is being created.With increasing computing power this allows real time evaluation of, for example, customer profiles.This may allow insurers to analyze a customers risk profile at the moment the customer is requesting a quote and give them a price within seconds.Variety describes the heterogeneity of the data.Sensor readings, text messages, images, all kinds of data is available.It can be automatically generated by technological artefacts or intentionally created by people trying to achieve a certain purpose.Three types of data are typically described by data quality scholars: structured, semi-structured, and unstructured data (