The purpose of our platform is to convert a big volume of trade data (more than 6 billion records) into value for our users, enabling them to transform this information into knowledge by combining insights, generated by our various tools. In this way our customers are able to make better strategic decisions or predictions, maintaining their competitive edge. Before we are able to publish or visualize this information, the data has to be processed and analyzed to guarantuee a certain quality and accessibility. To get a better understanding of this tranformation process we have described each step in detail for you below: 1. Collect or Extract Our data is coming from a variety of sources, most of the time semi- or unstructured and transaction based. This means that the data are not conform to a standard relational model, not always selfdescribing, in various formats and not typically hierarchical in nature. The data are mostly generated from operational systems using data entry applications which process day-to-day transactions (shipments) but also by scanned documents through OCR software (Optical Character Recognition), generating data (text) with a certain failure rate. Or statistical data reported by various countries, mostly structured but not always validated. 2. Transform Before the data can be imported/loaded we have to transform the data into a standard format by analyzing, decoding, renaming, transposing and mapping the data, so it will fit into our data warehouse. 3. Verify We verify our data by by quantitiv and qualititiv analysis using a statistical and data consistency framework. These frameworks monitor the total process, from collecting till publishing/visualizing the data on our platform. In this way we can evaluate the completeness and correctness of our data against the methods and procedures of our objectives. 4. Validate To guarantee our system meets our requirements and specifications of accuracy, repeatability, reproducibility, stability and safety, we validate our data and systems by data type-, code and cross reference- and structured validation. A few samples of our validation methods are: Plausibility checks: Checks fields to ensure data in these fields corresponds to the expected value Cross-system plausibility checks: Compares data in different systems to ensure it is consistent Checksums: This is a total done on one or more numeric fields which appears in every record Cardinality check: Checks that record has a valid number of related records 5. Cleansing Prior to the process of organizing information in a data warehouse, data cleansing or data scrubbing is crucial. Important tasks in data cleansing are: deduplicate / merge records detect incorrect or missing data values because of data entry errors inconsistent value naming conventions because of different entry formats and use of abbreviations such as ONE vs "1" or Ltd. instead of Limited Company names are written in various ways, misspelled or addresses are incorrect or polluted with additional information. 6. Organize and standardize Before the data can be mined they have to be organized, structured and standardized, having a common definition, format, elements and structure, fitting into later defined "business layers" for analysis and interpretation. To do so we use more than 20 types of complex algorithms and various programming languages. 7. Data enrichment Through data enrichment we are able to offer a significant improvement in the business value of our integrated data, a few examples of enrichment are: Matching incoming records with existing data for identification of new shipments, products, customers or suppliers Correcting invalid data based on other data in the record or database Interpolating missing values based on other available data 8. Data analysis One of the most important steps in the transformation to knowledge is the analysis of the data. Besides data exploration, datamining, modeling and correlation by algorithms, are quantitative and qualitativ statistics a major part of our data analysis. Only after the right analysis, the data will transform into information which we can use to build our knowledge out of. For your understanding: Data exploration: an approach to analyzing data sets to summarize their main characteristics, often with visual methods. Data Mining: the computational process of discovering patterns in large data sets involving methods at the intersection of artificial intelligence, machine learning, statistics, and database systems. Datamining involves 6 common classes of tasks: Anomaly detection - (Outlier/change/deviation detection) The identification of unusual data records, that might be interesting or data errors that require further investigation. Association rule learning (Dependency modelling) – Searches for relationships between variables. Clustering – is the task of discovering groups and structures in the data that are in some way or another "similar", without using known structures in the data. Classification – is the task of generalizing known structure to apply to new data. Regression – attempts to find a function which models the data with the least error. Summarization – providing a more compact representation of the data set, including visualization and report generation. 9. Visualization The primary goal of our data visualization is to communicate information clearly and efficiently via statistical graphics, tables and information graphics. Numerical data may be encoded using dots, lines, or bars, to visually communicate a quantitative message. Effective visualization helps our users to analyze and reason about data. It makes complex data more accessible, understandable and usable. Users may have particular analytical tasks, such as making comparisons or understanding causality, and the design principle of the graphic (i.e., showing comparisons or showing causality) follows the task. In these 9 steps ABRAMS World Trade Wiki transforms data into knowledge, we are confident our platform will be of a big help to keep your competitive edge!