Developing a Metadata Infrastructure for Official data : the ISTAT experience

Mining official data implies retrieving data from different surveys or administrative sources and properly interpreting them as measures of observed phenomena. In order to support such an activity, ISTAT is developing a metadata infrastructure which is based on two centralized metadata management system, and implies the development of tools for exploiting metadata in the data manipulation activities. Such a strategy is supported by the definition of a conceptual metadata framework together with proper documentation models for each class of metadata. 1 The ISTAT Strategy for Metadata Management: choices and involved problems Mining official data implies retrieving data from different surveys or administrative sources and properly interpreting them as measures of observed phenomena. Such an activity requires the availability of several classes of metadata concerning the characteristics and the information content of each exploitable source of information. To ensure the dissemination of such metadata to the data users is a primary task for National Statistical Institutes (NSI), nevertheless to introduce metadata management practices in the official data production is often a challenge. Most NSIs consider the development of a metadata infrastructure a long-term goal, which requires a carefully devised strategy. Moreover the increasing need for integrating data from several sources obliges the NSIs to pursue a policy of centralised metadata management. By means of homogeneously documenting data from different sources in a unique environment, a centralised metadata system provides the rough material for data integration. Therefore the core of the ISTAT strategy is the development of two centralised systems for metadata management, SIDI and SDOSIS, which manage metadata concerning the production processes of surveys and the information content of surveys and SIS, respectively. They will disseminate such metadata to both data users and survey designers. Moreover they are conceived as metadata servers for those data management systems and software tools which are exploited in the data production and dissemination activities. Another important experience which we are carrying on 2 Giovanna D'Angiolini is the development of ESPLORIS, a system which exploits metadata for assisting the user in extracting data from data collections produced by several sources. As analysts of real world phenomena, the end users of official data have a basic requirement: to search the large data collections which are produced by NSIs for retrieving those data which are better suited to the analysis of a given class of phenomena, and properly transform them. This is an exploratory activity, which requires the availability of proper metadata. Many OLAP/DW tools exist which are well suited to perform data exploration inside an organisation. Inside a single organisation the role of the different classes of users constrains their information requirements. On the contrary the official data users have unpredictable information requirements, therefore a richer amount of metadata is required for steering them in retrieving and transforming data. This is the reason why NSIs should invest in developing specialised tools for supporting the data users' exploratory activities. Moreover tools for metadata-based data retrieving are an important component of a general metadata infrastructure because a good metadata quality is ensured only if metadata are actually used in data manipulation activities. Last but not least, a NSI's metadata strategy requires a sound conceptual foundation. This implies answering such questions: which are the relevant classes of metadata for properly retrieving, transforming, analysing official data? How should we model such metadata? In sections 2 and 3 we outline our general conceptual framework for metadata specification and present our model OSI, which we use for modelling the information content of surveys and statistical information systems. In section 4 we discuss the main characteristics of the systems which compose our metadata infrastructure, namely SIDI, SDOSIS and ESPLORIS. 2. Classes of Metadata for Documenting Official Data There is a general agreement on the need for a general conceptual framework which specifies relevant classes of metadata, starting from an in depth understanding of the role of metadata in the data production and analysis activities (see for example [13], [14], [18]) In our opinion, a general conceptual framework for metadata specification should specify classes of metadata and relationships among them according to several dimensions, and provide the metadata managers with documentation models for each singled out class of metadata. Moreover, the different contexts in which metadata are used and the different ways of communicating and using metadata should be analysed. We propose to single out relevant metadata classes on the basis of such dimensions: the content of metadata, their level of abstraction, their scope, that is "what metadata describe", "how we look at metadata", "which extents of data are described", respectively . Our specification of the content of metadata, that is, "what metadata describe", is based on the following main concepts: a SOURCE of statistical information is any process which is activated to observe real world phenomena so as to produce statistical information. A survey is a source, Developing a Metadata Infrastructure for Official data: the ISTAT experience 3 an administrative data collection is a source too. A source produces data collections by means of applying proper data production techniques and procedures; a STATISTICAL INFORMATION SYSTEM (SIS) is an integrated collection of pieces of statistical information which concern related phenomena and are issued by different sources. SIS are built for satisfying various and/or unpredictable information requirements. They are typically produced by NSIs for public usage. These concepts define the contexts which are described by metadata: the documented data come from observing real world objects by means of specific techniques and procedures, for properly using data the analysts need to know what has been observed and the way it has been observed. Therefore two main classes of metadata are singled out, concerning the information content of a source or a SIS and the characteristics of each source as an observation process, respectively. These are the basic metadata classes. Other relevant metadata concern the quality and the characteristics of the issued data when regarded as the result of a particular repetition of a production process. Such metadata have a different explanation and specification, therefore we separately analyse them. An outcome of our work on such other classes of metadata is our implemented system SIDI, which manages quality indicators together with metadata about the survey production process, in an integrated way (see [8]). Further analysis of the two main metadata classes can be developed along two other dimensions: the level of abstraction and the scope. A level of abstraction defines a particular “way of looking at" the objects which we want to represent. By means of considering sources and SIS at different levels of abstraction we single out three main metadata layer: the conceptual layer, the organisational layer and the operational layer. Sources and SIS as knowledge bases are the main objects which are defined in the conceptual layer. Proper classes of conceptual metadata describe the production process which is associated with each source as well as the information content of sources and SIS. We single out two components in the specification of the information content of sources and SIS. The first component encompasses the specification of the observed part of the real world, which is expressed in terms of elementary statistical concepts such as statistical unit, variable, as well as the specification of structured objects which are derived from such elementary objects. Each concept has a definition and a set of links with the other concepts. We call such a specification a terminology. The second component is the specification of the issued data: their meaning is documented by means of representing them as associations of terms of a terminology. We denote such associations by the name Information Frames. Each source or SIS has its own terminology, however sources can share production procedures and terminology concepts, but they produce their own information frames. At lower levels of abstraction we describe the distribution of SIS among organisations and the implementation of SIS and sources as data production, transformation and dissemination systems. In the organisational layer we distinguish among data producers, data users and other organisations, in the operational layer we document the physical data repositories and the data manipulation procedures, that is, those data management and 4 Giovanna D'Angiolini manipulation systems which are built for supporting the production and the usage of statistical data. Along the dimension "scope" we distinguish between local and global metadata (see [14]). In the conceptual layer, local metadata are those metadata which describe the information content and the characteristics of each source of information. Global metadata are those metadata which are obtained by means of conceptually integrating or standardising local metadata. The specification of the information content of a SIS is an example of global metadata. Conceptual integration (or, equivalently, harmonisation) is the activity which is performed in order to produce global metadata concerning the information content. It is a complex activity, which requires comparing and matching the conceptual objects which belong to different source terminologies, by means of analysing their definitions; for this purpose, the object definitions must be expressed in a structured shape, as combinations of f