An information theory-based technology measures the importance of observations and leverages them to quantify the similarity between entities, improving accuracy and reducing the time required to find related entities in a population. (2005)

o handle the wide range of geographicscales and complex tasks that it mustadminister, the government splits its datain many different ways, collecting it at dif-ferent times and through different agen-cies. The resulting massive data heterogeneitymakes it impossible to effectively locate, share, orcompare data across sources, let alone achievecomputational data interoperability. Many settings urgently need some form of dataalignment or merging. For example, an air-qualityscientist at a state environmental agency such as theCalifornia Air Resources Board (CARB) reconcilesair emissions data from local regions to monitoroverall patterns and support air-policy regulation.In a homeland security scenario, analysts identifyand track threat groups using separately collectedand stored individual behaviors such as phone calls,e-mail messages, ﬁnancial transactions, and travelitineraries. Addressing these issues requires ﬁnding similari-ties between entities within or across heterogeneousdata sources. To date, most approaches for inte-grating data collections, or even for creating map-pings across comparable data sets, require manualeffort. Despite some promising recent work, theautomated creation of such mappings is still in itsinfancy because equivalences and differences man-ifest themselves at all levels, from individual datavalues through metadata to the explanatory textsurrounding the data collection as a whole. Some data sources contain auxiliary informationsuch as relational structure or metadata, which haveproven useful in interrelating entities. However, suchauxiliary data can be outdated, irrelevant, overlydomain-speciﬁc, or simply nonexistent. Therefore,a general-purpose solution can’t rely on such auxil-iary data. All we can count on is the data itself: a setof observations describing the entities.Applying this purely data-driven paradigm, we’vebuilt two systems: Guspin for automatically iden-tifying equivalence classes or aliases, and Sift forautomatically aligning data across databases. Thekey to our underlying technology is identifying themost informative observations and then matchingentities that share them. We’ve used our systems to align US Environ-mental Protection Agency (EPA) data between theSanta Barbara County Air Pollution Control Dis-trict (SBCAPCD) and Ventura County Air PollutionControl District (VCAPCD) emissions inventorydatabases and the CARB statewide inventory data-base, as well as to identify equivalence classes in theEPA’s Facilities Registry System (FRS). This workcan signiﬁcantly reduce the amount of human effortinvolved in creating single-point access to multipleheterogeneous databases.GOVERNMENT PARTNERSThe staff at CARB annually integrates the emis-sions inventory databases belonging to California’s35 air quality management districts (AQMDs) tocreate a state inventory. They submit this inventoryannually to the US EPA, which performs quality-assurance tests on state inventories and integratesthem into a national emissions inventory for use intracking national air-quality policies’ effects. To deliver their annual emissions data to CARB,air districts must manually reformat the dataaccording to the speciﬁcations of CARB’s CaliforniaEmission Inventory Development and ReportingSystem. Every time CARB revises the CEIDARSAn information theory-based technology measures the importance ofobservations and leverages them to quantify the similarity betweenentities, improving accuracy and reducing the time required to ﬁnd relatedentities in a population.