Mining the metadata: and its consequences

Loosely speaking, metadata is "data about data" and is held distinct from the data itself. The term had its origins in libraries for cataloging information, its use has been paralleled in information systems for help in establishing provenance [2]. Metadata has generally been treated as ancillary to the data itself, and less important in terms of privacy, security, etc. For example, access to a library's electronic holdings are often available to non subscribers, while access to content typically requires a subscription. Similarly in communications, metadata -- the who, what, when of a communication -- has been afforded fewer protections than the content itself, and it has been retained and mined by telecommunications companies that provide much more extensive privacy protections to the communications content. Communications metadata was originally collected for billing purposes, but it has proved useful in multiple ways. Considering this data in aggregate provides insight into where new services or increased bandwidth would be useful, and provides engineers information about use of the system (e.g., whether the spike in communications traffic this Friday morning is anomalous, or is standard for the day after Thanksgiving, etc.). It also, of course, has application in law-enforcement and national-security investigations. At least in the United States, the legal rationale for the lowered protection for the metadata is because the data -- the number called, the number calling, time, and length of call -- is shared with a third party. Yet IP-based services call this rationale into question. IP-based services offer the benefit of more complex communications technologies, but in recent work with Bellovin, Blaze, and Pell, we observed that IP-based services make murky when third-party use, the basis for lower legal protections, is occurring [1]. Even while technology may be obscuring the distinction between communications metadata and content, the law is moving in the direction of greater collection of metadata, and lowered legal protection, for communications metadata. Following the 2004 Madrid and 2005 London terrorist bombings, in 2006 the EC passed a Data Retention Directive, requiring communications providers to retain communications metadata for between six months and two years, in 2014, the Court of Justice of the European Union invalidated the directive on privacy grounds. Meanwhile, as the Snowden disclosures revealed, the US was also requiring telecommunications providers to deliver communications records to the government on a daily basis [12]. This collection in bulk began shortly after the terrorist attacks in 2001. The fact is that in many cases, such transactional information is remarkably revelatory. Communications metadata can reveal which groups are meeting, how frequently, and whether acitivity has increased, this is true even when the members of the group try to disguise their patterns by using new communication devices. It can also expose whether an acquisition is being planned, if a person has been diagnosed with a serious illness, when and where lovers are meeting [5], [6]. Even when anonymized, small disclosures of transportation metadata can expose the daily habits of particular individuals [11]. Combining the bits of metadata we shed elsewhere can create quite detailed portraits of where we go [7], whom we meet, who we are [4]. We are in a classic tussle, in this case between privacy and the value of metadata for certain types of investigations. How it will play out remains unclear (see, e.g., [8], [10], [3]). Of course, communications is not the only field where metadata provides high value. Currently metadata is used in software engineering to help establish insights into the presence of errors (see, e.g., [13]), to determine when to do regression testing [9], etc. Studying the metadata enables insight into the correctness of code using simpler, lighterweight tools than a direct examination of the code. Metadata also finds much use in security, where the study of metadata is remarkably valuable in anomoly detection, in forensic examiniations of attacks, etc. In this keynote, using communications metadata as my prime example, I will focus on privacy issues in mining metadata, and consider whether privacy concerns mean that we should revise our view of metadata. I will also briefly consider other ways we might categorize metadata.