Data retention

Data retention defines the policies of persistent data and records management for meeting legal and business data archival requirements. Although sometimes interchangeable, it is not to be confused with the Data Protection Act 1998.
The different data retention policies weigh legal and privacy concerns economics and need-to-know concerns to determine the retention time, archival rules, data formats, and the permissible means of storage, access, and encryption.
In the field of telecommunications, "data retention" generally refers to the storage of call detail records (CDRs) of telephony and internet traffic and transaction data (IPDRs) by governments and commercial organisations. In the case of government data retention, the data that is stored is usually of telephone calls made and received, emails sent and received, and websites visited. Location data is also collected.
The primary objective in government data retention is traffic analysis and mass surveillance. By analysing the retained data, governments can identify the locations of individuals, an individual's associates and the members of a group such as political opponents. These activities may or may not be lawful, depending on the constitutions and laws of each country. In many jurisdictions, access to these databases may be made by a government with little or no judicial oversight.
In the case of commercial data retention, the data retained will usually be on transactions and web sites visited.
Data retention also covers data collected by other means (e.g., by Automatic number-plate recognition systems) and held by government and commercial organisations.
A data retention policy is a recognized and proven protocol within an organization for retaining information for operational use while ensuring adherence to the laws and regulations concerning them. The objectives of a data retention policy are to keep important information for future use or reference, to organize information so it can be searched and accessed at a later date and to dispose of information that is no longer needed.
The data retention policies within an organization are a set of guidelines that describes which data will be archived, how long it will be kept, what happens to the data at the end of the retention period (archive or destroy) and other factors concerning the retention of the data.
A part of any effective data retention policy is the permanent deletion of the retained data; achieving secure deletion of data by encrypting the data when stored, and then deleting the encryption key after a specified retention period. Thus, effectively deleting the data object and its copies stored in online and offline locations.
In 2015, the Australian government introduced mandatory data retention laws that allows data to be retained up to two years. The scheme is estimated to cost at least AU$400 million per year to implement, working out to at least $16 per user per year. It requires telecommunication providers and ISPs to retain telephony, Internet and email metadata for two years, accessible without a warrant, and could possibly be used to target file sharing. The Attorney-General has broad discretion on which agencies are allowed to access metadata, including private agencies.
The Greens were strongly opposed to the introduction of these laws, citing privacy concerns and the increased prospect of 'speculative invoicing' over alleged copyright infringement cases. The Labor Party initially opposed as well, but later agreed to passing the law after additional safeguards were put in place to afford journalists some protection.
On 15 March 2006, the European Union adopted the Data Retention Directive. It required Member States to ensure that communications providers retain data as specified in the Directive for a period of between 6 months and 2 years in order to:
The data was required to be available to "competent" national authorities "for the purpose of the investigation, detection and prosecution of serious crime, as defined by each Member State in its national law".
The Directive covered fixed telephony, mobile telephony, Internet access, email, and VoIP. Member States were required to transpose it into national law within 18 months—no later than September 2007. However, they could if they wished postpone the application of the Directive to Internet access, email, and VoIP for a further 18 months after this date. A majority of Member States exercised this option. All 28 EU States at the time notified the European Commission about the transposition of the Directive into their national law. Of these, however, Germany and Belgium had only transposed the legislation partially.
A report evaluating the Directive was published by the European Commission in April 2011. It concluded that data retention was a valuable tool for ensuring criminal justice and public protection, but that it had achieved only limited harmonisation. There were serious concerns from service providers about the compliance costs and from civil society organisations who claimed that mandatory data retention was an unacceptable infringement of the fundamental right to privacy and the protection of personal data according to EU law.
In response to the report, on May 31, 2011, the European Data Protection Supervisor expressed some concerns on the European Data Retention Directive, underlining that the Directive "does not meet the requirements imposed by the fundamental rights to privacy and data protection".
In November 2012, answers to a parliamentary inquiry in the German Bundestag revealed plans of some EU countries including France to extend data retention to chats and social media. Furthermore, the German Federal Office for the Protection of the Constitution (Germany's domestic intelligence agency) has confirmed that it has been working with the ETSI LI Technical Committee since 2003.
Criticisms of the directive arose. The council's Legal Services was reported to have stated in closed session that paragraph 59 of the European Court of Justice's ruling "suggests that general and blanket data retention is no longer possible". A legal opinion funded by the Greens/EFA Group in the European Parliament finds that the blanket retention data of unsuspected persons generally violates the EU Charter of Fundamental Rights, both in regard to national telecommunications data retention laws and to similar EU data retention schemes (PNR, TFTP, TFTS, LEA access to EES, Eurodac, VIS).
Digital Rights Ireland brought the directive to the High Court of Ireland, which then brought it further to the European Court of Justice of the European Union. The case was also joined by the Constitutional Court of Austria. The Court on 8 April 2014 declared the Directive 2006–24/EC invalid for violating fundamental rights, stating that "the directive interferes in a particularly serious manner with the fundamental rights to respect for private life and to the protection of personal data".
This led further to that the member states in various degrees abolished or modified their implementations of the directive. Since the Swedish implementation of the directive was kept in a similar manner, the Swedish implementation was brought to the European Court by the telecom provider Tele2, and the case was merged with a similar case from the United Kingdom, initiated by three persons with intervention by Open Rights Group, Privacy International and The Law Society of England and Wales. Since the original directive no longer existed, the basis for the judgment was an exception to the Directive on privacy and electronic communications in its Article 15(1), referring to the possibility to exceptionally apply data retention for fighting serious crime. On the 21 of December 2016 the Court ruled that "the protection of privacy in the electronic communications sector must be interpreted as precluding national legislation which, for the purpose of fighting crime, provides for general and indiscriminate retention of all traffic and location data of all subscribers and registered users relating to all means of electronic communication." Blanket data retention was ruled out another time, but the actual consequences all over the EU are varied and under discussion since then.
Implementation of the directive was part of Act. No. 259/2010 Coll. on electronic communications as later amended. Under Art. 97 (3), telecommunication data are to be stored between 6 and 12 months. The Czech Constitutional Court has deemed the law unconstitutional and found it to be infringing on the peoples right to privacy.
As of July 2012, new legislation was on its way.
Denmark has implemented the EU data retention directive and much more, by logging all internet flow or sessions between operators and operators and consumers.
The German Bundestag had implemented the directive in "Gesetz zur Neuregelung der Telekommunikationsüberwachung und anderer verdeckter Ermittlungsmaßnahmen sowie zur Umsetzung der Richtlinie 2006/24/EG". The law became valid on 1 January 2008. Any communications data had to be retained for six months. On 2 March 2010, the Federal Constitutional Court of Germany ruled the law unconstitutional as a violation of the guarantee of the secrecy of correspondence. On 16 October 2015, a second law for shorter, up to 10 weeks long, data retention excluding email communication was passed by parliament. However, this act was ruled incompatible with German and European laws by an injunction of the Higher Administrative Court of North Rhine-Westphalia. As a result, on June 28, 2017, three days before the planned start of data retention, the Federal Network Agency suspended the introduction of data retention until a final decision in the principle proceedings.
In July 2005 new legal requirements on data retention came into force in Italy.
Italy already required the retention of telephony traffic data for 48 months, but without location data. Italy has adopted the EU Directive on Privacy and Electronic Communications 2002 but with an exemption to the requirement to erase traffic data.
The directive was transposed into law by Law 32/2008. In December 2017, D3 – Defesa dos Direitos Digitais, a Portuguese digital rights organization, presented a complaint to the Justice Ombudsman, based on the case law of the Court of Justice of the European Union, following several opinions of the Portuguese Data Protection Authority. In January 2019, the Ombudsman issued an official recommendation to the Justice Ministry, defending the need to change the national law, in order to comply with the CJEU case law. Some weeks later, in March, the Ombudsman received an answer by the Minister of Justice, where the Minister refused changes to the law. As such, in August 2019, the Ombudsman decided to ask the Portuguese Constitutional Court for a ruling on the constitutionality of the law. In 2022, the Portuguese Constitutional Court published its decision, striking down Law 32/2008 as unconstitutional. Among other things, the Court considered that an undifferentiated and generalized obligation to store all traffic and location data relating to all people did not respect the proportionality principle. In response to this decision, the parliament created a data retention working party, which studied the subject for more than a year and held several hearings with experts. In 2023, a law proposal was approved in the parliament. However, the President of the Republic decided to make use of its prerogative of asking the Constitutional Court for a preventive rule, before approving the law. In this ruling, the Constitutional Court once again decided against the proposed data retention regime, for similar reasons, as the law still required indiscriminate and general data retention of traffic and location data. The diploma was returned to the Parliament and did not become law. In 2024, the Parliament approved a new law proposal. This time, the President of the Republic opted for not requesting a preventive rule from the Constitutional Court, and so the law was published and entered into force. The digital rights association D3 – Defesa dos Direitos Digitais maintains that the current law is still a violation of fundamental rights, as it delegates core elements of a fundamental rights restriction to a special formation of the Supreme Court. This makes it impossible to demonstrate the required proportionality of the restriction, or to demonstrate how the data retention regime preserves the essential core of the restricted fundamental rights, as it must.
The EU directive has been transposed into Romanian law as well, initially as Law 298/2008. However, the Constitutional Court of Romania subsequently struck down the law in 2009 as violating constitutional rights. The court held that the transposing act violated the constitutional rights of privacy, of confidentiality in communications, and of free speech. The European Commission has subsequently sued Romania in 2011 for non-implementation, threatening Romania with a fine of 30,000 euros per day. The Romanian parliament passed a new law in 2012, which was signed by president Traian Băsescu in June. The Law 82/2012 has been nicknamed "Big Brother" (using the untranslated English expression) by various Romanian non-governmental organizations opposing it. On July 8, 2014, this law too was declared unconstitutional by the Constitutional Court of Romania.
Slovakia has implemented the directive in Act No. 610/2003 Coll. on electronic communications as later amended. Telecommunication data are stored for six months in the case of data related to Internet, Internet email and Internet telephony (art. 59a (6) a), and for 12 months in the case of other types of communication (art. 59a (6) b).
In April 2014, the Slovak Constitutional Court preliminary suspended effectiveness of the Slovak implementation of Data Retention Directive and accepted the case for the further review. In April 2015 Constitutional court decided that some parts of Slovak laws implementing DR Directive are not in compliance with Slovak constitution and Convention for the Protection of Human Rights and Fundamental Freedoms. According to now invalid provisions of the Electronic Communications Act, the providers of electronic communications were obliged to store traffic data, localization data and data about the communicating parties for a period of 6 months (in the case Internet, email or VoIP communication) or for a period of 12 months (in case of other communication).
Sweden implemented the EU's 2006 Data Retention Directive in May 2012, and it was fined €3 million by the Court of Justice of the European Union for its belated transposition (the deadline was 15 September 2007). The directive allowed member states to determine the duration data is retained, ranging from six months to two years; the Riksdag, Sweden's legislature, opted for six months.
In April 2014, however, the CJEU struck down the Data Retention Directive. Following the judgement, PTS, Sweden's telecommunications regulator, told Swedish ISPs and telcos that they would no longer have to retain call records and internet metadata. The Swedish government initiated a one-man investigation that stated that Sweden could keep on with data-retention. After that, the PTS reversed course. Most of Sweden's major telecommunications companies complied immediately, though Tele2 appealed this order before the Administrative Court in Stockholm claiming that the Swedish implementation should be reversed following the directive being declared unvalid, including the fact that the Swedish implementation went further than the directive, including registration of failed telephone calls and the geographic endpoint of a mobile communications. The appeal was rejected. The one holdout ISP, Bahnhof, was given an order to comply by November 24 deadline or face a five million krona ($680,000) fine.
Tele2 appealed the first level court rejection to the Swedish Administrative Court of Appeal, that sent the matter to the European Court of Justice of the European Union. That led to a judgement that once again invalidated blanket data retention of all communications of all citizens' communications to combat crime. See under European Union above.
The Data Retention and Investigatory Powers Act came into force in 2014. It is the answer by the United Kingdom parliament after a declaration of invalidity was made by the Court of Justice of the European Union in relation to Directive 2006/ 24/EC in order to make provision, about the retention of certain communications data. In addition, the purpose of the act is to:
The act is also to ensure that communication companies in the UK retain communications data so that it continues to be available when it is needed by law enforcement agencies and others to investigate committed crimes and protect the public. Data protection law requires data that isn't of use to be deleted. This means that the intention of this Act could be using data retention to acquire further policing powers using, as the Act make data retention mandatory.
An element of this Act is the provision of the investigatory powers to be reported by 1 May 2015.
The Data Retention and Investigatory Powers Act 2014 was referred to as the "snooper's charter" communications data bill. Theresa May, a strong supporter of the Parliament Act, said in a speech that "If we (parliament) do not act, we risk sleepwalking into a society in which crime can no longer be investigated and terrorists can plot their murderous schemes undisrupted."
The United Kingdom parliament its new laws increasing the power of data retention is essential to tackling crime and protecting the public. However, not all agree and believe that the primary objective in the data retention by the government is mass surveillance.
After Europe's highest court said the depth of data retention breaches citizens' fundamental right to privacy and the UK created its own Act, it has led to the British government being accused of breaking the law by forcing telecoms and internet providers to retain records of phone calls, texts and internet usage. From this information, governments can identify an individual's associates, location, group memberships, political affiliations and other personal information.
In a television interview, the EU Advocate General Pedro Cruz Villalón highlighted the risk that the retained data might be used illegally in ways that are "potentially detrimental to privacy or, more broadly, fraudulent or even malicious".
The bodies that are able to access retained data in the United Kingdom are listed in the Regulation of Investigatory Powers Act 2000 (RIPA). These are the following:
However, the Regulation of Investigatory Powers Act 2000 (RIPA) also gives the Home Secretary powers to change the list of bodies with access to retained data through secondary legislation. The list of authorised bodies now includes:
The justifications for accessing retained data in the UK are set out in the Regulation of Investigatory Powers Act 2000 (RIPA). They include:
The EU's Data Retention Directive has been implemented into Norwegian law in 2011, but this will not be in effect before 1 January 2015.
A 2016 anti-terrorist federal law 374-FZ known as Yarovaya Law requires all telecommunication providers to store phone call, text and email metadata, as well as the actual voice recordings for up to 6 months. Messaging services like WhatsApp are required to provide cryptographic backdoors to law-enforcement. The law has been widely criticized both in Russia and abroad as an infringement of human rights and a waste of resources.
On 29 June 2010, the Serbian parliament adopted the Law on Electronic Communications, according to which the operator must keep the data on electronic communications for 12 months. This provision was criticized as unconstitutional by opposition parties and by Ombudsman Saša Janković.
As from 7 July 2016, the Swiss Federal Law about the Surveillance of the Post and Telecommunications entered into force, passed by the Swiss government on 18 March 2016.
Swiss mobile phone operators have to retain the following data for six months according to the BÜPF:
All Internet service providers must retain the following data for six months:
Email application refers to SMTP-, POP3-, IMAP4, webmail- and remail-server.
Switzerland only applies data retention to the largest Internet service providers with over 100 million CHF in annual Swiss-sourced revenue. This notably exempts derived communications providers such as ProtonMail, a popular encrypted email service based in Switzerland.
The National Security Agency (NSA) commonly records Internet metadata for the whole planet for up to a year in its MARINA database, where it is used for pattern-of-life analysis. U.S. persons are not exempt because metadata are not considered data under US law (section 702 of the FISA Amendments Act). Its equivalent for phone records is MAINWAY. The NSA records SMS and similar text messages worldwide through DISHFIRE.
Various United States agencies leverage the (voluntary) data retention practised by many U.S. commercial organizations through programs such as PRISM and MUSCULAR.
Amazon is known to retain extensive data on customer transactions. Google is also known to retain data on searches, and other transactions. If a company is based in the United States the Federal Bureau of Investigation (FBI) can obtain access to such information by means of a National Security Letter (NSL). The Electronic Frontier Foundation states that "NSLs are secret subpoenas issued directly by the FBI without any judicial oversight. These secret subpoenas allow the FBI to demand that online service providers or ecommerce companies produce records of their customers' transactions. The FBI can issue NSLs for information about people who haven't committed any crimes.
NSLs are practically immune to judicial review. They are accompanied by gag orders that allow no exception for talking to lawyers and provide no effective opportunity for the recipients to challenge them in court. This secret subpoena authority, which was expanded by the controversial USA PATRIOT Act, could be applied to nearly any online service provider for practically any type of record, without a court ever knowing". The Washington Post has published a well researched article on the FBI's use of National Security Letters.
The United States does not have any Internet Service Provider (ISP) mandatory data retention laws similar to the European Data Retention Directive, which was retroactively invalidated in 2014 by the Court of Justice of the European Union. Some attempts to create mandatory retention legislation have failed:
While it is often argued that data retention is necessary to combat terrorism and other crimes, there are still others who oppose data retention. Data retention may assist the police and security services to identify potential terrorists and their accomplices before or after an attack has taken place. For example, the authorities in Spain and the United Kingdom stated that retained telephony data made a significant contribution to police enquires into the 2004 Madrid train bombings and the 2005 London bombings.
The opponents of data retention make the following arguments:
The current directive proposal (see above) would force ISPs to record the internet communications of its users. The basic assumption is that this information can be used to identify with whom someone, whether innocent citizen or terrorist, communicated throughout a specific timespan. Believing that such as mandate would be useful is ignoring that some very committed community of crypto professionals has been preparing for such legislation for decades. Below are some strategies available today to anyone to protect themselves, avoid such traces, and render such expensive and legally dubious logging operations useless.
There are anonymizing proxies that provide slightly more private web access. Proxies must use HTTPS encryption in order to provide any level of protection at all. Unfortunately, proxies require the user to place a large amount of trust in the proxy operator (since they see everything the user does over HTTP), and may be subject to traffic analysis.
Some P2P services like file transfer or voice over IP use other computers to allow communication between computers behind firewalls. This means that trying to follow a call between two citizens might, mistakenly, identify a third citizen unaware of the communication.
For security conscious citizens with some basic technical knowledge, tools like I2P – The Anonymous Network, Tor, Mixmaster and the cryptography options integrated into any many modern mail clients can be employed.
I2P is an international peer-to-peer anonymizing network, which aims at not only evading data retention, but also at making spying by other parties impossible. The structure is similar to the one TOR (see next paragraph) uses, but there are substantial differences. It protects better against traffic analysis and offers strong anonymity and for net-internal traffic end-to-end encryption. Due to unidirectional tunnels it is less prone to timing attacks than Tor. In I2P, several services are available: anonymous browsing, anonymous e-mails, anonymous instant messenger, anonymous file-sharing, and anonymous hosting of websites, among others.
Tor is a project of the U.S. non-profit Tor Project to develop and improve an onion routing network to shield its users from traffic analysis. Mixmaster is a remailer service that allows anonymous email sending.
JAP is a project very similar to Tor. It is designed to route web requests through several proxies to hide the end user's Internet address. Tor support has been included into JAP.
The Arbeitskreis Vorratsdatenspeicherung (German Working Group on Data Retention) is an association of civil rights campaigners, data protection activists and Internet users. The Arbeitskreis coordinates the campaign against the introduction of data retention in Germany.
An analysis of federal Crime Agency (BKA) statistics published on 27 January 2010 by civil liberties NGO AK Vorrat revealed that data retention did not make a prosecution of serious crime any more effective.
As the EU Commission is currently considering changes to the controversial EU data retention directive, a coalition of more than 100 civil liberties, data protection and human rights associations, jurists, trade unions and others are urging the commission to propose the repeal of the EU requirements regarding data retention in favour of a system of expedited preservation and targeted collection of traffic data.