case – but most often with the mere threat of new legislation –, represent an important challenge considering the sheer volume of third-party content posted on these platforms: 300 hours of video are posted on YouTube every minute;<fnref n="143" /> on Facebook, 510,000 comments are posted, 293,000 statuses are updated, and 136,000 photos are uploaded every day.<fnref n="144" />

On the part of the platforms, these efforts have lead to significant investment in tools based on “Machine Learning” systems aimed at spotting terrorist-related content in order to make them inaccessible. In September 2017, taking some time away from the UN General Assembly in New York, Emmanuel Macron, Theresa May and Paolo Gentiloni, then Prime Minister of Italy, convened a meeting with the tech industry to renew their demands (with a new takedown reaction time of 2 hours upon notice). According to Theresa May:

> “Industry needs to go further and faster in automating the detection and removal of terrorist content online, and developing technological solutions which prevent it being uploaded in the first place. We need a fundamental shift in the scale and nature of our response – both from industry and governments – if we are to match the evolving nature of terrorists’ use of the internet.”<fnref n="145" />

An unidentified British official also explained to the Guardian the state of mind behind these calls:

> “These companies have some of the best brains in the world. They should really be focusing that on what matters, which is stopping the spread of terrorism and violence. We want them to break the echo chambers.”

Monica Bickert, Vice President of Consumer Operations at Facebook, says

she often gets asked “when is AI going to save us all?” But, she says, “we’re a long way from that.” To make a determination of whether a given content abides by the terms of service of these platforms, a human intervention remains essential to look at context of a given post or a community. According to Bickert:

> “There are some areas where technical tools are helping us do this job. But the vast majority, when we’re looking at hate speech or we’re looking at bullying or we’re looking at harassment, there is a person looking at it and trying to determine what’s happening in that offline world and how that manifests itself online.”<fnref n="146" />

Accordingly, we are seeing a significant increase in the number of staff mobilized on content moderation. In 2017, Facebook moderation team grew from 3000 to 7500 people in only eight months,<fnref n="147" /> while Google’s YouTube announced even more ambitious plans.<fnref n="148" /> According to YouTube CEO Susan Wojcicki, “[Google] will continue the growth of our teams, with the goal of bringing the total number of people across Google working to address content that might violate our

<footnote n="143">143. https://fortunelords.com/youtube-statistics/</footnote>

<footnote n="144">144. https://zephoria.com/top-15-valuable-facebook-statistics/</footnote>

<footnote n="145">145. Hope, C., & McCann, K. (2017, September 19). Google, Facebook and Twitter told to take down terror content within two hours or face fines. *The Telegraph*. Retrieved from https://www.telegraph.co.uk/news/2017/09/19/google-facebook-twitter-told-take-terror-content-within-two/</footnote>

<footnote n="146">146. Madrigal, A. C. (2018, February 7). Inside Facebook’s Fast-Growing Content-Moderation Effort. *The Atlantic*. Retrieved from https://www.theatlantic.com/technology/archive/2018/02/what-facebook-told-insiders-about-how-it-moderates-posts/552632/</footnote>

<footnote n="147">147. Idem.</footnote>

<footnote n="148">148. Beavers, O. (2017, December 4). Google to appoint staff of 10,000 to weed out extremist content on YouTube [Text]. Retrieved December 5, 2017, from http://thehill.com/policy/technology/363242-10000-google-staff-set-to-weed-out-extremist-content-on-youtube</footnote>
policies to over 10,000 in 2018.” Most of these content-moderators are in fact hired by subcontractors in low-wages countries and face serious issues regarding social rights (Roberts, 2016).

Rather than putting public money in the justice system to allow it to catch up with the surge in speech, states are pressuring private companies who have developed large technical infrastructures and amassed enormous financial power to do it. Counting on the ability of these firms to develop technical solutions to the political and legal problems tied to online speech, they place hopes in the development of “Artificial Intelligence.” So far, this has not delivered expected results, and this means that huge numbers of low-wage workers need to be recruited to populate what are, in effect, privatized censorship bureaucracies.

During a conference in February 2018, Neil Potts, a public policy manager at Facebook, emphasized the similarity between the censorship role played Facebook and that of governments:

> “We do really share the goals of government in certain ways,” he said. “If the goals of government are to protect their constituents, which are our users and community, I think we do share that. I feel comfortable going to the press with that.”<fnref n="149" />

But this growing alignment between large online platforms and states, as we have seen, often escapes appropriate legal frameworks. To some extent, this gives more leeway both to the states and to companies with regards to these censorship activities. A key issue for the US companies is to protect their status has “neutral” hosting providers which give them a liability exemption, and which many policy-makers are willing to undermine to increase pressure on these platforms.<fnref n="150" />

**Economic cooperation and provider-customer relationships:** A last instance of cooperation between US tech companies and the French state is their business relationship. Many US companies, from CISCO to Microsoft, are important bidders of public tenders issued by public administration in the field of public and cyber security.

Despite the efforts of many players to ensure “digital sovereignty” by favouring French and European competitors, the alleged superiority of US companies’ products and services give them a significant appeal. The contract between Microsoft and the French Ministry of Defense have for instance come under harsh criticism on the basis of “digital sovereignty.”<fnref n="151" /> In 2016, US company Palantir has also teamed up with the French Internal Security Intelligence Agency (DGSI by its French acronym) through a €10 million contract, when the agency failed to cope with the vast amount of data seized during the house raids and digital seizures conducted under the state of emergency, which was in force in France between November 2015 and October 2017.<fnref n="152" /> Other

<footnote n="149">149. Madrigal, A. C. (2018, February 7). Inside Facebook’s Fast-Growing Content-Moderation Effort. *The Atlantic*. Retrieved from https://www.theatlantic.com/technology/archive/2018/02/what-facebook-told-insiders-about-how-it-moderates-posts/552632/</footnote>

<footnote n="150">150. For an overview of recent debates on intermediary liability in the EU, see e.g.: Greenfield, H. (2018, March 1). New EU Recommendation on Illegal Content Online Undermines Online Rights and Harms Europe’s Tech Economy. Retrieved March 9, 2018, from https://www.ccianet.org/2018/03/new-eu-recommendation-on-illegal-content-online-undermines-online-rights-and-harms-europes-tech-economy/</footnote>

<footnote n="151">151. Open Bar Contract between Microsoft and the French Ministry of Defense: April Calls on Senators to Support a Motion for the Creation of an Investigation Committee. (2017, October 18). Retrieved March 9, 2018, from https://www.april.org/en/open-bar-contract-between-microsoft-and-french-ministry-defense-april-calls-senators-support-motion</footnote>

<footnote n="152">152. Tesquet, O. (2017, January 27). Palantir, l'encombrant ami américain du renseignement français. Retrieved January 27, 2017, from http://www.telerama.fr/medias/palantir-big-data-renseignement,153229.php</footnote>
prominent figures of the French intelligence community have voiced their scepticism towards this partnership. But it also speaks to the superiority of US tech companies and the seemingly irresistible appeal for their services and products even for the French national security field.

US tech companies are also international investors that help maintain the competitiveness of the French digital economy, or can even be a substitute to public policies. In 2015, Cisco and the French government signed $200 million partnerships. According to Cisco, the company is:

> "committed to accelerating the digital transformation of France through a series of investments totaling more than $100 million. This involved training individuals for the new digital economy, investing in innovative start-ups, developing national network infrastructure, researching smart cities, and broadening cyber security expertise."¹⁵³

In part, these investments lead to joint-ventures with large firms or startups in the French digital economy. For instance, in October 2015, the management of CISCO and Thales announced a common "sovereign" co-innovation proposal to protect critical infrastructures against cyberthreats. IBM has also recently partnered with Engie, a French multinational electric utility company, to help it develop Big Data solutions.¹⁵⁴ More recently, in the field of Artificial Intelligence, Facebook and Google have announced major investment in their French R&D centres.¹⁵⁵ US tech companies like Cisco, Microsoft or Google are also increasingly investing in training programs in French universities and schools.¹⁵⁶

**Revolving doors between EU administrations and these “essential partner”:** These various forms of cooperation show the strong interdependencies that unite US tech companies and the French state, and help explain the softer tone of the 2017 SGDSN strategic review compared with the one adopted two years before. According to the document, the digital ecosystem leads to “the rise of new private actors, who impose themselves on the international scene as a challenge to the sovereignty of states but also as sometimes essential partners.”¹⁵⁷ They also need to be understood in the wider context of the “revolving door” between French and EU administrations and US tech

According to the former Director of the DGSI, Patrick Calvar, «*“les entreprises françaises qui développent des systèmes ne sont pas encore capables de répondre à nos besoins, alors que nous devons acquérir ce big data immédiatement. Nos camarades européens sont dans la même situation. Le choix n’a pas encore été fait mais, en tout état de cause, la solution sera temporaire. Tout ce que nous développons se fait en relation directe avec la DGSE, afin d’éviter les doublons, et la plupart de nos ingénieurs en charge de ces questions viennent de la DGSE. La moindre perquisition nous permet de récupérer des milliers de données. Nous avons donc besoin d’outils de big data pour répondre immédiatement à nos besoins.”*Audition de M. Patrick Calvar, directeur général de la sécurité intérieure par la commission de la défense nationale et des forces armées. (2016, May 10). Retrieved March 9, 2018, from http://www.assemblee-nationale.fr/14/cr-cdef/15-16/c1516047.asp

153. Chambers, J. (2015, February 24). How France is Embracing Digitization of Everything. Retrieved March 9, 2018, from https://blogs.cisco.com/news/how-france-is-embracing-digitization-of-everything
Cisco drives the digital acceleration of France through innovative investment, partnership and programs. (2015, October 8). Retrieved March 9, 2018, from https://newsroom.cisco.com/press-release-content/?type=webcontent&articleId=1721329

154. Filippone, D. (2016, July 26). Avec IBM, Engie monte d'un cran dans les smartcities. Retrieved December 21, 2017, from https://www.lemondeinformatique.fr/actualites/lire-avec-ibm-engie-monte-d-un-cran-dans-les-smartcities-65501.html

155. Google, Facebook Target Paris as a Center for AI Expansion. (2018, January 22). *Bloomberg.Com*. Retrieved from https://www.bloomberg.com/news/articles/2018-01-22/google-facebook-target-paris-as-a-center-for-ai-expansion

156. See, e.g.: Hérard, P. (2018, February 25). Google et Facebook formateurs au "digital" de chômeurs et d'étudiants : que fait l'Etat ? Retrieved February 25, 2018, from http://information.tv5monde.com/info/google-et-facebook-formateurs-au-digital-de-chomeurs-et-d-etudiants-que-fait-l-etat-222637

157. Secrétariat général de la défense et de la sécurité nationale. (2018). *Revue stratégique de cyberdéfense*. Paris. Retrieved from http://www.sgdsn.gouv.fr/evenement/revue-strategique-de-cyberdefense/
companies. In 2016, the Google Transparency project has for instance documented:

> "at least 80 revolving door moves between Google and European governments over the past decade as the company seeks to boost its influence in the region and head-off antitrust action and privacy regulation.

Google, now part of Alphabet Inc., has hired at least 65 government officials from all over the European Union since 2005. Those included Tomas Gulbinas, a former ambassador-at-large for the Lithuanian government, and Georgios Mavros, an advisor to a French member of the European Parliament, to name two recent examples. Google hired both officials as lobbyists in 2015.

During the same period, 15 Googlers have been appointed to government positions in Europe, gaining valuable contacts at the heart of the decision-making process. Those included Baroness Joanna Shields, a former managing director for Google, who was appointed the UK's Minister for Internet Safety and Security at the Department for Culture, Media and Sport; and Google's executive chairman, Eric Schmidt, appointed by British Prime Minister David Cameron to his business advisory council.”<fnref n="158" />

Google and Microsoft for instance declare spending between €4,25 million et €4,5 million annually to lobby EU institutions, and this translates in frequent meetings with EU commissioners.<fnref n="159" />

TOP 10 ORGANISATIONS

i

<figure><img src="image_8.png" /></figure>

Lobby organizations with the most high-level meetings at the European Commission since 2014 (source: Transparency International).<fnref n="160" />

French data journalist Alexandre Léchenet has also stressed the importance of the phenomenon at the French level, with at least 23 hires from the public sector for major US tech companies in

<footnote n="158">158. Google's European Revolving Door. (2017). Retrieved February 13, 2018, from https://googletransparencyproject.org/articles/googles-revolving-door-us</footnote>

<footnote n="159">159. https://github.com/alphaenix/donnees/blob/master/lobbies-gafamut/lobbies.csv</footnote>

<footnote n="160">160. http://www.integritywatch.eu/</footnote>
France and many high-level meetings.<fnref n="161" />

<footnote n="161">161. Léchenet, A. (2017). *L'influence tentaculaire des géants américains*. Retrieved from https://github.com/alphoenix/donnees (Original work published December 2016).</footnote>
# 4. Conclusion: Changing Bureaucracies, Security & the Rule of Law

In concluding this deliverable, we would like to go back to our main research questions and gather key takeaways from our case studies.

First, regarding the strategies of major Internet firms in navigating debates on state surveillance, despite a lack of extensive data on this issue, our research suggests that these strategies are handled and decided upon at very senior levels within the companies – in some extreme cases, Chief Information Security officers might not even been aware of the companies decision to comply with a given and contentious surveillance order. Besides the management of sensitive information, this can also be seen as a way to shield the governance of national security issues from internal dissent within these companies.

With regards to the factors determining the behaviors of these firms in the resistance-cooperation dilemma, our hypothesis seem to be validated by our case-studies. We initially found important differences depending on the considered country: in the US, overt resistance was staged in a prime instance of double-dealing, whereas in France such double-dealing was immediately called out by government officials leading to much less intense and much more discreet forms of resistance. But overtime, from 2015 on, the influence of the human rights field withered along the media attention to state surveillance issues. The agenda became dominated by the terrorist threat and similar trends towards greater cooperation materialized on both sides of the Atlantic – especially regarding the issue of the monitoring and takedown of terrorism-related content on online platforms –, suggesting strong transnational field effects not only from technology multinationals but also from security professionals (Bigo, 2016).

As a consequence, while we sometimes observed important variations in the processes depending on the countries under consideration – variations that can be explained by the greater proximity of US firms with the US government and the existence of existing public-private alliances in Europe where these firms economic and cultural capitals do not hold the same value in the administrative or security fields where they face strong private competitors –, these processes eventually aggregate in coherent mechanisms and produce similar outcomes overtime.

## Data governance as a new bureaucratic paradigm

As our case-studies show, these shifts in public-private assemblages for communications surveillance purposes need to be inscribed in a broader context of changes in modern bureaucracies.

In her work on neo-liberal bureaucratization, Béatrice Hibou have shown how, from the 1970s on, the logic of management migrated from the private realm to state institutions, and how the spread of these private rules and procedures meshed, contributing to the blurring of traditional public-private distinctions (Hibou, 2012). But through the imperatives of “efficiency”, of “cost-effectiveness”, of “flexibility” and practices of “auditing”, of “benchmarking”, bureaucratic practices within public administrations were increasingly perceived as alien to the values of the public sector and to its relationship to the social world. The abstract principles of neo-liberal democracies and the old
precept of abstract and general rules was pushed so far as to give way to a “manufactured unicity”
of the social world, leading to aggravated forms of depoliticization.

As Zuboff (2015) and others as well as our own research on the security field suggest, the large
technology firms on which we have focused in this report have nurtured a new bureaucratic
paradigm based on “data governance.” In 2014, Eric Schmidt and Jonathan Rosenberg, a former
Senior Vice President of Products at Google, co-wrote *How Google Works*.162 Adding to the doctrine
already forged by Google’s Chief Economist Hal Varian regarding “data extraction and analysis”,
“new contractual forms due to better monitoring”, “personalization and customization”, and
“continuous experiments,”163 the authors documented the business management lessons from
Google, an experience that led them to “relearn everything” they knew by exploring data-intensive
models, expanding the “new spirit of capitalism” with a horizontally-driven and creative workforce.
Some years before, some self-proclaimed Internet theorists like Jeff Jarvis had also called on
emulating Google across society.164

It these models that Schmidt is today helping spread at the Pentagon through the Defense
Innovation Board. In France too, the debate on the “reform of the state” has moved from the
premisses of the New Public Management described by Hibou to data governance. By referring to
concepts like the “Startup Nation” or the “platform State”,<fnref n="165" /> today’s reformers are re-modelling
bureaucracies and decision-making processes around the need to produce data, make it available
and usable, maintain its integrity and feed it to powerful data-processing tools that will be used to
optimize bureaucratic outputs. Even when these discourses claim to be opposing the hegemony of
US tech companies, they are in fact assuming the superiority of this model and diffusing them
across public administrations. It is an instance of “mimetic rivalry” (Girard, 2002), where what
Evgeny Morozov has termed *solutionism* serves as a new technocratic utopia (Morozov, 2013).

Public-private security assemblages in the age of data governance

Similar trends towards technical, managerial and technological responses to security challenges
have of long been sweeping the security field (Abrahamsen & Williams, 2010; Bonelli, 2010). But
today, public-private hybridization – especially when it comes to monitoring and controlling
information flows – are also increasingly focused on data governance.

Faced with the challenge of adapting surveillance and censorship practices to the challenging digital
environment – which was supposed to undermine state sovereignty and law enforcement –, national
security professionals have increasingly resorted to alliances with private actors who master new
forms of management, have the technical know-how and operate communication infrastructures. In
today’s digital economy, that means not only dealing with large, “home-grown” firms (sometimes
former state monopolies) whose relations to the state are long-established, with long-running forms
of cross-socialization between public and private security professionals, but also with newcomers to
the field like tech firms who came to dominate the digital economy in recent years.

<footnote n="162">162. Schmidt, E., & Rosenberg, J. (2015). *How Google Works*. London: John Murray.</footnote>

<footnote n="163">163. Varian, H.R. (2010). Computer Mediated Transactions, American Economic Review 100(2): 1–10. Varian, H.R. (2014). Beyond Big Data, Business Economics 49(1): 27–31.</footnote>

<footnote n="164">164. Jarvis, J. (2011). *What Would Google Do?: Reverse-Engineering the Fastest Growing Company in the History of the World* (Reprint). New York; Enfield: HarperBusiness.</footnote>

<footnote n="165">165. In France, see e.g.: Algan, Y., & Cazenave, T. (2016). *L'État en mode start-up*. Paris: Eyrolles. Bertholet, C., & Létourneau, L. (2017). *Ubérisons l'État ! Avant que d'autres ne s'en chargent*. Malakoff: Armand Colin. Pezziardi, P., & Verdier, H. (2017). *Des startups d'État à l'État plateforme*. CreateSpace Independent Publishing Platform.</footnote>
The large tech firms on which this deliverable has focused are – because of the structure of their technical, economic and organizational capitals – a prime locus to observe these trends. We have seen that, despite the fact that conflicts are being staged, and under the conditions where their capital is gaining the upper hand across the administrative field, new cross-socialization spaces are being created, alliances are being formed between these actors and public security professionals. In periphery of these new cooperation avenues, conflict remain. Such is the case for instance when state officials use securitization discourses, the threat of legal sanctions or denounced the double-dealing of tech companies to “force” cooperation in security matters, or when the personnels of these companies stage their resistance to the security policies and demands of the state. But as we have seen, they can in part be attributed to the actors’ strategies, determined by various constraint structures, as they progress towards greater cooperation in a field that is by its very nature a competitive space, reaped with power relations and struggles.

Although these emerging public-private security assemblages tasked with managing information flows are particularly visible when they embark large tech companies used daily by billions of Internet users, these large online service providers are hardly the only private actors involved in such assemblages. Old tech and utility companies in the transportation or energy sectors are also increasingly investing in Big Data analytics (partly as a result of cybersecurity public policies), sometimes in partnership with their US competitors, and getting closer to the security field. Many small companies specialized in data analytics or vulnerabilities are also partnering with intelligence agencies to sell their products and services (Deibert, 2013).<fnref n="166" />

For these two later types of companies, who can also count on a kind of technical-organizational capital currently sweeping the administrative field, resorting to “revolving door” hires with people coming from intelligence agency is usually easier.<fnref n="167" /> For the most part, they also have the luxury of not being engulfed in the sort of public scandal that large consumer-facing companies like Google, Apple, Facebook and the like had to deal with during the Snowden disclosures, thus decreasing the likelihood of resistance. Even when they are, like for instance Orange and Qosmos in France, their internal corporate culture and their already completed inclusion in public-private security assemblages mean that they are more likely to just keep a low-profile rather than stage resistance, while relying on the support of “deep state” officials. Finally, because of their business operation and the nature of their products, these companies will typically not face the sort of regulatory stakes – and in particular conflicts of law and judicial cooperation mechanisms – which, as we have seen,

<footnote n="166">166. See e.g.: CIA hackers' favourite suppliers. (2017, March 22). Retrieved March 13, 2018, from https://www.intelligenceonline.com/grey-areas/2017/03/22/cia-hackers--favourite-suppliers,108226984-art
Paris wises up to darknet data leaks. (2018, February 14). Retrieved March 13, 2018, from https://www.intelligenceonline.com/surveillance--interception/2018/02/14/paris-wises-up-to-darknet-data-leaks,108294132-art</footnote>

<footnote>NSA, GCHQ look to private sector for cyber-offensive firepower. (2017, June 21). Retrieved March 14, 2018, from https://www.intelligenceonline.com/government-intelligence/2017/06/21/nsa-gchq-look-to-private-sector-for-cyber-offensive-firepower,108250738-eve</footnote>

<footnote n="167">167. See, e.g.: FedData recruits at NSA, its special customer. (2017, May 24). Retrieved March 14, 2018, from https://www.intelligenceonline.com/corporate-intelligence/2017/05/24/feddata-recruits-at-nsa-its-special-customer,108235644-bre</footnote>

<footnote>Former spy chief turns to new cyber-security role in the private sector. (2017, October 26). Retrieved March 14, 2018, from https://www.thesun.co.uk/news/4768235/spy-boss-robert-hannigan-who-quit-due-to-ill-health-lines-up-lucrative-private-deals/</footnote>

<footnote>Guibert, N. (2014, January 14). L'ex-M. Grandes Oreilles de la DGSE part dans le privé. *Le Monde.fr*. Retrieved from http://www.lemonde.fr/economie/article/2014/01/14/l-ex-monsieur-grandes-oreilles-de-la-dgse-part-dans-le-prive_4347510_3234.html</footnote>
further complicate the dealings of US platforms with the state officials of the countries in which
they operate.

Overcoming the digital challenge at the price of the rule of law?

In concluding this deliverable, we would like to tie our findings to some of the hard-pressing
questions regarding the rule of law and human rights in the digital age: What are the political
consequences of the emergence of data governance as a new bureaucratic paradigm and the
corresponding reassembly of public-private assemblages in the security field?

The alliance of traditional state power with the reach and depth of the digital infrastructure mastered by a handful of companies has turned mass surveillance into a practical reality. In the arms-race to amass and process data, these new “Big-Data security assemblages” are now implementing automatic, preventive, preemptive and/or predictive policing approaches. There are reasons – and growing evidence – to doubt that the new “regimes of truth” established by modern surveillance will in any meaningful way provide long-term solutions to security issues (e.g. Aradau & Blanke, 2015; Ferguson, 2017). Like New Public Management, technological solutionism in the age of data governance, bolstered by marketing discourses coming from the private sector, might only be recreating a veil of illusion of technocratic control, while putting evermore distance between bureaucracies and the social world they wish to make more orderly.

In the process, these socio-technical assemblages charged with the surveillance and censorship also take our political systems further away from the rule of law. Legal norms and principles, once programmed into computer code governing algorithms, become diluted, invisible and therefore hard to challenge (Rouvroy, 2012). Indeed, when Google or Facebook develop algorithms aimed at ensuring the speedy removal of allegedly-terrorist content, who is able and allowed to make sure that this piece of code works as intended, safeguard protected speech, etc.?

In the course of the ongoing negotiations regarding online censorship between private companies and government officials, the later seems all to content to re-establish effective mechanisms to control the vast amount of public expressions flowing on the Internet while doing away with the need to invest resources in their police and justice system to do so, let alone to escape some of the “old safeguards” that our legal systems have developed overtime to protect freedom of expression but which make them ill-suited to process the surge in communications entailed by digital technologies (Tréguer, 2017b). In this regard, these new public-private assemblages governing data flows are an illustration of what some have called a “hybrid rule,” that is “a set of practices deployed by political elites that rely on the private sector to shield national security activities by expanding state power while constraining democratic accountability” (Hurt & Lipschutz, 2015). The line between the public and the private might be increasingly diluted, but the resulting arrangements have strong structural effects that need to be inscribed in the genealogy of modern state power.
References

Abrahamsen, R., & Leander, A. (2015). *Routledge Handbook of Private Security Studies*. Routledge.

Abrahamsen, R., & Williams, M. C. (2010). *Security Beyond the State: Private Security in International Politics*. Cambridge University Press.

Aradau, C., & Blanke, T. (2015). The (Big) Data-security assemblage: Knowledge and critique. *Big Data & Society*, 2(2), 2053951715609066. https://doi.org/10.1177/2053951715609066

Assange, J. (2014). *When Google Met Wikileaks* (1ST edition). O.

Ball, K., & Snider, L. (2014). *The Surveillance-Industrial Complex: A Political Economy of Surveillance*. Taylor and Francis. Retrieved from http://www.scopus.com/inward/record.url?eid=2-s2.0-84906009380&partnerID=40&md5=aa053badcfba61421cd51546a352e42c

Barbrook, R., & Cameron, A. (1995). The Californian ideology. *Mute*, 1(1), 44–72. https://doi.org/10.1080/09505439609526455

Barlow, J. P. (1996, February 9). A Cyberspace Independence Declaration. Retrieved from https://w2.eff.org/Censorship/Internet_censorship_bills/barlow_0296.declaration

Barty-King, H. (1980). *Girdle Round the Earth: History of Cable and Wireless*. London: William Heinemann.

Bauer, M., Lee-Makiyama, H., van der Marel, E., & Verschelde, B. (2014). *The Costs of Data Localisation: Friendly Fire on Economic Recovery* (ECIPE Occasional Paper No. 12). Brussels: European Centre for International Political Economy. Retrieved from http://www.ecipe.org/app/uploads/2014/12/OCC32014_1.pdf

Bigo, D. (2016). Sociology of Transnational Guilds. *International Political Sociology*, 10(4), 398–416.

Bigo, D., Jeandesboz, J., Martin-Mazé, M., & Ragazzi, F. (2014). *National programmes for mass surveillance of personal data in EU Member States and their compatibility with EU law* (Study for the European Parliament's Committee on Civil Liberties, Justice and Home Affairs No. PE 509.979). Brussels: European Parliament. Retrieved from http://www.europarl.europa.eu/RegData/etudes/etudes/join/2014/509979/IPOL-LIBE ET(2014)509979_EN.pdf

Black, E. (2012). *IBM and the Holocaust: The Strategic Alliance Between Nazi Germany and America's Most Powerful Corporation* (Expanded Edition edition). Washington, DC: Dialog Press.

Boltanski, L., & Chiapello, È. (1999). *Le nouvel esprit du capitalisme* (Essais). Paris: Gallimard.
Bonelli, L. (2010). Les modernisations contradictoires de la police nationale. In *L'État démantelé* (pp. 102–117). La Découverte. Retrieved from https://www.cairn.info/l-etat-demantele--9782707160195-p-102.htm

Bourdieu, P. (2014a). *On the State* (1 edition). Cambridge Malden, MA: Polity Press.

Bourdieu, P. (2014b). Séminaires sur le concept de champ, 1972-1975. *Actes de la recherche en sciences sociales*, (200), 4–37.

Brito, J., & Watkins, T. (2011). *Loving the Cyber Bomb? The Dangers of Threat Inflation in Cybersecurity Policy* (Working Paper). Mercatus Center, George Mason University. Retrieved from http://mercatus.org/publication/loving-cyber-bomb-dangers-threat-inflation-cybersecurity-policy

Brust, R. (2012). Letters of the Law: National Security Letters Help the FBI Stamp out Terrorism but Some Disapprove. *ABA Journal*, 98, 56.

Casilli, A. A. (2017). Digital Labor Studies Go Global: Toward a Digital Decolonial Turn. *International Journal of Communication*, 11(0), 21.

Chander, A., & Lê, U. P. (2015). Data Nationalism. *Emory Law Review*, 64(677), 678–739.

Charbon, P. (1991). Genèse du vote de la loi de 1837, origine du monopole des télécommunications. In C. Bertho-Lavenir (Ed.), *L'État et les télécommunications en France et à l'étranger*, 1837-1987 (pp. 11–22). Genève: Librairie Droz.

Christakis, T. (2017). *Data, Extraterritoriality and International Solutions to Transatlantic Problems of Access to Digital Evidence*. *Legal Opinion on the Microsoft Ireland Case* (*Supreme Court of the United States*) (SSRN Scholarly Paper No. ID 3086820). Rochester, NY: Social Science Research Network. Retrieved from https://papers.ssrn.com/abstract=3086820

Coleman, G. (2011). Hacker Politics and Publics. *Public Culture*, 23(3 65), 511–516. https://doi.org/10.1215/08992363-1336390

Deibert, R. J. (2013). *Black Code: Inside the Battle for Cyberspace*. Random House LLC.

Doyle, C. (2015). *National Security Letters in Foreign Intelligence investigations: Legal Background*. Washington, D.C: Congressional Research Service. Retrieved from https://fas.org/sgp/crs/intel/RL33320.pdf

Easton, D. (1957). An Approach to the Analysis of Political Systems. *World Politics*, 9(3), 283–400.

Ferguson, A. G. (2017). *The Rise of Big Data Policing: Surveillance, Race, and the Future of Law Enforcement*. New York: New York University Press.

Flichy, P. (1991). *Une histoire de la communication moderne : espace public et vie privée*. La Découverte.
Foster, J. B., & McChesney, R. W. (2014). Surveillance Capitalism: Monopoly-Finance Capital, the Military-Industrial Complex, and the Digital Age. *Monthly Review*, 66(3). Retrieved from http://monthlyreview.org/2014/07/01/surveillance-capitalism

Fuchs, C. (2013). Political Economy and Surveillance Theory. *Critical Sociology*, 39(5), 671–687.
https://doi.org/10.1177/0896920511435710

Fuchs, C., & Trottier, D. (2015). Towards a theoretical model of social media surveillance in contemporary society. *Communications-European Journal of Communication Research, 40*(1), 113–135. https://doi.org/10.1515/commun-2014-0029

Gasparin, A. de. (1837). Exposé des motifs et projet de loi sur les lignes télégraphiques présentés par M. le ministre de l'Intérieur. In *Procés-verbaux de la chambre des députés* (pp. 189–196).

Girard, R. (2002). What Is Happening Today Is Mimetic Rivalry on a Global Scale. *South Central Review*, 19(2/3), 22–27. https://doi.org/10.2307/3189861

Glaser, B., & Strauss, A. (1999). *The Discovery of Grounded Theory: Strategies for Qualitative Research*. Aldine.

Goldsmith, J., & Wu, T. (2006). *Who Controls the Internet?: Illusions of a Borderless World*. Oxford University Press, USA.

Haggerty, K. D., & Ericson, R. V. (2000). The surveillant assemblage. *British Journal of Sociology*, 51(4), 605–622.

Harris, S. (2014). *@War: The Rise of the Military-Internet Complex*. Boston: Eamon Dolan/Houghton Mifflin Harcourt.

Headrick, D. R. (2012). *The Invisible Weapon: Telecommunications and International Politics, 1851-1945* (Reprint). OUP USA.

Hibou, B. (2012). *La bureaucratisation du monde à l'ère néolibérale*. Paris: La Découverte.

Hurt, S., & Lipschutz, R. (Eds.). (2015). *Hybrid Rule and State Formation: Public-Private Power in the 21st Century* (1 edition). Routledge.

Jacomy, M., Girard, P., Ooghe, B., & Venturini, T. (2016). Hyphe, a Curation-Oriented Approach to Web Crawling for the Social Sciences. In *International AAAI Conference on Web and Social Media*. Köln, Germany: Association for the Advancement of Artificial Intelligence. Retrieved from https://hal.archives-ouvertes.fr/hal-01293078

Kuner, C. (2015). Data Nationalism and Its Discontents: A Response to Anupam Chander & Uyê P. Lê. *Emory Law Journal Online*, 64(2089). Retrieved from http://law. memory.edu/elj/elj-online/data-nationalism-its-discontents.html

Lécuyer, C. (2007). *Making Silicon Valley: Innovation and the Growth of High Tech, 1930-1970*. Cambridge, Mass.: The MIT Press.
Levine, Y. (2018). *Surveillance Valley: The Secret Military History of the Internet*. PublicAffairs.

Levy, S. (2001). *Crypto: How the Code Rebels Beat the Government Saving Privacy in the Digital Age* (1st edition). London: Penguin Books.

Lyon, D. (2015). *Surveillance After Snowden*. Polity Press.

Manes, J. (2015). Online Service Providers and Surveillance Law Transparency. *Yale Law Journal Forum*, 125, 343.

Mattelart, A. (2010). *The globalization of surveillance*. Polity.

McChesney, R. W. (2013). *Digital Disconnect: How Capitalism is Turning the Internet Against Democracy*. The New Press.

Mills, C. W. (1959). *The Power Elite*. Oxford University Press.

Mitchell, T. (1991). The Limits of the State: Beyond Statist Approaches and Their Critics. *The American Political Science Review*, 85(1), 77.

Morozov, E. (2013). *To Save Everything, Click Here: The Folly of Technological Solutionism*. New York: Public Affairs, U.S.

Mosco, V. (2014). *To the Cloud: Big Data in a Turbulent World*. Boulder: Routledge.

Pasquale, F. (2015). *The Black Box Society: The Secret Algorithms That Control Money and Information*. Cambridge: Harvard University Press.

Peoples, C., & Vaughan-Williams, N. (2010). *Critical Security Studies: An Introduction*. Milton Park, Abingdon, Oxon; New York, NY: Routledge.

Pohle, J., Hösl, M., & Kniep, R. (2011). Analysing internet policy as a field of struggle. *Internet Policy Review*, 5(3), 426–445.

Poupard, G. (2016). Towards European Digital Sovereignty. *The European Files - Cybercrime, Cybersecurity and Cyberdefense in Europe*, (40), 36.

Powers, S. M., & Jablonski, M. (2015). *The Real Cyber War: The Political Economy of Internet Freedom* (1st Edition edition). Urbana: University of Illinois Press.

Project, B. C. (2016). “Don’t Panic”: Making Progress on the “Going Dark” Debate. Cambridge: Berkman Center for Internet and Society. Retrieved from https://cyber.law.harvard.edu/pubrelease/dont-panic/

Roberts, S. T. (2016). Commercial Content Moderation: Digital Laborers’ Dirty Work. In B. M. Tynes & S. U. Noble (Eds.), *The Intersectional Internet: Race, Sex, Class, and Culture Online* (pp. 147–160). Peter Lang Publishing.
Rogers, M., & Eden, G. (2017). The Snowden Disclosures, Technical Standards, and the Making of Surveillance Infrastructures. *International Journal of Communication, 11*(0), 22.

Rouvroy, A. (2012). The end(s) of critique : data-behaviourism vs. due-process. In *Privacy, Due Process and the Computational Turn*. Routledge. Retrieved from http://works.bepress.com/antoinette_rouvroy/44

Rubinstein, I., & Van Hoboken, J. (2014). Privacy and Security in the Cloud: Some Realism About Technical Solutions to Transnational Surveillance in the Post-Snowden Era. *Maine Law Review, 66*(2), 488–533.

Schiller, D. (2000). *Digital Capitalism - Networking the Global Market System* (Reprint). MIT Press.

Schiller, D. (2014). *Digital Depression: Information Technology and Economic Crisis* (1st Edition edition). Urbana, Chicago: University of Illinois Press.

Schulze, M. (2017). Clipper Meets Apple vs. FBI—A Comparison of the Cryptography Discourses from 1993 and 2016. *Media and Communication, 5*(1), 54–62.
https://doi.org/10.17645/mac.v5i1.805

Siokas, E. (2018). Network Analysis of EU-Funded R&D Collaboration in the European Security Research Programme: Actors and Industries. In *The Emergence of EU Defense Research Policy* (pp. 221–245). Springer, Cham. https://doi.org/10.1007/978-3-319-68807-7_12

Tilly, C., & Tarrow, S. (2015). *Contentious Politics* (2nd edition). New York: Oxford University Press.

Tréguer, F. (2015). Hackers vs States: Subversion, Repression and Resistance in the Online Public Sphere. *Droit et Société, 91*(3), 639–652.

Tréguer, F. (2016a). From Deep State Illegality to Law of the Land: The Case of Internet Surveillance in France. Presented at the 7th Biennial Surveillance & Society Conference (SSN 2016): "Power, performance and trust". Retrieved from https://halshs.archives-ouvertes.fr/halshs-01306332/document

Tréguer, F. (2016b, October 26). French Constitutional Council Strikes Down "Blank Check Provision" in the 2015 Intelligence Act. Retrieved October 27, 2016, from http://verfassungsblog.de/french-constitutional-council-strikes-down-blank-check-provision-in-the-2015-intelligence-act/

Tréguer, F. (2017a). Intelligence Reform and the Snowden Paradox: The Case of France. *Media and Communication, 5*(1), 17–28. https://doi.org/10.17645/mac.v5i1.821

Tréguer, F. (2017b, November 2). *Pouvoir et résistance dans l'espace public : une contre-histoire d'Internet (XVe-XXIe siècle)* (phdthesis). EHESS - Paris. Retrieved from https://halshs.archives-ouvertes.fr/tel-01631122/document

Wagner, M. (2009). Warrantless Wiretapping, Retroactive Immunity, and the Fifth Amendment.
*George Washington Law Review*, 78, 204.

Webb, J., Schirato, T., & Danaher, G. (2002). *Understanding Bourdieu*. SAGE.

Weinstein, B. (2015). Legal Responses and Countermeasures to National Security Letters. *Washington University Journal of Law & Policy*, 47, 217.

Williams, M. C. (2010). The Public, the Private and the Evolution of Security Studies. *Security Dialogue*, 41(6), 623–630. https://doi.org/10.1177/0967010610388210

Zuboff, S. (2015). Big Other: Surveillance Capitalism and The Prospects of an Information Civilization. *Journal of Information Technology*, 30(1), 75–89.
https://doi.org/10.1057/jit.2015.5
# Appendix 1: Roundtable “The Internet, Private Actors and Security Challenges” (CERI, October 9th, 2017)

## Programme

In the first of three conferences to be held over the next year, Didier Bigo (CERI-Sciences Po), Laurent Bonelli (ISP-Paris-10 Nanterre) and Sebastien-Yves Laurent (CMRP-Bordeaux) from the ANR project UTIC are bringing together representatives of major online service providers for a high-level experts roundtable.

Participants will look at the ways in which technology firms engage with policy-makers and law enforcement agencies to address today's major security challenges: How did their relationship with intelligence and law enforcement agencies evolve amidst heated post-Snowden debates on surveillance and privacy? What are the main legal hurdles faced by online service providers to protect the rights of their users, and what changes in legislation are called for? How do these companies adapt their business practices to help address today's security challenges?

By looking at these important issues at the intersection of policy, law and technology, the roundtable will analyse public-private relationships in the fields of surveillance and security, offering an opportunity for a much-needed discussion between key international stakeholders and researchers.

To facilitate the discussion, the roundtable will be divided in two parts during which representatives of leading Internet companies will share their insights in interaction with researchers. The audience will have an opportunity to join the discussion during Q&A sessions.

### Participants:

- Acadia Senese - Google, Legal Counsel, Law Enforcement & Information Security

- Nicole Jones - Google, Senior Law Enforcement & Security Counsel

- Marc Mossé - Microsoft Europe, Director of Government Affairs

- Gail Kent - Facebook, Director for global cybersecurity, surveillance and law enforcement

- Maud Sacquet - Computer & Communications Industry Association, Senior Manager for Public Policy

### 14h00-14h30: General introduction

Alain Dieckhoff, Director of CERI-Sciences Po and Didier Bigo, Sciences Po-CERI, UTIC Coordinator

### 14h30-16h15: Roundtable discussion (first part)

In the first part of the discussion, representatives of leading Internet companies will share insights about how post-Snowden controversies on surveillance have impacted their work and relationships with other stakeholders, and what changes in their business practice ensued

Chair: Olivier Chopin, Deputy Director of the Reims campus, Head of Academic Affairs

Discussant: Félix Tréguer, Sciences Po
## 16h45-18h30: Roundtable discussion (second part)

In the second part, we will focus on today's key legal priorities for Internet companies, who have to face a fragmented legal environment and strike a balance between competing interests. What are today's main legal hurdles and what are the opportunities for reform?

Chair: Alex Mcleold, UQAM-Montréal

Discussant: Joris van Hoboken, University of Amsterdam

Q&As session

## 18h30-19h00: Concluding remarks

Ronald Deibert, Citizen Lab, University of Toronto

Thierry Delville, Ministerial Delegate for the Security Industries and the Fight Against Cyber Threats

# Draft transcription by Barthélémy Michalon

Introductory words by Alain Diekhoff, director of the CERI

Good afternoon and welcome to the centre for international studies which I have the pleasure to head for some years now

It's my pleasure to welcome you to this conference on the Internet private actors and security challenges. So this conference is the first of three conferences to be held over next year. It's part of the ANR UTIC which is coordinated by Didier Bigo, Laurent Bonelli and Sébastien-Yves Laurent. Didier is professor here in Sciences Po, Laurent Bonelli is at the university of Paris-Nanterre and Sébastien-Yves Laurent is in Bordeaux.

This ANR (national research project) is this afternoon bringing together representatives of major online service providers and we'll start the discussion on things which are of course important nowadays, which are the security challenges brought by the tremendous development as we know, sometimes unfortunately I would say, of the Internet.

Anyway I am happy to welcome all those people here on this round table for the whole afternoon. We have several distinguished chairs and also discussant which will be introduced by Didier in a moment inspired. I am pretty sure that you will have a good start today for the first conference and very fruitful debates for the whole afternoon. Very briefly, for those who don't know CERI: it is the largest research unit here in Sciences Po, with about sixty full-time researchers and professors (I do not go into the details of the status which are quite different and complex for non-French and even for French sometimes I would say). The centre is dealing with very different issues of course: [boundary?] studies, international relations, international political sociology, the question of defence, the question of security, which are different matters of course have always been quite important here at CERI and I must say that Didier has played a very important role in the last at least 25-30 years in this field and I see that with this conference he's going on with his deep involvement in Security Studies.

Without taking more time, I will give the floor [to Didier] and he will introduce the analysts and the whole project maybe this afternoon.

[APPLAUSE]

[3'30 - Didier Bigo]

I will try to be very brief also, but just in a nutshell, to give you an idea of what is this project called UTIC or RESO: Renseignement, Espionnage, Surveillance, Obéissance, or in English Intelligence, Espionnage, Surveillance and Obedience (RESO, Network and ISO for norms, technical norms and how they work). What do we want to do? We began this project in 2015 and it will continue until 2019.
As Alain said, it involves three universities, so we have here Sciences Po with the CERI but we have also Nanterre and Bordeaux but we are also a lot in touch with all the universities around the world, we have contacts with a lot of people working in Germany, in the UK, also in Canada and we will see that with also the people arriving and I think it is important because it cannot be addressed as a national question.

This is sometimes one of the problems of the framing of the research because we have a lot of projects. We are by far not the only one to work on that but what we have seen is that most of the questions, especially in the US, are organised regarding the national security of the US. Or the connection with the so-called GAFA and the way they are organised together and it turns a little bit as if by definition there is a centre and there are the margins. Maybe we have to think more transversally and to try to understand what are the relations at the transnational level between all these different aspects.

So the first point that we wanted to analyse is really to review today's communications surveillance technology and the transformation brought by new computing capacities, integration platform or the so-called big data analytics because here of course you have a lot of research and very good historians who have done research about specific agencies: the history of the NSA, the history of the GCHQ, a little bit less about DGSE (we have good historians nevertheless in the room), and also about the other intelligence services, what are the connections between the ones which are not specialised in signal intelligence or on the Internet but in other activities anti terrorism, policing, law enforcement and how they are connected.

What is certainly important is to try to understand how Internet has modified or not - and that's really a question - the logic of surveillance: is it just an acceleration, an intensification, or do we have to reframe our reasoning when we speak about the Internet if we compare let's say with what is now an old story - which is not so old - about Echelon and the signal intelligence communication? So this is one of the elements and of course we have a lot of discussion about what does it mean to collect data in bulk. I suppose that we will come back to this discussion at some point.

The second is to understand the variation of the relation to technology among and between law enforcement agencies and their respective countries as well as the justification underlying surveillance practice. Also, to be very simple, just in a way of introduction: a lot of people consider that because technology can do it, it's done and so in some way a lot of NGOs also want to be activists, with sometimes very good reasons, have the tendency to accept that: because it's possible it's done. So what are the social use of Internet? What are the social use of the Internet by intelligence services? How far do they go into this question?

[9'06] And the third is certainly to try to analyse how this question of what they call national security and claims to secrecy is connected now with the question of the global nature of Internet. So it looks like it really turns around security studies but that's not what we want to and that's crucial. Because if we analyse the question through the perspective of security studies we immediately introduce a major bias which is to take the position of one actor as if it was the central actor and that this actor are the different states or the interstate relations and not to look at the different agencies, the relation and competition there are between them and also to understand that in many cases they are not at all in a position of controlling everything, that they may on some point but it's not just, let's say, this kind of vision where you have a state surveillance of an Orwellian kind and its diffusion: it's certainly more complex than that, it's more horizontal.

[10'37] But at the same moment you have lot of inner resistance at all levels. And typically with the question of if national security government is about the fear, the anxiety of the [...?], of global insecurities, one of the questions has been: what is the solution? National security, [versus] global insecurity. An international collaboration of states and agencies regarding what they call themselves sensitive information on suspect groups? Yes, it looks like a solution but if you have the acquisition of information globally, if you have links between the relevant information and if you want policies of prevention, you arrive to a second major tension: national security, shared secrets but at the national level [versus] acquisition of information on suspects at the global [level]. So your national security is purely national arrived at what? A form of egoism, a form of incapacity to regulate nationally. So, what's going on? If it's seen as insufficient you have to have a belief in technology, you have to consider, and that's maybe one of the frame which has to be discussed, that, for most of them, technology will resolve what political judgement can not. In that case, they focus their attention on what the technology is regarding mobility of people, mobility of capital and, centrally, mobility of information.

[12'43] And then, what happened was that the focus of many intelligence services has been toward the Internet as a source of organising their new activities and the old ones with different methods. That's why, certainly beyond signals intelligence, investigating the Internet has become a key element of this logic. But the management of data is done by private actors, so what are the relations? These actors have been put in a situation of constraint, to be obliged to deliver some data, through changes in the legislation. They have tried (and there the category of private actors [encompasses] a lot of different companies) and they did not have all the same reactions (it will be interesting to discuss about that). How
to comply to legislation but how also to give to your consumers forms of protection, security about their own rights?

[14'13] So a lot of companies who were in charge of managing data have been more and more included into different stories about what kind of relations we may have with our public, but this public has privacy rights, they are citizens, they are citizens and foreigners, they live somewhere but they use Internet to be connected... So what is the logic of rights and what are the forms of resistance?

[14'51] I will stop here, because I don't want to “destroy” the discussion by imposing some views. On the contrary, I think what is important is just to be ready to deconstruct some categories we spontaneously have (the state, the private actors, the others...) and to blur the lines. All of these actors and the relations they have. So I will leave now the floor but I will first ask the president of the first round table to introduce our different partners. I really appreciate that you have accepted to come.

[15'45 - Olivier Chopin] Thank you Didier. [...] My name is Olivier Chopin, I am member of Sciences Po and the academic director of the Reims Campus, so for the undergrad programme in Sciences Po. My personal research deals with intelligence studies and surveillance and I thank Didier Bigo for asking me to chair this first round table today.

[16'20] I am very thrilled to introduce the first discussion and a lot of high level representatives of leading Internet companies will share with you today their insights and their reflections on how controversies that happened after Snowden's revelations affected their works, their relationships between themselves, with the judiciary, with governments and all the different actors involved in the story, even of course the general public and the audience. We would love to have a real dialogue, not only a discussion but a dialogue between yourselves: how it affected and changed your business or your representation of your own activity.

[17'20] Félix Tréguer will be your discussant in a few minutes: he will listen to everything you say and will have a discussion with you here within the panel and will keep around 30 minutes to have questions with you [in the audience]. Of course do not hesitate to prepare your questions and to ask questions later.

[17'44] As we have enough time, maybe we can start with a personal presentation by yourselves, so I will leave you the floor for two minutes each of you to present yourself and give maybe some background and some context about your activity and how it will impact your discussion in a few minutes.

[18'06 - Acadia Senese] Good afternoon everyone and thank you so much to the centre for having us here this afternoon. We are just so thrilled to be able to participate in the discussion in this topic which is near to me. I'm attorney in the information security team, advising Google on data disclosure requests from around the world and as part of my work I also support the policy team for the work they are doing globally to update and reform surveillance law. Prior to joining Google I was a federal prosecutor within the Department of Justice and so working for law enforcement with working cases I used to be on the requesting end, sending requests to companies to obtain data so I now have an interesting vantage point being on the receiving end of those requests, having to balance conflicting laws across the world and in advising Google on their data disclosure obligations.

[19'12] But I think you know as an opening remark we very much agree that surveillance reform is a transnational question, it's not one that particular to any one country and it is really going to require agreement of nations to reform our surveillance laws. We're doing quite a bit of work and updating, er, insisting that the US Congress updates laws that allow companies to be able to respond to demands from other requesting nations (and I will talk a little more about that) but it's certain and we agreed that in an era where technology has proliferated in our everyday lives where there are real security threats to individuals but also a real interest in privacy that the law needs to be updated to reflect all of those things. Anyway, thanks very much, looking forward to having a discussion with my fellow panelists and all of you today.

[20'18] I am Marc Mossé, from Microsoft Europe. Hi everyone, thank you, just a few words about who I am: I am working at Microsoft, I am leading the Government Affairs team in Brussels. Prior to that I was the government affairs and legal director for France, for about ten years, and before I was an attorney at the Paris bar association and I used to work with Robert Badinter, the former Chief Justice in France. I worked with him for five years. So I have some appetite from this time on civil liberties, fundamental rights, and all these topics we are covering today, aside of technology. And just to add one sentence on what you have said: all these topics should be treated through a transnational approach, for sure.

[21'24] I am Gail Kent from Facebook and apologies for being slightly late since I have come from Brussels. I lead for Facebook globally on security, law enforcement and surveillance which means three things in practice: when we are developing new products I look at what the likely impact is going to be on law enforcement, both from how people might use these products but also what details and what information law enforcement are likely to be asking us. I also provide advice to our colleagues around the world and I think we are going to talk a lot about the transnational elements
of surveillance today, when there are new legislations proposed in any country and then lastly I contribute to the debate
on some of the huge issues that we're talking about today, whether that's encryption, whether that's about mutual legal
assistance reform, whether that's what cybersecurity looks like in 2017 and onwards.

[22'34] The two things I draw from doing that and doing my current role are 20 years I spent in London in law enforcement in the UK as part of the National Crime Agency, leading on international operations and lately on international cybercrime operations that included a lot of working transnationally, including with Europol and Interpol and also I think now 5 years in academia, I had the great joy of landing in Stanford on a Fulbright Fellowship the day after Snowden [matters?] revelations: looking at transnational data sharing and MLAT reforms, that was quite a leak from, you can imagine, UK law enforcement and the civil service and to Stanford at the height of the Snowden revelations.

[23'28] So I am very much looking forward to enter this discussion today and I will definitely echo what both Microsoft and Google have said - and I suspect you won't disagree with each other quite a lot - but I think there are two things we have to think about here, or three things probably: One is that it is absolutely a transnational problem and I was in the US last week and the same issues I am sure we are discussing here today were discussed there. Secondly that there has to be greater transparency in these discussions: I think it is incredibly surprising the lack of transparency that exists in most countries around the world about what surveillance legislation looks like and even within Europe there are a lot of countries that are not transparent about how they carry out surveillance, even what they call surveillance. And then the last thing, and I think it is one of the reasons I am so happy to be here today, is that this can't just be a conversation only between governments or a conversation between companies, this also has to involve everyone talking together and a key part of that is academia and the NGO community because we can't do any of this unless we're clear that we've got to have the right human rights frameworks. Thank you very much.

[24'46 - Olivier Chopin] Thank you Gail, you anticipated my first question which would be the day after Snowden but first let's welcome Maud Saquet, who works in the Computer and Communications Industry Association in Brussels.

[25'07 - Maud Sacquet]. Exactly. Well, It's the CCIA for short, which, if you want the story, used to be known as the Communication Industry Association and after a few weird phone calls to the CIA, the actual real CIA asked us to change our name, to add another letter. Officially now it's the CCIA, so it's a trade association that represents tech companies in Washington and Brussels. On my side I am based in Brussels and work, I would say, mainly now on content issue and availability of online intermediaries, so content removal, copyright, audiovisual, etc. but I follow also from a slightly greater distance all the surveillance and law enforcement issues that are happening in Brussels, mostly because of my background, because prior to joining CCIA I was working for Yahoo! here in Paris and during those five years I was handling law enforcement relationship with French authorities so practical experience into what it means to receive a data request and then decide what kind of data should be handed over to law enforcement.

[26'22] There is no much I can add to what has already been said in the introduction: the transnational aspect of the discussion has been highlighted and transparency as well. This is something CCIA has been fighting for for a long time: I mean having offices both in Washington and Brussels gives us quite a unique opportunity to work on those matters on both sides of the pond and try to co-ordinate what is being done between Washington and Brussels on these matters so that is indeed very close to our heart on these issues.

[26'50 - Olivier Chopin]. Thank you very much. My first question will be what you record from the day after Snowden, at the beginning of June 2013, revealed that he had a lot of classified information proving that the NSA had been able to tap directly into Verizon, first, and the day after The Washington Post and the Guardian revealed that the secret program called PRISM gave the NSA the ability to access servers from the biggest tech companies, and especially apple Google and Microsoft. So that was like a thunder, of course, those moments were recorded: you may know the documentary "Citizenfour" by Laura Poitras or even some fiction movies have been made about Snowden's life, so it was really a turning point in the history of surveillance. We of course as specialists could suspect a lot of those surveillance activities but then there was public proof of all those activities. So our first question is very simple and maybe even a personal one: Do you remember the moment - although you have already said a little bit about that - when this information was released? Were you already working on those issues: surveillance, civil liberties, the probability that some day the surveillance capabilities would be revealed to the general public? How did it impact you at the time? Maybe we can start with you, Acadia.

[28'43 - Acadia Senese]. Sure: I was actually still with the government when the Snowden's revelations hit, so I can't speak about what Google was feeling or thinking at the time, although I understand from my colleagues who were living and breathing these Snowden revelations at Google that there were 1) surprised by the revelations but 2) also frustrated because the portrayal of the revelations was not always necessarily accurate, so there was frustration with the government for its failure to correct and fix the inaccurate portrayals of some of the programs that were being discussed. But then on the other side of that, there were already ongoing efforts at Google to 1) deploy SSL across Google systems 2) to engage for example into digital due process coalition and to argue for surveillance reform and 3) to educate our
users about security, like using two-factor authentication. So Google didn't necessarily make any changes, although it
certainly accelerated ongoing efforts that already existed at the time in terms of security, assistance [?] and educating its
users.

[30'06 - Marc Mossé] I tried to remember what I was doing this day - you told me before the panel that you would ask this question, so I tried to figure out where I was. In fact I was reading the book, which is not a George Orwell book, but is a book maybe less well-known and I would recommend you to read it: it is "Le Palais des Rêves" (The Palace of Dreams), by Ismail Kadaré. It's a very interesting book because the main character of this novel is a guy with a very precise role, which is to monitor the dreams of people at night and come through the secrecy of life of everyone until the day he realises that on the list he will be the next whom the dream would be checked and monitored. So he realises that this day you are monitoring the dreams of others but one day it could come to your dreams. It's not just an anecdote because it was true, I was reading this book at that time, a very short novel and I would recommend you to read because it's a very nice way to think about where we are going today.

[31'39] On the technologic side, as Microsoft we were already working on the security and we had made at this time already a lot of investment and one entity which is called Digital Crime Unit because cyber security as a whole is very important for the company we were working on this and on the law and the general framework on the access to data which is not only surveillance but also the judicial requests because everyday as a telecom operator we get a lot of requests on investigations on the web. So we were already working on this and of course we accelerated, we sped up on this topic because immediately, and this is my immediate, I would say, feedback to these days, in addition to Ismail Kadaré's book, it was the questions from the customers and how trust is so important in the use of technology and in particular in a time when technology would be everywhere. The question of trust really went on the top of the discussion with the customers, with the government (not only the agencies but also the policy-makers), because at that time they realised that this question of trust could impact a lot the development of technology across the world.

[33'18]. If I had to summarize these days, I would say, first, read the book because it's a really nice way to think about the importance of fundamental rights at any time, the principles and the values are timeless and second how trust is important in the technology world.

[33'46 - Gail Kent] So I had just landed in California and I had come from the UK government so I don't know how many of you worked in European civil service but you can imagine that they are much more rigid than in California. In California in itself it was a bit of a shock and it was very very different from the Home Office and the National Crime Agency, where I was in London, to arrive in Stanford. And I was working with Jennifer Gaskell - sorry, for Jennifer Granick<fnref n="168" /> - who was one of the people that really strongly pushed for reform on the basis of the Snowden revelations, so I had gone from working for a conservative (with a large and small “C”) government to be with one of the most reformers - or advocates for US surveillance reform, so it was a huge cultural shock. I think I can say that it was one of the things that absolutely came out in my time at Stanford was realising that one of the most important things that we need to do - and I think you will hear me say this a lot today - is to try to understand different perspectives.

[35'05] But if I go back a bit to what I was thinking and what was Facebook thinking: obviously, I wasn't in Facebook at that time because it happened another two years before I joined but exactly the same as Acadia Senese has said: looking back into our understanding of a lot of details now, there was a lot of shock and surprise and the fact that these allegations were being made and the fact that the US government was not correcting them. Exactly the same as the other companies, the three things that Facebook started doing was making sure that we doubled on our own security - that is the most important thing to us: you cannot be the steward of two billion people's personal data without absolutely having to look after it properly and saving things securely. Pushing for reform and looking at what those standards would be - I'll come back to that - and then again working on who could be or who were the different groups that we could work with to do that. Another one [endeavour?] was to ditch the processes of the "Reform Government Surveillance"<fnref n="169" /> alliance, which all of our companies are part of and which looks to or pushes to clear standards and surveillance reform around the world.

[36'30] I think for me Snowden was not just about that this absolute shock personally and understanding this sort of clash of these two worlds, it's also been known for how it has meant, positive and not so positive, over the last four years. I think we have to remember that wasn't just one moment in time, since we have seen safe harbour in the discussions around the privacy shield, which is very much what can be the impact of US surveillance outside of the US, then the 702 debate<fnref n="170" /> which is still going until [it terminates? 37'12] will be renewed this year. For those of you who don't know that, this is the part of the US legislation that allows US law enforcement to make requests on companies like Facebook, Google or Microsoft for non-US citizens' information.

<footnote n="168">168. http://cyberlaw.stanford.edu/about/people/jennifer-granick</footnote>

<footnote n="169">169. https://www.reformgovernmentsurveillance.com/</footnote>

<footnote n="170">170. https://www.justsecurity.org/31098/702-reform-debate-heating/</footnote>
[37'30] And just as this genuine need for greater transparency: what we saw through Snowden was a reinforced need to discuss what actually was happening and further to be going for a better clarity and we saw that in the US, we also saw the impact on the other Five-Eyes countries, so the UK was obviously in the discussion around the Investigatory Powers Act<fnref n="171" />, New Zealand, Canada and Australia have also had their own debates. But I think one of the things that we just don't see enough are those sorts of debates within countries.

[39'09] Whilst there is a lot I can disagree with in the Investigatory Powers Act - one thing the UK did do was have an hopeful / awful lot of reviews and trying to get there and try to listen to... at least let a number of different voices talk in that discussion. I think that's something we certainly would like to see even more of, in these discussions we were listening to other groups but also, more importantly, understanding the impact of legislation in one country on another. That's not just directly on companies like ours, but what happens if a country that's less democratic copies legislation from a democratic country: what does that mean? What is the impact of surveillance legislation on Human Rights is not just privacy and freedom of speech but what impact does it have on people's ability to connect as well. I think that is something that we really need to be, sort of, moving towards having. I think it does come a break because of Snowden, these discussions about what actually surveillance means for all of us from all these different perspectives.

[39'22 - Maud Sacquet] For my part, I was still working at Yahoo! at the time of the revelations. Something I mostly remember is a lot of shock, a lot of e-mails, a lot of phone calls, from colleagues, from friends asking "what is it exactly you're doing?", perhaps a weird look from other teams saying, "well, what is legal doing actually, on the fifth floor?". So, a lot of questions, I will try to perhaps be too much, but try to understand where it was coming from, and especially trying, struggling to reconcile what I was reading in the press and what I was actually doing.

[40'12] For me on my day-to-day work I was doing my best to be helpful to law enforcement, to explain how things work, to be available, to help them in their investigations while at the same time protecting our users' rights and privacy, for example by interpreting as narrowly as I could a data request, to only hand in the strict necessary level of data [that I should?], which seemed the right balance I would say. So I was quite struggling to reconcile what I was hearing and what I was doing and then trying to explain it in a way that make sense to friends and family, actually wondering what my job was, suddenly. So that's perhaps the personal aspects of these revelations.

[40'56] From a purely perhaps policy side, and tech association policy side, so obviously I wasn't working there at the time but I also asked my colleagues about how they lived those revelations and what it changed for them in Washington mostly at the time. What was interesting from their feedback was that for them it was not new, in the sense that especially US surveillance reform had been something they had been working on for years, I mean since the adoption of the PATRIOT Act it has always been something that the tech industry was lobbying, I would say, in Washington and trying to rein in those kinds of programs as much as they could and partnering with academics, with NGOs, to try to feed a bit the discussion on those topics already pre-Snowden. One example again, so it comes back to Yahoo! (I think it will be the last example about Yahoo!), but that was following the Snowden revelations: Yahoo! was able to communicate on the fact but they were actually fighting against the US government in 2007 as far as I remember, calling some amendments of the surveillance program and fought indeed before the Foreign Intelligence Surveillance Court (FISC) because they thought that those amendments were overbroad and unconstitutional but [42'20] they lost but following these revelations they were able to declassify some of the proceedings of that Court. So it's just an example to show that really from a legal and policy perspective those facts were already happening before the Snowden revelations - I think that's the really first thing that my colleagues highlighted to me - while it was less visible at the time perhaps.

[42'40] The change that the Snowden revelations did for my colleagues was that suddenly those topics, which were part of a long list of topics we were covering, were pushed front and center, and that US surveillance reform suddenly became one of the main priority, the legislation we were working on and most of their lobbying efforts were being directed toward that topic.

[43'07 - Olivier Chopin]. Thank you. My next question will be a follow-up to Maud's answer: if now you look back to the four years that have passed since that shock and revelations, would you say that you learnt a lot of things about government surveillance, or did you have confirmation of things you know about because tech companies are involved, as you just said, in the process of collecting data, or did you revise your perception of what global surveillance from a government or from many governments can mean at some point? What did you learn, actually? Did you learn things and what was the most intriguing or most important thing that you discovered when Snowden did that revelation? Or maybe nothing at all? Who would love to answer first?

[44:16 - Gail Kent]. I think that what I learnt is what country you do not know anything about, so you now have a lot of information on what the US does, I mean you can argue you did not have enough, but we have all the Snowden

<footnote n="171">171. http://www.independent.co.uk/life-style/gadgets-and-tech/news/investigatory-powers-act-bill-snoopers-charter-spying-law-powers-theresa-may-a7503616.html</footnote>
revelations themselves, we have the various abuses that have taken place most notably under the privacy and civil
liberties, boards that we had to beat in Congress, we've had companies pushing for greater transparency, we still have
ongoing cases that the companies continue to raise, pushing for [exacting the amount of thing? – 44'58], for making
sure the US government is able to get the minimum and correct amount of information.

[45'09] France has also had a discussion around terrorism legislation, I think what is shocking here are the large number of countries that we genuinely do not understand what their legislation actually means. I see this in somebody whose job is reading this: there is a huge amount of opaqueness in surveillance legislation, often it is, I think Hungary, Polish laws, both of whom have had discussions in the last eighteen months, there we are looking at different layers of legislation on top of each other and try to understand what it means. I am not talking [specifically?] about those countries now but where there is a generic constitution you can almost do what you want for national security reasons.

[46'09]. That's why for me I am surprised there hasn't been more of a discussion and even within Europe, if this is in
your interest I recommend you all read the Fundamental Rights Agency report on surveillance in Europe because that
also shows you that even within the member states there is a huge variety of openness about what different countries do.

[46'38] On top of that I think we still are grappling with the issue, again something that I continue to learn as a result of the Snowden revelations: there are no standards for what good surveillance legislation looks like. And I speak and unapologetically believe there are times in which law enforcement should gain access to information: I have investigated cases, whether they were rapes, or murder, or people trafficking where I absolutely believe the companies have the responsibility to work with law enforcement in a Human Rights framework to help protect the Human Rights of the victims of these crimes but there still isn't clarity, even I think within Europe, let alone internationally, about what good surveillance legislation looks like.

[47'34 – Acadia Senese]. To follow up on Gail’s comments and also on Mark’s, I think one of the biggest revelation from the Snowden disclosures was, frankly, a lack of understanding of what legislative landscape looks like, both in the United States and outside the United States with respect to surveillance laws. That lack of understanding – to Mark’s earlier point – deeply contributed to a lack of trust among users. For companies like Google, users trust is critically important and one of the ways that we help users gain our trust [?!] is to be transparent with them about how we respond to government demands for user data. So in the wake of Snowden and this again is ongoing work, we have been fighting against the government to allow companies to be able to talk about the requests that we have been receiving, to disclose the number of requests, for example FISA requests and national security letters.

[48'36] That took lawsuit to do against the government to be able to disclose, in aggregate, the volume of requests that we are receiving, and we do that in the transparency report. But also it took lawsuits and ongoing lawsuits, some of which are still not public, to be able to describe to the public and the users what those requests look like, what the requests entitle the government to, how the companies define those requests and how the companies map those requests to the types of data that the companies may be able to disclose.

[49'10] So there's a common understanding among users and companies about how it is that we define metadata, how it is that we define content and how companies behave when they receive a request on those categories of data. So transparency is so important to us, it's important to the debate on surveillance reform but it's also important so that there is a common understanding about what it is we are discussing, what are the reforms we are working towards and so we can work together to drive coalitions to reform these laws.

[49'48 – Marc Mossé] Just a few words on this. In fact this is a very complicated issue because it's not only about surveillance, it could be about criminal investigation: we get requests every day from the authorities, I would say every [GOs?], every country. So it's very complicated stories and we absolutely need to frame the story in a way which is understandable and that we can fix with a concrete solution.

[50'21] Just a few comment here: as Google and Facebook we have a transparency report twice a year to give information on these requests. You get the statistics and you can access online to these reports: this is a first sign because, as you have said just now, it's a very important point to ensure transparency on this issue. Second aspect, and maybe the most important, is the rule of law. I do not want to be provocative but the question of public safety, privacy and fundamental rights is not a new topic, it's not a new debate. For years we had this, with our telecom (surveillance on your phones), even before the mobile phones we had this discussion about surveillance and access to data: it's not new. So because the debate is not new the values we are using are not new either. When we are talking about freedom of expression, privacy, freedom of communication, all these values are still valid and these are timeless values. It's not because technology is new that the values disappeared and the rule of law is critical here: yes, we give access to some data, only when the request is done according to the rule of law of the country where we operate. This is super important to consider.

[52:13] And I would love to give you one example because this is a post-Snowden story and maybe we'll talk about it
during the second round table this afternoon: The case we have today against the US government. Two years ago we had a request, a warrant from the US government, to have access to data stored in Europe and we refused to defer to this warrant. We went to the Court, we won before the Court of Appeal of New York and now the case may be taken by the Supreme Court: we'll know this in the coming days. It was critical for us because it was to set the US government. Of course you can ask us to access to data but only if you go through the rule of law. As a company, as [part of] the private sector and, I would say, as a citizen, it should be the North star. Principles and rule of law. Because, again, technology will not change the principle.

[53'30 - Maud Sacquet] There is not much I can add, to be honest. Perhaps just one point on what we learnt following the Snowden revelations. From a very Brussels perspective, I was wondering "what is actually the US doing?" We need to know, we need to understand and following a few years of debates, of reform, now the actual narrative in Brussels on those issues is: we know more about the US surveillance system than we do about the European ones. And that's an issue. Speaking for the Commission or Brussels institutions. And that's not normal. And so again, maybe (let's see how things evolve in the next few years), but maybe then the focus of the EU institutions will actually shift to what is actually being done within our borders rather than what is being done in the US, because we have now a clear understanding of what is happening over there. So that's really the change that we are seeing now, and that very striking, I would say.

[54'30 - Olivier Chopin] Now we are going to begin a series of more precise questions, but following up what you have just said: you talk about general principles and you publish transparency reports. That's maybe more for the general public than for your users or customers. How did your companies manage to restore confidence and trust, as a provider, to customers on a day to day basis? Because I believe not everyone who uses a Gmail account has read the transparency report from your company. So it's a general discourse. So how did you face the question of distrust between your customers and yourselves?

[55'18 - Gail Kent]. So I think the first thing that Facebook did was coming out straight away after the Snowden revelations and Mark Zuckerberg made a statement saying [that] we never have offered any government access, not to the US government or any [other] government to our servers. And that remains exactly the same today. So I think it's also us being transparent. I can't agree more with what Marc has said about trust as fundamental: if you lose the trust of your users, then they stop using your services and also, worse than that, they stop trusting the internet. And if we lose the trust people have on the Internet then we lose all the amazing benefits that we all gain through the access to connectivity that we have. So it's about being transparent, it's about people understanding when we do give over data, so we all publish those transparency reports and we also publish - the same as Google, Microsoft as well and a lots of companies - very clear guidelines for when we do provide data and what data we do provide to law enforcement. So it's being clearer now, maybe every user isn't look at the transparency reports and isn't looking at the amount of data their own country is asking for, but if they do want this information, it is available and it is there. We obviously do panels like this, where we are trying to talk as much as possible about what we do and to provide information. I think that, lastly, the most important thing we are consistently fighting for (both in Courts but also when there is new legislation and discussion like the Commission one on e-evidence) is the ability to notify our users if we are asked to provide their data. Now, something that we believe as incredibly important so when law enforcement comes and asks for any of your data, that we can say: "you know what, law enforcement has asked for those data". So we give the users themselves the ability to challenge that.

[57'48 - Olivier Chopin] That was not the case before Snowden's revelations, you came to that...

[57'54- Gail Kent] I am trying to remember whether it was the case in the US because I think it's actually part of... it comes under US law and again it's something we're [stumbling upon?] about just not knowing very much about the EU. One thing the e-evidence process has thrown up [?] is that user notification really varies from country to country, so there are some countries where it's obligatory, or [in as in most countries] obligatory unless it's going to undermine the investigation, they often can ask a prevention of a company notifying the user until the investigation reaches a certain point. Some countries do not mention it, some countries make it an offense in itself to notify the users, which is undermining due process. And I think that shouldn't be the case. One of the principles of government surveillance is the ability to tell a user that their data is being requested.

[59'03] We all see a huge variety of requests everyday, most requests that Facebook get are ones I think what most people in this room would agree with we should be providing that data. But they are ones where we definitely don't want to be providing that data or where we believe the user should have the right to contest them.

[59'30 - Acadia Senese] We have been similarly advocating for the ability to notify our users when we receive a request and I think here is an example where US laws are a little bit better on this point in the sense that we aren't prohibited under US law from notifying our customers of the existence of a request, unless it's accompanied by a specific non disclosure order. So in the United States unless we are prohibited from doing so we do notify our users and some of them go to Court and challenge the request. And that challenge happens while the company stays on the sidelines and
waits for the litigations to play out. So there is that avenue in the US and interestingly some of the states - California being one of them - require the law enforcement officers themselves to notify the users when they make a request to a provider. So even if a provider didn't notify the customer, law enforcement would be required to do so themselves. At least on that front we've made headway with respect to being able to provide notice to our users, to tell them not only that we have received a request but in some cases being able to provide a copy of that request so they can make a challenge in Court.

[1:00'50 - Marc Mossé] Very briefly, because most has been said. First, we reiterate the fact that we don't give access to all data. It's clear. We just provide access to data when there is a request compliant with the rule of law. It's super important. And even in this case, as you have just said, first we have the law enforcement department to go to the customer, first. It's really the principle. And if not we ask to be transparent with the customer and we won the case with the Seattle Court - I don't know if you are aware of this one - because the law enforcement agency refused that we disclose the request to the customer and we won this case. It's a really important point. I am sorry to be repeating myself here but, to understand how we work in this question of trust, we also go back to the rule of law.

[1:02'00] It's not our decision, this is really in the respect of the law, in the US but it could be as well in other countries, such as in Europe. We had a lot of discussion, even with the French government, to work on a framework which is really respectful of the principle.

[1:02'20 - Maud Sacquet]. Perhaps a few things. First on the possibility to inform the users whose data have been requested, I remember indeed when this concept was discussed at the European level, within the [countries?] of Europe at least, where US colleagues were surprised that we didn't have a ready answer as to whether we could disclose the data request to the users actually because it was very complicated to figure out, especially in national law, because it was not something that was taken into account when the law was written and so something that we could easily answer. So it stirred a lot of debate about what was actually possible under European law or French law.

[1:03'13] From a more "policy" perspective, this issue of transparency reports and of disclosure of the type and number of requests was something that was key. In our lobbying in Washington for US surveillance reform with the adoption of the USA Freedom Act a couple of years ago, more transparency, especially regarding the types of requests that we were receiving and the possibility for companies to disclose this request, that was one of the elements of the reform of US surveillance. That's something that the companies and the industry as a whole fought for, to include more clarification and more transparency on that point.

[1:04'00] The other and last interesting reform of US surveillance (and maybe that will be discussed later) is that the US government also adopted what is called the Judicial Redress Act, adopted at the beginning of 2016 if I am not wrong, in which the idea actually was to go back to the idea of trust. Well, how do we ensure that all European users and European citizens actually trust the US government and US service providers, that we are not going to do a mass collection of their metadata in surveillance programs like the one that Snowden revealed? Through this piece of legislation now Europeans have the right to review for inaccurate data to be corrected to [by?] US federal agencies [?], they have the right to challenge the fact that their data were shared by the government [companies, instead?] with US federal agencies. They have the right to challenge their own agency and US federal agency if their data wasn't fully disclosed [??]. I mean: there has been again some measures that were adopted at the US level to try to address the trust issue and to give European citizens similar rights than the ones American citizens can enjoy in the US on that matter. That was something really important that we fought very much for in Washington.

[1:05'37 - Olivier Chopin] Could some of you explain if (or not) you have some relations with advocacy centers for civil rights or privacy, or civil liberties. Did you interact with organizations like ACLU<fnref n="172" /> or FIDH in France? If you did, how? Could you give some examples of how you managed to access to them or work with them in order to advocate more for that trust with the government?

[1:06'10 - Gail Kent] Sorry for jumping in so quickly. As I said at the start, understanding all the different perspectives is really important and especially understanding what are the Human Rights implications of surveillance. It means talking to people who spent their time thinking about this in a huge amount of detail. So we have regular conversations with a lot of different NGOs who were thinking about this through a number of different perspectives. What I would say is these are things that we are also thinking about ourselves. This is not that we go to Human Rights groups to get the Human Rights perspective because we aren't capable of thinking about it. We absolutely are. One of the reasons why I joined Facebook was because of the mission Facebook has and because of the idea that we could connect the world and we can do that in a way that is connecting the world safely and taking into account Human Rights. I would say that possibly one of the misconceptions that people have is that the companies are all about power - and we'll talk about that later [...].

<footnote n="172">172. American Civil Liberties Union.</footnote>
[1:07'44] It's really important to be understanding what we are doing from all the different perspectives. Part of that is talking with people at the UCLA, the Electronic Frontier Foundation, CDT<fnref n="173" />, the Government Information Technology<fnref n="174" /> [?], Privacy International... and trying to get all those different perspectives. One thing I would like to say and that Maud was mentioning earlier is that there is a lot of debates in Europe around privacy, it's incredibly important to us as Europeans (I can still see that I'm a European at the moment, as a Brit).

[1:08'30] What I have found is that there isn't as much conversation around surveillance and what European governments are doing, so there is a slight vacuum compared to what is in the US for NGOs talking about this sort of issues. But when they exist I am really really happy to have those conversations, to make sure we can understand their perspectives and be given a different way of looking through things that we haven't thought through that already.

[1:09'02 - Acadia Senese] NGOs are critical components of the debate and the conversations that we're having between and with companies and the governments and the NGOs talking about surveillance reform, you know, RGS<fnref n="175" /> being one of the groups that Google participates in. I think one of the main reasons why the coalition and those working groups are so critical is the education that occurs when you're in those conversations and you are talking as a company to an NGO, to a government official about where the pinpoints are, what the needs are, what the concerns are, and trying to move towards consensus on what reform could be like, reform that includes basic human rights standards, due process of law, privacy protections... Then each one can bring something very different to the table but it's in those conversations, with all three groups in the room, when really (in my opinion anyways) the progress is being made.

[1:10'05 - Marc Mossé]. I would same the same: NGOs and citizens are critical in this process. It's a multi circular approach and the dialogue is not only between the Tech industry and the government, it's really the NGOs and the citizens involved in these discussions. If I look at the New York case, the one I mentioned earlier, we have seen in the Court of Appeal a lot of Amicus Briefs, a kind of intervention (which it is not exactly the case in terms of procedure) from the NGOs. You can go to our website, digitalconstitution.com<fnref n="176" />, and you will see all the briefs from these NGOs, they brought their arguments in the case to the Court. Here are the principles at stake and as part of this ongoing dialogue or, I would say, conversation, we absolutely need to have the citizens and the NGOs involved.

[1:11'09]. I just remember one thing ... Some years ago, when I was leading the French team, I have been heard by the National Assembly committee in charge of one of these reports on Fundamental Rights in Internet and we had a lot [of such reports?]. I have been questioned by the MP saying "in France and in Europe we are taking a lot of care of privacy and they have no interest in this in the US". I said "no, you're wrong" because at that time (I am talking about maybe 8 or 9 years ago, it was a report issued by Patrick Bloche and Jean-Yves Warsmann<fnref n="177" />), at that time the civil society, the NGOs in the US were much more involved and active on this topic than in France. Because often we consider that in France and Europe we are much more active but I've been impressed of that time by how the NGOs and the citizens were really involved on this topic in the US. And it came to my mind at this time that this story could go a little bit further if really, across the world, NGOs and citizens worked together.

[1:12'25] It's not EU against the US, it's how we build a bridge between our common values because in fact if you look at the reality of this question, it's about values that are largely exactly the same. They can be interpreted in different ways, we could have in the US, I would say, a more consumer approach to privacy rather than a Fundamental Rights approach in Europe - we can discuss this - but at the end of the day the principles are the same, the values are the same and what we have to build is a common approach on this question of principles.

[1:13'04 - Gail Kent] Can I just add something that is really really important? One thing I have seen a lot of is particularly (I think we'll come in the second half to discuss Mutual Legal Assistance Reform and I think we have already mentioned some of the reforms in the US Congress) EU groups complaining about US surveillance because that's not fair and US groups complaining about EU surveillance because that's not fair, which does sort of put companies very much in the middle, rather than, exactly as you said, a much more cross-cultural distinction about what would "good" look like. Because at the moment we can't be in a position where US groups can say "all this is OK in the US but you Europeans can't gain access to any data" or European groups saying "It's OK what's happening in Europe but you, US are really really bad and you are spying on everyone. It looks like they were talking past each other<fnref n="178" /> and we absolutely do need that, I would say, conversation about what that does really look like in reality and what that look like for all of us.

<footnote n="173">173. Center for Democracy and Technology - https://cdt.org/</footnote>

<footnote n="174">174. https://www.gov.uk/government/collections/ict-strategy-resources</footnote>

<footnote n="175">175. Reform Government Surveillance.</footnote>

<footnote n="176">176. https://blogs.microsoft.com/datalaw/</footnote>

<footnote n="177">177. Probably http://www.assemblee-nationale.fr/13/rapports/r4326.asp</footnote>

<footnote n="178">178. Talking past each other is an English phrase meaning two or more people talking about different subjects, while they believe that they are talking about the same.</footnote>
[1:14'28 - Marc Mossé] If I may just add one point, because it's a very important aspect in the discussion. Some years ago, we had in France a bill on surveillance, some could remind this, and a that time the entire Tech industry, maybe for the first time ever, all the business association and IT association worked together to lodge an Amicus Briefing to the attention of the Conseil Constitutionnel (because it was seized, at that time) and one of important part of the brief from the industry was to say "OK, whatever you decide on this bill, the most important is to go to the principle in a way to build this bridge with the US" because at that time we had the decision on *Riley vs. California* in the US, we had this case on the way on the warrant I mentioned earlier and the idea in this memo from the industry was to say "OK whatever you decide, your decision should pave the way for such a way for a dialogue with the US and a dialogue across the world".

[1:15'57] I have to say, with the decision of the Conseil Constitutionnel, maybe we have not been totally followed but it was a first step thanks to the Quadrature du Net and others the Conseil Constitutionnel has been able to give more details on some points but it's an ongoing dialogue, for sure.

[1:16'19 - Maud Sacquet] Perhaps a last point, more generally, on the cooperation between the industry and NGOs, which is indeed a key one, as I had highlighted before. It's also fascinating, for me working and representing the industry in Brussels and working with NGOs. It's indeed perhaps a bit more done in the US than it is at the European level. And also there is, I would say, a friend/enemy relationship where we agree on some issues and we absolutely [emphasis] disagree on other issues. So it's often a question of "you are working with this NGO" - I say: "Yes, of course" - "But they hate you on that one" - I say "Yes, but we are also friends". So it's really weird sometimes, this kind of relationship that we can have, but at least on many issues we are able to cooperate, for example right now on content issue and copyright reform, which is, actually, crucial for fundamental rights, I could talk about it later I think, but it's about content removal.

[1:17'20] But on many issues such as surveillance, intermediary liability, content issue, etc., there are many areas where there is a lot of cooperation possible between NGO and industry and that's where, I think, we are the most powerful, heard and relevant. But it's not the case on all issues. Perhaps the main problem that we would have at the EU level compared to the US is that there are NGO in Brussels that are covering European issues but many NGOs are actually national and it's not easy for them to be, even by financial means, to be aware of everything that is happening in Brussels, to come in Brussels and be heard, so they often work with different groups, associations that will represent several NGOs but that limits perhaps the number of representatives that you can have in Brussels and perhaps they are less heard than they could be or should be on those issues. This is a concern, it's improving honestly, year after year, but that may be the main difference between the US and the EU policy on how this kind of cooperation is done.

[1:18:37 - Olivier Chopin] Maybe to focus on one of the points you have just addressed a few seconds ago, this question will be for Gail, Acadia and Mark first, and I'll have another one for you, Maud, in a few seconds. That's because it's about collaboration between companies. I am kind of struck by the kind of consensus you reach here on those issues: you are different companies and sometimes in a kind of competition on the market. So can you explain: do you interact with each other after Snowden's revelations? Did you come to talk to each other about how you should respond, what the proper attitude would be towards the revelations, or how do we push for new legislation in Europe, in the US? Is there some kind of coordination between the companies or not? We'll have a kind of reverse question for Maud in a few seconds.

[1:19:30 - Acadia Senese] Maybe I think the companies align on issues in a very similar way because we have very similar difficulties, for instance responding to government demands for users data from all over the world where there are conflicting laws, where, as Marc said, we are law-abiding companies, where we all respond to lawful demands or requests but we have to navigate conflicting laws to be able to do so in an accurate way. So many of the pinpoints that Google has, for example, are shared with Microsoft and Facebook.

[1:20'11] So we do align on many issues, even if it's not so much a planned collaboration or coordination but we land on the same spot. With Microsoft for example as Marc has mentioned, the litigation in the US involving the government's ability in the US to use a search warrant to compel the production of data that may be stored beyond the borders of the US. Unsurprisingly, Google had similar litigation after the Microsoft case proceeded to the Appeal Court and Microsoft there filed an Amicus Brief in one of Google's cases. So we do have, and we joined with Amici with Facebook on other matters as well, so there is collaboration, even if not so much coordination, but certainly, I think, from Google's perspective, there is great value in identifying as an industry some of the reforms that are needed, bringing a common voice to the table when we are drafting with government and NGOs so that we can move forward.

[the other speakers approve the idea]

[1:21'42 - Olivier Chopin] So, to Maud, as a policy officer based in Brussels and working for different companies, did you feel that at some point the Snowden revelations were a turning point in the way companies reacted or coordinated maybe more closely and even the relations between legal and tech guys within the companies working for solutions ...
Do you feel there was a collective reaction and maybe that's not the same in Europe and the US...

[1:22'20 - Maud Sacquet] That's an interesting question from a trade association perspective because by default all members, all companies we represent have to coordinate within the association to agree on common position that the association will put forward publicly. So there is always coordination and I can guarantee that the companies are clearly competing against each other because sometimes it's a nightmare to actually get a common position that you can go and advocate for.

[1:23'00] On surveillance, specifically, I would say the coordination is perhaps slightly easier than on other topics because of the principle, because of actually, there are not hundreds of ways to approach that issue compared to content, telecom policy... So it's perhaps a bit more clear in that area, it's easier to get coordination. It's also interesting to see coordination between trade associations and between the different sector' representatives you can have. Often time [?] even though you could also argue that tech associations compete against each other as well, for additional members and for leadership on certain issues. So we are definitively competitors but it's well recognized that we are often more powerful if we act together, so in many issues, including surveillance, you will see a lot of coordinated action, common joint letters, for example signed by the entire tech sector in Brussels advocating for reform and that we ask for coordination between trade associations and within companies to agree on that messaging, so it can be common event, it can be associations replacing each others at working groups meetings if some can't make it that day but we have to be there...

[1:24'28] So it's also a close-knit community, where we try to act as the best as we can and to have a unique voice on those issues.

[1:24'39 - Olivier Chopin] Thank you. Two last questions before I leave the floor to my colleague. After the Snowden disclosure of all the secret information coming from secret agencies in the US, people asked for a new notion of digital sovereignty in Europe. There was a push for that concept and a way to securitize personal data in Europe, legally and technically maybe stronger than it was or it is possibly in the US. How did your organizations react to that notion? Was that taken into account in the development of your technologies and services you provided? And the final question: are you worried that it could lead to some kind of new economic protectionism if we enter into the notion of digital sovereignies around the world? I know it's a complex question...

[1:25'58 - Gail Kent] I think that one thing Snowden certainly brought to light was the importance of protecting your data. It's something that is important to us anyway but the idea that any government can be trying to gain access to, I think it has been really focused by Snowden. One thing that is absolutely clear to us increasingly as there are a lot of debates around access to data around the world and you see numbers of different countries, from Russia through France, UK, South Africa, Uganda... All passing different types of surveillance laws that we cannot just rely on the rule of law to protect our data. So that has to be done through using the very best technology - and we haven't talked about encryption yet but it is a really key part of that because as we all said trust is exactly what we are about, and showing our users that they can absolutely trust us with their data. For us it does not matter where you are in the world, we should provide you with exactly the same service. It doesn't matter if you are in the EU, the US or subsaharan Africa, we should keep your data at the same level of security, but we should also enable you to access quickly and efficiently.

[1:27'32] So for us it's not about where the data is, it's about how we can protect and enable people to gain access to it. We also understand that for some people or companies, knowing that their data is in jurisdiction at a level of confer (?), that is not the way that the Facebook model works, we are about connecting people and making sure you've got access to data. So one of our concerns is that you do end up, through a misplaced surveillance policy, looking at data localization model which would undoubtedly have an impact on economic benefits. There are number of countries (Russia the most obvious one) that are really pushing for that at the moment. We've been mistaken if we think that's only being done for security reasons, I think there is an economic driver behind that as well.

[1:28'36 - Acadia Senese] I think that what we call data localization efforts are the pendulum swinging in the complete opposite direction as where we think it should land post Snowden revelations. Data localization arguments, such as if you keep the data in a country, in a specific jurisdiction for example, in the context of the Snowden revelations, is outside the reach of US surveillance. But that's actually a misguided understanding of how US surveillance laws work: to put data outside the US is in some ways losing the restrictions on US government's ability to get that data - or any other government's ability to get that data.

[1:29'22] So data localization is not the answer to the security of the data. That comes with tools like encryption, with users understanding how the data works and certainly for Google and other companies that offer the services it offers in a way that is fast and efficient and reliable is to do so in a way where the data is free flowing. That is why, and we'll talk about I think in the second half of the panel, most of the advocacy work we're doing in terms of legislative updates, the touchstone for the reform is not the location of the data but instead the location of the user. And that's the user location that should drive the protections in the law and should dictate which law applies to the handling of that data.
[1:30'16 - Marc Mossé] It's a very difficult question in fact. But I would say sovereignty means two things. It means a lot, but at least two things: first, it's the ability for the citizens to choose their government and [second] the role of the state is to protect its citizens. This is what sovereignty means, in more than a nutshell, in very few words. So if you go back to the topic of today, I would say the question is not about border. Sure, sovereignty is very important, we should not underestimate its importance. It resonates a lot to the citizens and even with technology, with internet, with the clouds, the Artificial Intelligence, sovereignty is still a cornerstone of the relationships between the states and between citizens. However, it doesn't mean protectionism. It means, or it goes back to the question "how do we protect the most efficiently the rights of the citizens? And as it has been said, the question is more how we can protect the data wherever they are rather than to force the companies to locate the data in every country. It's really important.

[1:31'45] If you look at Microsoft's data centers in Europe, in the Netherlands, in Ireland, in France, in Germany, in other countries at the same time we are totally in favor of free flow of data because it's part of how we can build a very dynamic and healthy ecosystem with the internet. So it's a balance between all these aspects but at the end of the day sovereignty should go back to one crucial point which how to protect the rights of the citizens. This is really the key aspect of this.

[1:32'30 Maud Sacquet] Digital sovereignty is, well, a very complicated concept. I think, again as a trade association, our mission statement is to advocate for open market, open networks and free competition. So, by default I would say protectionism or sovereignty is not something I would stand for. One example is we know that Europe wants to be a leader in privacy regulation and so impose its sovereignty around the globe on privacy. Again, I am not a specialist on that matter but I do not believe that there are many countries that have actually implemented similar privacy regulation as to what Europe has developed. So that is perhaps something the EU should think about.

[1:33'19] More specifically on data localization, that has been an important topic of debate in Brussels. The Commission published mid-September a proposal on free flow of data, which is the idea of free flow of non-personal data within the EU to try to fight against those calls of data localization. One example, which we use quite often is that in Germany actually, per law a company can't store accounting data outside Germany. What's the reason? Why can't you store them in a server in Denmark for example? That's the kind of trend that we see more and more in many countries and that this proposal tries to address so that we can have free flow of data, non personal data, that could then be organized in a way that makes sense and that is more efficient, a safer way that a company can offer to its customers. That's why I think we would not go for [in favor of] sovereignty.

[1:34'29 - Marc Mossé] I want to react to your point because beyond this discussion on the digital sovereignty which is, somehow, a physical question, which is not only the relationship between the private sector and the government, it's also the relationship between each of us - the citizens - and the technology, which is the very old debate on whether the technology will dominate the human. The real question of digital sovereignty is this one and it means that we absolutely need to give to each of us the capacity to control the technology. This is really crucial and having the government controlling the technology will not help on this question of digital sovereignty. The same with the companies of the IT sector. You made the point on education earlier this afternoon, and I think this is one of the key point. Education is really important because we are talking about a lot of questions: the rule of law, the US law, the European law... at the end of the day we have to be conscious that it's a question of education, how people will be able to control the technology, to understand what is at stake with the use of technology. If we don't go this way and invest in education, on how people can understand and manage the technology, we can do a lot on the legal side / framework but we'll miss one of the important point, which is really about the core of sovereignty, which is how people can control their environment.

[1:36'18 - Olivier Chopin]. Thank you. With that vertiginous perspective I will now leave the floor to Félix Tréguer, my colleague, who will act as a discussant for this panel and ask you some extra questions and come back to different dimensions you mentioned.

[1:36'40 - Félix Tréguer] Just a few points, I'll try not to be too long because we want to save room for the discussion. One is to thank you for this very transparent discussion, it's very interesting and were grateful to have all of these very expert insights on surveillance and controversies around privacy in the digital age. Although the topic of the discussion was a bit broader, because we talked about security challenges we focused so far at least on surveillance. My first question is actually to complement the various topics that we have addressed so far before going back to what you said.

[1:37'26] About the challenges in your relationship with governments, because I think that in the past years you've been navigating in ambivalent and contradictory environment or context in that respect because on the one hand you have this greater demand for privacy rights and transparency and very dense and tense legal debates and on the other we've seen terrorist attacks and security policies creating a lot a pressure on your companies to cooperate with law-enforcement agencies. So, what are the biggest changes in the way you communicate and interact with public authorities and in particular law enforcement authorities? I remember, a few weeks after the Charlie Hebdo attacks in
2015, French minister Bernard Cazeneuve, Interior Ministry at the time, went to the Silicon Valley and discussed with some of the executives of your companies and maybe you could talk a little bit about this process and how you interacted with policy-makers in that respect.

[1:38'44] Then, to go back to some of the things you said, it's interesting that you've mentioned – and we will talk about encryption in the next part of the discussion – but you talk about technological fixes and encryption in particular as the first reaction of your companies to the Snowden disclosures and Acadia said these efforts were already under way at Google at the time, that those disclosures only accelerated that process... I am still wondering: the web 2.0 concept was introduced in the mid 2000s, then it would become the cloud and the idea that Internet users could defer the protection and the handling of their data to online service providers and one of the marketing arguments in the second half of the 2000s was the fact that these companies were better suited than average users to protect the security of the data. My question was why did it take so long, until the Snowden disclosures, for relatively basic encryption measures to be deployed on your products and, maybe that's not the case but I would be very interested in having your intake on why did internet users had to wait until 2013 to have SSL encryption for instance on some very widely used online services.

[1:40'20] Then, one last point, or maybe two. One would be the ability of your companies to resist in engaging this debate on surveillance in a way that, maybe... resisting some of the proposals made by policy-makers and people coming from law-enforcement agencies and in this respect I think there is one interesting example and I just want to talk about it and maybe have your feedback and impression on it: it's the case of Yahoo! Because Yahoo! was one of the last... you know, there was this very famous slide about the PRISM program, where you have the dates at which different companies started engaging in the PRISM program, which is, basically, data request coming from Intelligence, the FBI and the NSA (or the FBI I think) to Internet companies. Yahoo! was said to have resisted the participation to the PRISM program and was even threatened apparently by the FBI to be fined 250,000 dollars a day if they did not comply. I was sort of puzzled by this and I was wondering whether the explanation for that resistance at the time was the fact that Yahoo! had suffered what was then probably the biggest scandal ever for a giant company, which was the Shi Tao request in China in 2005 where Yahoo! complied with an order from Chinese authorities to release data on a Chinese dissident and there was a big controversy in the US and there were public hearings in the US Congress where the leaders of Yahoo! were summoned by Congressmen to plead for forgiveness to the family of Shi Tao, they were called moral pygmies, something quite violent for the leaders of this company.

[1:42'23] So I was wondering whether that scandal maybe prompted Yahoo! to devise internal procedures that would allow them to maybe... something that would explain why their participation to the PRISM program was delayed for some years. But at the same time in the post-Snowden context we see Yahoo! [cooperating]... last year the press agency Reuters publish a report saying that Yahoo! had been served with an order to install a scanning device on US-based servers to scan the e-mails of Yahoo! users and apparently the chief technology officer of the company was not even aware of this order to which other executives complied, the board of the company apparently. So that also begs the question of the evolution in context and the fact that the company might be able and willing to resist to illegitimate practices or things that would fall out outside of the rule of law or such a definition of it and why in some other contexts a few years later they would not. So my question would be whether the Snowden scandal led to changes in the way, maybe within your companies, you might handle these... I don't know, something similar to whistleblower protection, internal to your companies? Are there things that were devised and that would allow the legal team and people working on privacy issues to talk more with engineers? Have there been some changes internally since 2013?

[1:44'20] The last question, also going back to the issue of differences between countries and the debate around data localization is the fact that we saw in Europe very harsh language coming from leading politicians against your companies. I remember in France in 2013 there was a first attempt at legalizing access to metadata and I remember a minister pushing very strongly against the criticisms of NGOs saying that users gave away their data to private companies and they shouldn't be worried about the state because at least there is a sort of democratic control over the state and really using Internet Service Providers as a scapegoat in a way, at least something that would justify the practice of the French state in terms of surveillance. At the same time I remember also that then president Obama talked about data protection in Europe as a form of economic protectionism, so what do you make of these discourses that sort of, on the one hand, you see European politicians being very critical of US companies and on the other, that's probably a separate question, about the way privacy is handled in Europe, and you underlined the fact that we make a lot about the differences and the fact that privacy is well protected in Europe whereas it is not in the US, and that might not be true. Do you think that sometimes data protection is used by European authorities as a way of escaping a debate on their own practices and also a way of protecting European companies with respects of some of your competitors on digital market?

[1:46'45 – Acadia Senese] I will start with your very first question and we'll go from there. I think you asked how do we navigate our relationship with the government post Snowden. I think I can illustrate that for you just by describing you how an average day in my life looks like because on one hand in any single given day I can be advising Google to disclose data to assist in emergency like terrorist attack and an hour later I can be drafting a brief to be filed in Court to
sue the government about being able to notify customers about the existence of a request. So I think we can be both advocating for a reform and pushing the envelope about how the law can be improved while at the same time being good stewards of the data recognizing real threats to the society and doing our part where the law allows it, to disclose for example data where to do so over an emergency. So it's not an incongruent thought to think that we can be both doing our part to protect but also doing our part to advocate for privacy and security. So I do - it's a very common day for me - on one hand I am fighting the government and I have very heated phone calls with the government official about a demand that they are making that Google thinks is unreasonable and inappropriate and on the other hand I might have another phone call from another government official who needs an information because there's an imminent threat in Paris for example. So that's very common for me and I think that dynamic is very good because companies can come to the floor and say "look what we are doing when there are threats". I think all of us have emergency disclosures processes that we staff 24 hours a day, every day of the week, every day of the year, where we are responding to emergencies around the world, no matter the country and on the other hand litigating and pushing for legislative reform. These two things actually pair very well together to say the companies are in a unique spot, where we have a vantage point where we see conflicting laws and are trying to navigate through those and have solutions or suggest a solution in a way that moves forward. So that's the first question, I forgot what the second one was but I'll appoint [?]

[1:49'25 – Marc Mossé] As you have just said, we have the process to answer for emergency calls and if I remember the Charlie Hebdo attack, we were asked by the French government to provide some information. We have been able to provide the data in 45 minutes while the terrorists were still active, and it has been done according to the US law in case of emergency when people could be killed or there is a risk for the life of people you can give access to data in certain circumstances we have be able to provide this data in 45 minutes so there is an efficient way to collaborate and at the same time to respect the due process of law.

[1:50'24] When the Bataclan attack started I was at home and my older son came to me and said "something is happening, I don't know what". I turned on the TV and I looked at the events and I called my colleagues, well, in fact at the highest level of the company and I said "something is happening in Paris, I don't know what, it could be extremely serious and we have to be ready to answer the questions that could come from the law enforcement department". I know that you have done exactly the same in your respective company. We have been able to provide information in no more than 90 minutes during this attack. So there is way to collaborate in respect of the law and it's important to remind it because, again, I do not see any opposition between public safety and privacy, and it's not new because of technology. We had to solve this question for years and if you go through the jurisprudence of the Conseil Constitutionnel or even the Conseil d'Etat, the Court has always tried to find a right balance between this fundamental right. There is no hierarchy in the fundamental rights framework, they are at the same level and we have to find the right balance between this. The question with the technology is that the principles are not new, the values are stake are not new but the way they can be impacted is new. We have to think collectively on how to find this balance in this new era and how we can modernize a law in ways that we can apply these principles in an efficient way.

[1:52'34] I remember the second question, you mentioned GDPR (General Data Protection Regulation): I don't know if we have a consensus here, maybe not on this very point, but I tend to think this is now one of the standards for the world and we have to agree on this. It's very important because, again, we have to consider this regulation on privacy as a starting point to build this bridge I mentioned earlier between our countries. What has been built through the GDPR is something that really puts privacy on the top of consideration and it's something that could help us to build this trust I mentioned at the beginning of my intervention. Even for the US it will be an important step that it could benefit from, in order to build something that could help a trustworthy computing era. But I don't remember the third question.

[1:53'52 – Gail Kent] The question I remember – you had about hundreds in there – was about how our relationship matured and changed with governments and how has our response to security matured. I don't think it has changed but both have matured. If you look at four years, at the age of the technology computing it's a huge amount of time. It's a third of Facebook life, so the importance of data protection has always been there but, as with every single technology company, we know that there have been developments that have enabled us... I think we're going to talk about encryption next, but I think we are always at the forefront of those technological and policy discussions so we have been putting even more emphasis on that, just as you have seen around the world, cybercriminals becoming better and better so we know that we have to do even more in order to protect ourselves. I don't think that was a result of Snowden, that was already in place, but it has undoubtedly changed and developed just as I don't think that our relationship with governments in it [has?] this sort of complexity that Marc and Acadia have talked about has necessarily changed, I think that it is just sort of managing that complexity on both sides has matured, so I think governments – if I put myself back of my law enforcement hat on – the relationship with law enforcement authorities is a very straightforward one, particularly in countries where the main landline providers had once been owned and run by the government, you know, it's a very straightforward relationship.

[1:55'55] If you look at the companies whose origin is in California that's a completely different mentality. One of the things that really hit me when I first entered it in 2013 was just how true it is to the idea of California being at the frontier and feeling so far away from government, and this belief that you can do anything, that is something that is just
so different from a civil service mentality. Therefore, by default there is a very different relationship anyway. If you had asked any of my colleagues in 2013 if there is any place in Facebook for terrorists, they would have given you exactly the same answer as I give you today, in that there is no place, as in any of our companies, there is zero tolerance. What has changed is that we need to be talking about this, to be overt about it, particularly after Snowden and to be very candid. There was a real concern talking about anything you were doing that was stopping something the government also wanted to stop would be misinterpreted, as a worker in the government it was I think naïve that we understand that we need to talk exactly about what we do to keep all sorts of bad actors of our platforms and also an understanding of the fact that we just talked about how we use technology to do that and being incredibly transparent in doing that.

[1:57'42] Other ways in which our relationship has developed with governments is through things as, you talked about the Bataclan, aside from providing data, how can we help? The Bataclan was the first time we turned on the Facebook safety check as the result of a man-made incident as opposed to natural disasters. I know that was something that the French government has talked us about before we turned it on, it was agreed it was a very positive thing to do and hopefully was helpful. The other thing that changed since Snowden in terms of relationships with the government is what we are all here talking about: the need to get some more systematic framework, that they can't just sort of be ad hoc, we need to be working on what does good look like, what's the framework for whether it's content, removal, whether it's about alerting governments about terrorist content or whether about providing data on an international level. That's another sure sign [?!] of the maturity of the companies.

[1:59'00 – Maud Sacquet] Perhaps I can take the last two questions I noted. One of the questions was: were you speaking hard towards this proposal from policymakers and you mentioned Yahoo quite often in that example. I was not involved in any of those issues at Yahoo! at the time when I was working there. Why was Yahoo! the last one to comply? With this issue, I have no idea, I think the fact that we knew now [?!] that Yahoo! challenged some amendments in 2007 in front of a Court might explain that delay and as far as I know all the proceedings are now available online, so that quite could be an idea. I know as well that following that situation in China, Yahoo! was the first company to implement a program that was a Human Rights program, so you had a department within the company where you would have called some people that were hired to review a lot of the policy decisions on data, on surveillance, some on advertising, depending on the exact scope, to think about all those issues from a Human Rights angle in a systematic way. That can perhaps explain some of the things you mentioned but that's mostly really guess from me as to why companies reacted differently. Yahoo was the oldest Internet company at the time, so perhaps went through a couple of issues first that the others experimented perhaps at a later time. That was the point I had on Yahoo!.

[2:00'40] How to resist proposal from policy-makers more on a broad perspective? For me, it's all the work you do as a lobbyist. As an industry representative, it's how you convey publicly your opposition to the proposals that are being made by the government and explain, as an industry, why this is relevant or less relevant, or why they go in the wrong direction or in a slightly better direction. I think that's the entire purpose of the work and you do that by talking with company to understand the product and the exact position and then you try to finalize and to mount campaigns and also simply persuade governments because most of the time people are not experts on the issue we work on every day and not aware of the some of the issues that are actually raising, and that's the virtue of just talking with everybody just to try to have everybody on the same level of understanding. Actually it's only how you try to do that.

[2:01'52] The last point that I remember you mentioning was about a kind of state control, company control and the way a European politician would criticize US companies to try to avoid facing some of the issues that they have to deal with nationally. This is where as a trade association I have a bit more freedom to give my opinion than a company on that one but this is one of the huge debate that we're having in Brussels right now. The European Commission has launched, about three years ago now, what we call the digital single market, so it's a broad area of reforms in every direction and tech issue we can think of. We've been bombarded with this proposal in every possible way. Let's just say very simply that the tech industry is extremely critical of most of this proposal. I think perhaps there is merely one or two I would think of which I could agree with, more or less. Instead of creating a digital single market, you are actually going backwards on many of those issues. Because of that, most of the narrative today in Brussels is the Europeans vs. the US companies, all of that is forged to try to trap a few American companies and it won't do anything to enable European companies to grow within the European market.

[2:03'19] Is that true or not, publicly all of them [Commission officials?] will say "no it's not the case, it's not protectionism, we are not going after US companies, it's for everybody", if it's true in practice, I would say that at least there is a part of truth in that narrative.

[2:03'35 – Olivier Chopin] Thank you very much to all of you. So we have some time for a few questions from the audience, so please raise your hands to ask your question.

[2:03'54 - Amish]. Thank you for this very good talk, my name is Amish. It's definitely true that all of you to some extent are in the same boat, navigating a quite difficult line between cooperation with the government and a duty to your audiences and towards the citizens. I want to ask you whether or not there's a kind of understanding in the field that
you're operating that a relationship with no cooperation may eventually lead to a relationship or coercion. And perhaps at some point, do you have an understanding that if you were to give ground and have a relationship of cooperation, it's in order to avoid or prevent more of a relationship of coercion? My main example, thinking about that kind of happening would be that earlier in the UK we had an attack outside of our Parliament and it was very big on the newspapers calling the killer the "WhatsApp killer", because of two-way encryption, meaning that it was for a society's security that perhaps it could lead to forceful move against WhatsApp. After this, as for a relationship with the government is sort of more established, that it becomes a sort of normalization with this kind of relationship that expands over time, particularly with the use of big data... Big data is so vast, unimaginably vast, that sometimes you don't really know what you are looking for in the data intel you have access to and it even leads to some critics of big data who worry about privatization of big data because you don't necessarily know what you are looking for and the government does not necessarily know what to ask for until they have access, until they can apply filters and can really dive into it. The final point I really wanted to ask is because of this sort of privatization of big data, that hasn't really been discussed today, until the same extent, is on the other side, the issue of trust with the consumers and it's a second front where I know that Uber has had a big PR problem with the relationship with the consumers and it becomes sort of creepy because you've had a privatization of big data with the rise of experimental rationalities with behavioral economics, which means that you've also got another front of trust between what the company itself does with this big data. Thank you.

[Comments by Didier Bigo about having all the questions, then the answers and finally a well-deserved pause]

[2:07'08 – unnamed person] I am just interested in an answer to the Yahoo! question from you three because it puzzles me also: how come that it was only Yahoo! who was in this legal debate in the US and the other big data actors weren't. So somehow Yahoo! was facing this problem and the others just were agreeing... Was there also this big demand for data for the other companies? The second question I have concerns a little bit the boundaries of data provision, so rule of law was somehow framed by Marc as a boundary where you provide data but now in France we see that with the state of exception some rule of law gets overcome, like there are problems with the rule of law in France at the moment I would say. How do a Tech company deal with it? I just take France as an example but you could name thousands of examples I guess in countries... Is there a Human Rights or moral guideline for you for providing data to governments?

[2:08'48 – unnamed person] Thank you. I have two specific questions. The first one is about the Privacy Shield. One of the consequences of the Snowden revelations was the invalidation of the Safe Harbor and then the adoption of the Privacy Shield. Currently this agreement is under review, well the adequacy decision is under review, I would like to have your position on the implementation of this Privacy Shield and for instance some comments on the functioning of the ombudsperson mechanism. The other question is about some current negotiation with regard to the access to e-evidence at the European level. We eluded it at some point, I would like to have your comments about the position of your respective companies with regard to access to e-evidence for public authorities in the context of criminal proceedings and how it can be compatible with your stance with regard to strong encryption, end-to-end encryption for instance. Thank you.

[2:09'55 – Olivier Chopin] Thank you very much. As this question will be addressed specifically in the second round table I think maybe you can answer to the first two questions.

[2:10'13 – Gail Kent] If I can draw a theme between, you mentioned your first question is about coercion, around the rule of law and also is relevant to the e-evidence part. Facebook certainly feels that there are cases in which we absolutely should be providing data. It's not that we believe that we should never provide data to government. If you are thinking about domestic violence cases, terrorist cases, rapes, murders... there are a lot of cases where governments, to Marc's point, are protecting the citizens and we're all safer because they're doing that. In terms of sort of what there is [??] we have a three-pronged approach to when we believe we should be answering those requests for data, and I know for being in various panels with Microsoft and Google that the same is true for them as well. The first is that we have to... whether it is a conflict of law - we haven't talked a lot about conflict of law today but it's relevant I think to the question about the rule of law: which laws do you apply? As a US company we have to comply with US law and often what you find is that there is a conflict of laws. So US does not allow us to provide content outside of US jurisdiction and that's one of the issues being debated in Congress at the moment. That means that if any other country is asking us for content – I'm talking about when we have the content, so not WhatsApp, where we do not have and never have the content – we cannot provide that unless the US legal processes follows up. It basically means mutual legal assistance treaty. But we do have basic subscriber information and in some cases additional metadata.

[2:12'13] So the three-pronged approach that we follow when it comes to when we should provide information is 1) we need to have the legal process of the country that is requesting, the jurisdiction cannot just ask us, they have to follow their own legal process, 2) that user has to be within the jurisdiction of that country and 3) the request has to be human rights compliant. So that means it cannot be a request pretending to be murder when it's actually for political activism. Speech issues are particularly problematic but we look into every single request to make sure that we are complying with all of that. And that's very much the debate in a lot more details than we've been having with the Commission on
e-evidence – as you said, we are going to talk more about that. But that’s our way in terms of trying to do with the conflicts of law.

[2:12'20] In terms of coercion, what we found so far... and we can talk about what's happened in the UK government, a particular case of interest of mine, where it comes down to trying to explain to the UK government exactly what data we do have and what we can comply and what we can't provide. This doesn't break end-to-end encryption, it doesn't require us to retain data that we have got no business need to. And not only explaining what that does look like in reality but explaining to them what we think the downside of them changing our business model, not just for what it causes to ourselves but for the overall ecosystem. I will talk at length about encryption but it's about trying to have that conversation so we never get to the point where they think they have to force us in Court to do something, which we think will severely undermine the security of our users but also the security of the ecosystem as a whole.

[2:14'45 - Olivier Chopin] Thank you so much. In the name of Didier and everyone here, a warm thank you for your candid answers and your dialogue, for talking about those tremendous issues today. Thank you very much.

[APPLAUSE]

Didier Bigo specifies that some face-to-face discussion can take place during this long break but after the coffee.

[2:46'20 - Alex Macleod, chair of the second roundtable): Just one thing before we start: I come from the perspective that Didier said we were not coming from, which is security studies and IR theory, which does not mean that I don't think that we now have to adapt because this is the great thing I think about this [...]? second roundtable, is bringing a stack to reality.

[2:47'11] One of the things I tried to do myself is teaching. I am teaching a course on the cinema and security, which go from Cocoon, to Fourteen Days to Snowden, which, believe me, is an interesting way of looking at it. So I will not say anymore about that, because you are here to listen to other people.

[2:47'34] This second part of the roundtable is about the legal debates. And as you know, there has been a lot of questions raised about the legality of the interventions of the various companies, of the way governments handle it, of the way legal systems handle these questions very differently, the fact that [these governments] have tried to pass through what we call Mutual Legal Assistance Treaties to try and bring some sort of order into the whole situation. So first of all, a question which I've been asked to pose which is on approaches to surveillance reform. As you know, in both the US and the UK, your companies have been involved in legislative reforms aimed at protecting privacy, especially the question of encryption, and other things we will come to in a few minutes. The first question is to Acadia about the changes in the Patriot Act in 2015 and whether you think it achieves better protection for the privacy of US persons. What are the perspectives for non-US persons, and in particular the FISA Act Section 702, on which are based the NSA's broadest capacities, such as the programme Upstream?

[2:49'06 - Acadia Senese] Just to clarify, which changes to the Patriot Act are you referring to?

[2:49'11 - Alex Macleod] I've got absolutely no idea. I have just been given these questions.

[2:49'14 - Acadia Senese] So whoever asked the question so we're clear on what we're discussing.

[2:49'18 - Alex Macleod] Anyone is clear on that? I am sorry, I didn't write the question.

[2:49'25 - Félix Tréguer] It was referring to the fact that the NSA and the FBI have deferred the retention of data to private companies.

[2:49'34 - Acadia Senese] Okay so the Patriot Act is up for renewal again this year...

[2:49'43 - Félix Tréguer] It's about the USA Freedom Act of June 2015 that we talked about earlier.

[2:49'50 - Acadia Senese] Let me try to get to a broad answer here. Part of the Patriot Act authorizes the US government through FISA requests - which are judicially-ordered requests, although by special courts in the United States - to authorize the US government to seize electronic communications of non-US citizens. As part of the collection, or as part of the production in response to FISA requests, the content of communications belonging to US citizens might be included in those productions. One of the things currently up for debate, in the US, is whether protections will be afforded to the contents of the communications belonging to US citizens. At the moment, a production made pursuant to a FISA [request] does not also require the government to obtain a US search warrant in order to also seize that data.
[2:50'54] So there is a debate in the US now ongoing about even if the government can seize content of communications - again judicially authorized by a special court - for a non US-citizen, can also as a corollary keep the content of communications that belongs to US citizens. That's now being debated, Google and others advocating for the need for a warrant for that other types of data as well. We've seen improvements in the law, that I talked about earlier, including the reporting of the number of requests that companies receive, both national security letters and FISA. So, there are some improvements, although I think it will be all up for debate what we see come with this Congress at the end of this year.

[2:51'44 - Alex Macleod] Thank you very much. The second question will be to Marc. It's about Silicon Valley companies working in France. They seem to be less vocal regarding surveillance laws adopted in France in 2015 to legal surveillance capacities that had previously been adopted with very little legal basis. Is it true or do you believe that it is more difficult for these Silicon Valley companies to intervene in France?

[2:52'19 - Marc Mossé] I could say that we are not in Silicon Valley. We are from the North of the West Coast. I'm sorry I will answer for you, from the Silicon Valley. On this, I can say for Microsoft, we have been involved in discussions with the government at the time. We have been heard in the National Assembly by the rapporteur. We have made our point. And as I mentioned earlier at the beginning of the afternoon, when the Bill was under scrutiny by the Conseil Constitutionnel (the kind of Supreme Court in France, although it is fact much more complex in France, where we have three supreme courts), the entire IT industry, and beyond the IT industry the business ecosystem, lodged an intervention to the Conseil Constitutionnel to make its point. Of course with the clear view that we need to fight terrorism - there is no discussion, it's a no-brainer, of course - at the same time it should be done in full compliance with [higher legal] principles. It was in a nutshell the sense of this memo. This memo is still available on the Web, so you can easily find it and read it. What was interesting in this memo if I may - it's another way to answer your question - is that as part of the introduction we had several paragraphs to say: "this topic should not be fixed only in France. This question requires a transnational cooperation, because the Internet is not a national question, it's an international question".

[2:54'35] And we, when I say we, the entire industry was calling for a set of principles that could be part of the constitutional heritage of our nation. I'm not fully sure that the Conseil Constitutionnel went on this path with its decision on the surveillance bill, but we have seen after on different cases brought to the court by La Quadrature du Net and other NGOs that the Conseil Constitutionnel, like the Conseil d'État, have made some moves opening the door to further evolutions. So to answer on this question, I would say yes we are developing our point to the government in a very transparent way. When we have to say something, we say it in a transparent way. And yes, the French industry - not the Silicon Valley in France but the French industry, including I would say companies such as Dassault System or others that are member of this association - brought this amicus curiae, this intervention, to raise some points. Again, not [against the idea] to fight terrorism - again we are totally aligned with this idea to fight terrorism - but to find the right balance between fundamental rights and how to work in a way [through which] you still protect public safety and you still protect privacy and fundamental rights. It's really important to keep in mind that we need a balance between both dimensions.

[2:56'21 - Alex Macleod] Thank you. This time it's for Gail, and it's about Great Britain and the Investigatory Powers Bill in the UK. What has been the attitude and the way of evaluating on the part of Facebook and other Internet companies over the debate on the Investigatory Powers Bill?

[2:56'45 - Gail Kent] I've said in the first panel: one of the things that was very positive about the Investigatory Powers Bill was, I think as the result of the Snowden revelations, there was an understanding that none of this could be done in secret anymore. And I think that in itself was huge step. I will bore you all a little bit about my own background: I joined, over twenty years ago, what was then the National Crime Intelligence Service, then Serious Organised Crime Agency and then the National Crime Agency. And all of these agencies had the sole UK non-terrorist responsibility for interceptions of communications. Which means that it was the only agency that can intercept telephones or other data on behalf of UK police forces for non-terrorist related incidents.

[2:57'46] One of the jobs I did was running the suite that carried out the interceptions. And this is relevant because I had people in that suite whose funerals I have been to - not related to working for me, they died of old age - who never told their partners what they did for a living. So they were involved in [countering] drug trafficking organizations, or people trafficking, or money laundering, or serious murders or corruptions cases. These are people who spent their lives protecting UK nationals from terrible crimes, and they never revealed what they did to their partners. So that amount of secrecy was embread to people and so we all know about these posters of the war with "loose lips costing lives": It was an incredibly secret environment. And I think that, to have gone from that - and for this I think Snowden definitely does get credit - to get to the point where we are now talking more is a fantastic step forward, and I think the Investigatory Powers Act really took that on front and center.

[2:59'09] It had three reviews. One by David Anderson QC - who was Independent Reviewer of Terrorism Legislation
at the time - and it's written in an incredibly witty tone. Anyone who is really interested in this area I would advise you to read it<fnref n="179" />, or at least its recommendations and its summary, because it goes into, probably better than anything else I've read, the different perspectives of the stakeholders. Because it's something it took very seriously, he was speaking to everyone he could to understand [...] the perspectives of the civil society community, of the intelligence community, of the police community, of Internet companies, of politicians in terms of what they were trying to achieve on surveillance reform.

[2:58'58] Secondly we had, the Royal United Service Institute had a panel of eminent - probably a very sort of French model in fact - public intellectuals, again looking and debating this: they had ethicists, they had philosophers, law professors on it, they had previous heads of two intelligence agencies and senior police officers, again to grapple with these issues.

[3:00'28] That was the second review and the third review was the own UK Members of Parliament and Members of the House of Lords [from the] intelligence and security committees. They debated this as well. All of those three reviews went into what the legislation should look like. It was a very public review. I think that sort of process in itself was a very positive one. Whether we ended in exactly the right place is probably a different matter. But certainly as part of that, Facebook and the other companies and all the different stakeholders were asked to contribute publicly during review, and were able to have as many conversations as we wanted with people who were reviewing. So from that perspective it was very positive.

[3:01'24] The other part that ended up in the right place is the part on understanding conflicts of law. Understanding the impact that one country's legislation has on other people. I think that in itself was a second positive step: understanding that we're a global community we have to work together. I think the one part where undoubtedly the UK government failed and I think it could have done a little better is that the legislation is not necessarily any more transparent - though on this I am not an expert - than the previous legislation was. And that is a disappointment. And David Anderson himself has said that what one thing he wanted to achieve was that an informed reader could make sense of the law. And I that it isn't the case. And I worry that ambiguity doesn't benefit anyone because it allows people on both sides to over-interpret legislation. So we haven't got to encryption yet but it means that there is often over-interpretation of the what the law currently allows in terms of removing encryption protections. I certainly see some exaggeration in the press coverage about what the UK law allows you to do, and again it gets interpreted during the road of "the UK is removing encryption, we can do the same".

[3:02'54] But equally, and Marc said this at the start, that we have to look at what do we think about surveillance legislation: If that is turned upon ourselves. I think that ambiguity can also mean that, are we 100% sure that the law enforcement officers and the judges, and the oversight commissioners who are interpreting that, are interpreting it in the way that it was written, and will they continue to do that in a five or ten years time? I know my previous colleagues and I believe they made the right decisions, but can I see that the law offers them that clear framework? I'm not 100% percent sure of it.

[3:03'38 - Alex Macleod] Thank you. And the question for Maud is about the EU Charter of Fundamental Rights. Would you agree with those that say that the surveillance activities of US-based law enforcement agencies is not always enough to respect the Charter? And the other question, which is closely connected, is: What is your view on the Privacy Shield as a way to protect those rights which we see are not so well protected when it's used by the US law enforcement?

[3:04'15 - Maud Sacquet] So on your first question, I've less worked on purely US legislation so my understanding is perhaps a bit more high-level. Is current US surveillance law respectful of the Charter of Fundamental Rights? I would have difficulty saying yes or no. What I know is that the adoption of the USA Freedom Act, while not perfect, and we recognized as an industry that it's not perfect, is still the first time that the US surveillance regime was reined in, at least somewhat, in the past few decades. And that is the first time that, we, we'll come for that, I would say, it's better oversight over new programs, it's not perfect, my understanding is that there are still some bulk collection of some foreign data, which we would of course like to see removed. We also think there could be better and potentially stronger oversight, but at least it's a first step in the right direction and this is something we should welcome and then continue to improve as much as possible.

[3:05'30] On your second question regarding the Privacy Shield, that's a very interesting issue: when the Privacy Shield [in fact she is probably referring to the Safe Harbor instead] was struck down by the European Court of Justice, actually stakeholders were currently negotiating to try to update the framework. What we ended up with was a text... first, we were happy that we ended up with something since it was important to have a framework to enable personal data transfer between Europe and the US. Without that it would have been a lot more complicated to operate. Again, it's not

<footnote n="179">179. https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/434399/IPR-Report-Web-Accessible1.pdf</footnote>
perfect but we still think there has been an improvement between the two versions. In this new version we have the annual review of the Privacy Shield, [while] it wasn't the case of the Safe Harbor, but now we can potentially tweak the agreement every year in light of modifications of laws or what's happening and return of experience as well or development of new systems so it's a lot easier to do tweaks and try to adapt the framework as best as possible.

[3:06'48] We have also the ombudsperson who is there to review European complaints, that is also why it was important that the US adopted the Judicial Redress Act which also gives some guarantees to European citizens that they could challenge US federal agencies based on the data that these agencies would have. So I think broadly speaking on your two questions I would not say that both texts are perfect, certainly not, but it's the first improvement perhaps that we have seen on that field for a long time and that's something to encourage and to continue to talk with stakeholders to improve it as much as we can.

[3:07'30 - Alex Macleod] Thank you. I would like to follow up the question I have just asked because this is a much more general question that I think is very important: we sort of touched the first part of the roundtable, and that's the question of different legal systems and how you can really compensate for the differences between legal systems, work between them, how for example, for Europeans you get around the first and the fourth Amendments which of course have all sorts of restrictions, especially in the Fourth Amendment for what counts as evidence. I mean, if you are getting evidence which would be considered to be illegal according to the US and law enforcement agencies and especially the courts, it obviously does create problems for the agencies in Europe. So I think that's a first question I'd like to ask you is: How do you see the possibility of reconciling these legal systems... it has already been done through the so-called Mutual Legal Assistance Treaties but I am sure there is more to eat than that.

[3:08'43] Another question that I'd like is to come back to what Marc said earlier about the question of sovereignty. I think it was a fairly short definition of sovereignty but we are in an age where people claim that Internet was supposed to be global, etc. but what do you do (and that's something I would like to ask Acadia in particular) when certain very important countries like China say "we will not accept access to this type of information", especially for example on questions of Human Rights in China or Taiwan independence, I think those are questions which now the Chinese would not allow their own citizens to access and not even, if I've understood rightly, get to them through VPNs (apparently you can also control that). So there's a huge question there that sovereignty is being exercised in the first question that is the legal system, because each legal system is very different (the French one if different from the British one and the British from the American) and then, second, of course, is what you do when certain states say "this sort of information is not going to be available for our citizens, period". So what do you say to those who say "Oh, but we have the global internet"?

[3:10'14 - Acadia Senese] I'll take the first question first and I'll talk about what's going on in terms of removals. In terms of resolving conflicts of laws, Google is an American company so we are compelled to comply with US law but we're also a global company and that means that to the extent that we're permitted to do things under US law we do so. That for example means that we comply directly with French law enforcement requests that are served directly on Google and because US law permits Google to respond to those requests directly and provide non-content information we voluntarily do so.

[3:11'00] In parallel, we are working with other companies, NGOs, the governments to advocate for MLAT reform, and part and parcel of MLAT reform, that is reforming the system by which countries formally engage with one another. That is France, for example, formally engaging the United States to make a request on behalf of the country of France to US company. That system is broken in terms that it can't handle the volume of the requests that are coming through it. It is a manual process requiring human review and processing that we can't, as we say in the Valley, scale to a place that it needs to scale to.

[3:11'48] So in parallel we are working to update the Stored Communications Act, which is the US law that prohibits the disclosure of content to France, for example, and so all of these things need to happen together. Congress needs to update the Stored Communication Act to allow the disclosure of content to a non-US governmental entity, in parallel we need a framework that exist [?] either bilateral agreements between, say, US and France, that would allow France to come directly to the US [companies] or multilateral agreements which would contemplate multi-state membership, so long as those states meet basic Human Rights standards, that there is a due process of law and that there are privacy protections. Those multilateral frameworks based on the location of the user would then facilitate and enable this cross border data request in a way that's clear (there's a clear legal framework for it) and can meet the demand for those requests.

[3:13'00] About China, I am not a content-removal expert, I don't pretend to be one, so to the extent to which we are talking about censorship of content based on the country where a person resides, that is not my area of expertise. Companies handle disagreements with content censorship in various ways: one of the ways that it can be handled is by not doing business or offering certain services in certain countries because companies aren't willing to assume the obligations that come with offering a service in a particular jurisdiction. Another way that IT companies handle this is
by removing content and I think some others will probably speak to that as well but it's varies by jurisdiction, it varies
by companies it varies by what the business needs are and it varies by the demands are being asked by the posting
jurisdiction.

[3:14'05 - Marc Mossé] Maybe I will talk a little bit about the cooperation between the governments and the conflict of law. I would love to spend a few minutes, maybe a few seconds, on the New York warrant case, which is a case I mentioned during the first roundtable. Just to explain the situation: it was a warrant issued under the SCA (Stored Communication Act) requesting that we provide content data from one of our customers, stored in our datacenters in Ireland. We declined to do so. Of course we had a strong disagreement with the US government about this, we didn't have the same reading of the SCA and we went to the Court, the second Circuit Court of Appeal of New York and in 2016, July 14th, we won this case. It was a little bit surprising for some but we won this case with two questions, and these questions are very important, because it was about the conflict of law but, beyond it, it was about privacy and sovereignty, the two questions at stake.

[3:15'43] After the decision of the Court of Appeal of New York, the government of the United States had the opportunity to stop, to maybe try to modify the legislation (the SCA) or to go to the Supreme Court. They have made the choice to go to the Supreme Court. And the Supreme Court, as you may know, can decide to take or not the case. And the Supreme Court will decide tomorrow or next Tuesday, October 16th, to take or not this case. If the Supreme Court decides to take the case [...], we will have this debate at the high level of the juridical system in the US and it's really a case that opens the debate on these different aspects.

[3:16'45] Again, on this case we are not against the investigation, it was not our position. It was not terrorism but it was about narcotraffic. We didn't [...] on the investigation. The question was really about the principle. The data were stored in Ireland, it means in Europe, and it was about privacy. If the Supreme Court takes the case, it will be an opportunity, not only for us to defend our position, but also for some maybe to intervene and to give their opinion. We mentioned the possibility for NGOs to be active on this kind of topics. As I mentioned earlier during the procedure before the Court of Appeal we have seen some NGOs lodging, filing an Amicus Brief, maybe some will do the same at the Supreme Court level. We could see as well other companies, of course, but we can see governments making their point which is not supporting Microsoft or not, which is really mainly about the principle.

[3:18'03] It would be interesting because we often talk about the question of building a bridge between the US and Europe, building an international framework, revising, modernizing the law in digital era and here we really have an opportunity, through this multi circular approach, to push this point of view. What should be modern law in digital era to both protect public safety and privacy? I don't know what the government will do, I don't know what the French government will do on this but they have the opportunity to make their point and to lodge an Amicus Brief. If I go a little bit further, beyond this New York warrant case, we have at the same moment the discussion in Europe (it was the last question just before the break): the e-evidence discussion.

[3:19'00] As you know, or if you don't know I will tell you, there is a discussion at the European level to build a sort of new framework still to access to e-evidence in a way which is compatible with both: public safety protection and privacy. Even between European governments it's not so easy. So the Commission is working on a regulation that could be issued on the coming months or year but it's extremely important because it will create this framework on having a possibility to access to data when an investigation requires it and protecting the rights of the citizens in a modern era.

[3:19'53] If I make a link between the two stories, the New York warrant case and the e-evidence regulation under discussion, you can see that there is some opportunity to build this bridge I mentioned earlier. Because if the Supreme Court takes the case and decides that the Court of Appeal was right, which means supporting the Microsoft position, it would mean that the US government will have to either modify its law or enter in discussion with other governments (why not the EU) to frame this international legal system. It would be even easier if at the same time the EU has a new framework, the e-evidence framework for cooperation between member states, one system, and this system could help to build this bridge with the US.

[3:21'05] I am drawing a sort of ideal scenario, of course, but it's really about what we have discussed during the first round table: how could we build this new legislation in the digital era? It's extremely concrete. And this is a way to not be trapped in this discussion opposing public safety and privacy. We don't have to oppose both. We have to work in a way that we find a right balance between two constitutional rights, two important principles, that citizens care about. And here, with the New York warrant case, if the Supreme Court decides to take it, and with the e-evidence discussion which is under way in Europe, we have two pieces of a puzzle that could help us to build this bridge, because, again, as I think, there is a worldwide constitutional heritage with the same values that we could really integrate in something that could build the future of internet.

[3:22'24 Gail Kent] So you started by asking how do you make these different laws match up and I do think we should not underestimate the difficulty of that. Surveillance laws are some of the oldest in the world. Most of them in Europe
were drafted in the 80's or 90's. Last week in India I was looking at their laws: 1885, they drafted the Telegraph Act. On top of this there is the fact that the laws are so old and need to be reformed for a digital era. You also have, you can put it on top of that, not just what the law says, because what I definitely learnt in this job is reading a law, even if you understand the language it does not actually tell you that much, you need to understand the culture that comes with it, the history of whether the law is even made for implementation or not because there are a lot of countries around the world where to pass a law is the same as having done something because there is nothing like being policy-maker, politician and proving that you've done something by passing a law, because you've done something but whether that ever get implemented can often be a different question altogether.

[3:23'44] I think all of that put together means that there is a layering of secrecy on top of it, it means that these are not laws that sort of naturally fit together and they were never meant to, because they are about national security so they are not part of harmonization of EU member states or anything that should be generally covered by any UN Convention in any sort of details, so it's a really really difficult area. Having said that, I completely agree with you that we are now at a critical point where we have to do something because the pressure on companies is great, but more importantly, the voice that we don't often think about, though we often think from a users and Human Rights perspective, we do not often think about the victims of crime and what it means when you try to do an investigation and you are not getting data that might help prove that somebody is innocent or might help get resolution for a victim of crime.

[3:24'55] All of this means that we absolutely have to do something and as Marc said there is the e-evidence project, it could for the first time in Europe come up with a framework that helps provide a structure for what good surveillance legislation looks like within member states. There is also the pressure for US reform, there is the Microsoft case and there is the "Internet and Jurisdiction" project, which is right here in France, which is looking at that from a much more global scale. But all of that will require countries to think about their legislations as well, because the two parts I see in terms of resolving this issue is building the bridge between how does France request information from the US - that's the agreement part of it - but it's also making sure that French legislation can be explicitly recognized as good legislation by the US. We see this in the US reform debate at the moment - if you are really interested in this it's worth watching the Congressional discussions around access for other countries to data that's held by US companies so it can give you a sense of how strongly Congress feels the responsibility, partly because of things such as the Yahoo case, to make sure that data isn't being given inappropriately to other countries. But US Congress certainly takes the responsibility of not just thinking that data could be handed over to anyone.

[3:26'40] So what does good surveillance legislation look like? I think that's going to be one of the very difficult bits of the e-evidence project is making sure that explicitly every member state have clear human rights safeguards. Every member state will argue that they already implicitly have a [good] legislation because of signing up to the European Convention of Human Rights but is that enough if it's not going to be explicit in here? I do not think it's going to be easy. Having said all of that, I am really surprised how far we've come in four years. When I first started research in this, nobody was talking about it and there was what was called the MLAT problem and we just had to assume that it would be there forever.

[3:27'31] Now I feel that we've got reform in front of Congress, you've got a case that will possibly go to the Supreme Court which would mean that we would get an extra level of attention, the Commission is taking this very seriously, not just for the sake of us but because it believes it's a key part to solving part of the terrorism issues, and you also have countries like India and Brazil really pushing for this as well. So I think that if there's a time that we can manage to put all those different legislations together, that's now.

[3:28'09 - Maud Sacquet] There is not much I can add because I think everything has been perfectly covered. perhaps still a couple of points. First with one practical example. I remember being on the phone with a French police officer and explaining to him that I couldn't hand him any of the data because it was data this was data only available to the US companies, so he had to go through the MLAT process to be able to receive it. The answer I got was "OK, go to the US, no chance, case closed, request in the bin" I put on the phone. It's his decision obviously but I wouldn't have liked, as a private citizen, to see may case handled that way. I think that was the time I realized that what we did [indeed?] before was clearly necessary, especially for the MLAT process.

[3:29'08] From a policy perspective, again, nothing I can really add. What we are focused on as an institution both in Washington and Brussels is, indeed, reform of those processes. My colleagues in Washington are working on the International Communication Privacy Act where we think it still can be refined but it's an interesting area of progress in those issues. There is also discussion around bilateral agreements between the US and [some big?] countries such as the UK, again it's an interesting way to look at it but there's always questions about which country, which crimes, which data, how, etc.? It's an extremely complex issue to settle and so we are working on those issues.

[3:30'02] From a Brussels perspective there are two things that can be done by the industry broadly speaking: first, communicating with the European stakeholders, explaining to them what's happening in Washington, what is actually moving there and asking them to support the different efforts by the US government to try to improve the situation
because of more feedback, the most constructive discussion that we can have between Brussels and Washington on these issues is important, as the role as an industry that we can play. Then there is the own European process on e-evidence, where again we can contribute to working groups and trying to create that framework which is indeed necessary. We have to think of cooperation regarding data request between two European countries, a bit the same way we would think of it between France and the US: the German police authorities would have to go through the French authorities before the French authorities can hand over data requests to the company based in France. So then you as a company you have questions and you're trying to clarify the scope of the request or the scope of the investigation, then you call the French police officer who has absolutely no idea what that request is really about, he has to go to the German colleague to try to figure that out, with the language issue you can just imagine what's this look like. So even within Europe basically there is progress to be made and that's why the e-evidence program is so interesting right now and we welcome it. While we can see some issues, my understanding of the e-evidence framework right now is that the Commission is looking at many different options as to what exactly could be done and that the proposal should come out normally early next year at some point but that's really under discussion. So that might change.

[3:32'05 - Marc Mossé] Very quickly, because I do not want to be seen too optimistic or jumping into wishful thinking, but I was reading a quote by Justice Breyer, a member of the Supreme Court, francophile and francophone. In his book "The Court and the world" that has been issued in 2015¹⁸⁰ - a very interesting book I recommend you to read, in addition to "Le Palais de Rêves" [mentioned earlier] - he says something which is very interesting: our world is increasingly interdependent and questions about overseas application of American law and how our Constitution should apply in the modern age cannot be avoided.

[3:32'58] Again, we are at a moment when we are crossing the path and sometimes there is some momentum like this where people can mobilize to try to draw the new framework. Honestly, not being too much optimistic here, I am seeing a lot of signs that maybe we'll go in a positive way for the fundamental rights.

[3:33'35 - Alex Macleod] This ought to be the last question but I think this is a very important one that has already been raised, which is the encryption question. It was raised of course by the San Bernardino attack and involved a company that is not present [today], Apple. It raised an extremely important question about how far states can go to demand that encryption should be handed over so the police could at least have access to what they consider to be possible vital information. Now I think all the companies and organizations involved on this panel have had at least something to say about it, so I would turn to you and say: how do you see it now?

[3:34'25 - Acadia Senese] So, the Apple-FBI unlock debate touched all our companies, it certainly touched Google. In that particular case it was framed as an encryption issue but actually it was really a file about really antiquated law. In the Apple-FBI case the government tried to use a law that was passed in the 1800's, called the All Writs Act and they tried to use that law which was basically a law that authorizes a court to issue an order if necessary to effectuate an already issued order by the court. So in this case the concept was that the court had issued a search warrant, and ancillary to the search warrant the court then issued an All Writs Act order to Apple requiring Apple to do something that it wouldn't otherwise do in the normal course of business. In the case of the San Bernardino matter, what was at issue there was that the order compelled Apple to write additional code that didn't already exist so that the government could gain access to the device.

[3:35'52] So the debate was not so much about encryption and what was regulated / not regulated did apply here, but rather could the government compel a company or third party to do something that it currently didn't have the capability to do and did not otherwise have a business need to do, when it otherwise wasn't connected with the crime at issue. So that was the debate there. Google brought in an Amicus there saying "no, you could not use the All Writs Act in that way" and the debate fizzled out because the government otherwise gained access to the phone, through other means, and didn't pursue the case any further in court and so with respect to providing access to phones that conversation has largely gotten away in the United States, and may be surfacing again, but there hasn't otherwise been a challenge to the All Writs Act.

[3:36'44] So with all of that sort of backdrop to Google's position with respect to encryption, which is "yes, companies should be able to encrypt their data and to give users the ability to encrypt their data and that should be a choice, that encryption should be there when it makes good business sense to do so, when it makes sense for the service, when it makes sense for the user. But otherwise it shouldn't be regulated, no. Of course, recognizing that there is no panacea for the real true legitimate concerns and needs of the government to be able to investigate crimes. But also, on the other end, the need to balance the true need for privacy and security in the data at issue. So I leave it there, I know others are trying to get at the end [?] but that was the framing of the San Bernardino debate.

[3:37'35 - Alex Macleod] I just want to be fair and let Maud intervene without having her thunder stolen by the others.

<footnote n="180">180. https://www.nytimes.com/2015/09/20/books/review/stephen-breyers-the-court-and-the-world.html</footnote>
[3:37'42 - Maud Sacquet] That's a shame because that's actually the question I know the less about, so this time I cannot really say "Oh, I have nothing else to say". My Washington colleagues at the time that were on the San Bernardino case actually were quite vocal about [it]. As industry we think that end-to-end encryption is key but also was somehow the reaction of some of the privacy concerns that a lot of our users had. I think in that debate - of course the main debate is terrorism vs. privacy, most of the cases are terrorism, that comes first - it's actually some debate that was already existing in the 1990s (in my understanding) in the US as well when we started to think about encryption and third-party keys and all of those things. The debate was already there and the arguments from law enforcement and industry were actually the same already at the time.

[3:38'52] As industry position, it's very simple: we think encryption is good, we do not think that backdoors should be implemented... I mean, perhaps it could be a solution in some cases but if you look at all the damages that it would cause, we do not think, if you have to balance security and privacy, this is the concern that should win. Very simple because, if any criminal knows that these backdoors were implemented, they would simply use other services, less well-known services where companies potentially would not be as law-compliant as the main communication services that we use today. So we think it could actually damage law-enforcement activities there. Then, if you think broadly about trust, that we were talking about at the beginning, we are actually starting to regain trust, we have done a lot of efforts to maintain the trust of our users in tech services, and having, especially the US government implementing those backdoors, then all that work, all the transparency work that has been done would simply again fall apart.

[3:40'04] It could also have a chilling effect on the speech of dissidents for example, again using a lot of our services and knowing that encryption can protect them, especially in countries where the current regime is perhaps not as democratic as it should be. That's also a big concern, as industry, that we want to raise. In short, we know there are concerns but still think that in most cases privacy concern should perhaps take the lead, that law enforcement should often look at the many of the other options they have to actually solve the case. In many many cases, those cases were solved without really having potential access to that data, so that's something to take into consideration.

[3:40'57] Broadly speaking then in Europe the encryption debate has been raised quite a lot, more recently I think by letters from the French and German governments, all of this this summer if I am not wrong, saying that after the French and German elections the issue of encryption should be tackled at the European level, while at the same time we have a Commissioner, Ansip<fnref n="181" />, in Brussels, who has said very clearly that he is for encryption and doesn't want to see any backdoors implemented there. So that's the kind of political fight that we are starting to see at the Brussels level and that we follow very carefully and closely.

[3:41'39] To finish, my understanding as well is that there should be a Communication on encryption<fnref n="182" /> at the end of the year...

[3:41'45 - Gail Kent] ... Maybe next week

[3:42'47 - Maud Sacquet] next week, perfect... We will be looking at it very closely when it's released.

[3:41'57 - Gail Kent] So obviously, as WhatsApp is one of our companies and WhatsApp is end-to-end encrypted, this is a debate that we have thought a lot about. It's also interesting coming from the law enforcement perspective. One of the frustrations - that had been my career for such a long time - I have when looking at this debate around the world is you very much see the same pattern: you see cybersecurity experts and people that are looking at data protection consistently emphasizing the importance of encryption and of increasing and getting better encryption particularly when we look at, I mean every week there is a different name that you can throw at it: Equifax (the breach of the moment)... We are witness to an increasing number of breaches to huge numbers of data. And we all know that encryption is key to protect access to data.

[3:43'00] You see cybersecurity experts and policy-makers emphasizing this, as you said, like Commissioner Ansip and then you will see, usually in the back of a terrorist attack, and understandably caused by frustration for not being able to gain access to a piece of data, whether that was the FBI not being able to get into the San Bernardino attacker's iPhone, you see a request by policy-makers coming from the law enforcement side for access to data. I think my biggest frustration is that it's very very rare for that debate to be looked at holistically and for encryption to be looked at [all around?].

[3:43'40] It is fortunate there is a debate and there'll be a Communication next week by the Commission, it's going to be as part of the terrorism package, which I think is totally unfortunate, and it's not part of the cyber security package although it was mentioned in the cyber security package but again it was put into one rather than the other. I think they

<footnote n="181">181. https://ec.europa.eu/commission/commissioners/2014-2019/ansip_en</footnote>

<footnote n="182">182. https://ec.europa.eu/home-affairs/sites/homeaffairs/files/what-we-do/policies/european-agenda-security/20171018_eleventh_progress_report_towards_an-effective_and_genuine_security_union_en.pdf</footnote>
will probably acknowledge that as part of the debate.

[3:44'03] And aside from the frustration for not looking at it holistically I think there is a frustration about not looking at this in a long-term way. I had the great pleasure of listening to one of your colleagues, Brian Lamacchia<fnref n="183" />, one of the great Microsoft cryptographers, talking about this and I also spent a lot of time with Matthew Green, who's a fantastic cryptographer in the States. And listening to them talking about how cryptography has developed, we are very much at the Wright brothers stage of developing technology. We are just about off the ground and we are doing this in a very precarious way and if anyone gets any doubts about that, the FBI helped by somebody else, managed to get into the iPhone. It wasn't that it was an entirely impossible thing of doing that.

[3:45'05] What Wright brothers technology means is that if we start tinkering with that now, if we do anything to reduce the effectiveness of encryption, we'll be basically setting ourselves back and we wouldn't do that with cures for cancer anything else so why are we doing it with cyber security? The Wright brothers technology analogy also works when it comes to the solutions that are presented to us. And, believe me, WhatsApp gets presented with a lot of solutions to the end-to end encryption problem. The same problem that you have is that if you are trying to scale up some of these theoretical solutions. That's again like trying to build a jumbo jet using Wright brothers technology for their first flying machine. We really need to be understanding that when we look at what we're asking companies to do in terms of changing their technology. We really are stopping ourselves from getting the very best security and technology that we could possibly have and there are lots of different examples of when we've done that in the past, whether it's the MD5 hash or whether it's the controls that President Clinton put in place in terms of strong encryption that have had impacts on security as we know it now.

[3:46'27] The fact that there is end-to-end encryption doesn't mean that companies shouldn't be responsible and shouldn't be part of the discussion on what does being a responsible company if you provide end-to-end encryption mean. I think that again it comes back to the e-evidence debate: what data is that we should be providing, under what circumstances, so we're not saying to law enforcement "you can never have anything at any point in time". We are being very transparent about what can be provided, and what can be used for, and what circumstances they were ever breaking our end-to-end cryptography. And I can guarantee that WhatsApp would never be breaking end-to-end encryption, because it is so fundamental to the company, also we believe in the future of a safe Internet ecosystem.

[3:47'39 - Marc Mossé] A lot of things have been said so... first of all, we should keep in mind - it has been said, but... when weakening the encryption we may undermine both privacy and public safety. We have to keep this in mind, especially in a time when cybersecurity is on the top of the agenda, so we really have to take care of a comprehensive approach of this question. I would add another point which is often when government, not only government but agencies and others, are talking about encryption to me it's not always clear what they are talking about...

[3:48'27 - Gail Kent] I do not think it's clear to them either, Marc...

[3:47'39 - Marc Mossé] Maybe it's your assumption here, but the fact is do they speak about stored data? Do they speak about data in real-time transit? It's not exactly the same and it will not be solved through the same rule or process or technical means. We are in a time, again we perfectly understand the need to fight against terrorism, there is no discussion about this, but the question is: What are the keys, what are the scenarios to address and what are the concrete proposals?

[3:49'04] Once we have this on the table, the discussion could take place. Until that, it's a theoretical discussion and a discussion where we cannot find any real answer because we are not necessarily talking about the same things and the same problem. So I'm not getting back from the question, I am just saying that at that time we absolutely need to take into account all the aspects of this question which is, again, public safety, privacy, lawful access, cybersecurity as well and we need to approach this question through a comprehensive approach, yet again, and try to design a framework which is the one that we need: an international framework.

[3:50'00 - Alex Macleod] Thank you very much. This is the end of this part of the roundtable. Finally we'll have a discussant, who is Joris von Hoboken.

[3:50'10 - Joris von Hoboken] Thank you for having me here, I'll give a few comments and I'll go to the Q&A with the audience. I am an Internet and fundamental rights scholar, I did some work on transnational surveillance, cloud surveillance issues, encryption, human rights and intermediate liability. I wanted to just mention, there was this issue around how was your Snowden day at the beginning, and I had a pretty funny one because I was at a privacy conference, the Privacy Law Scholars Conference which took place in Berkeley that year and basically after imagine "la creme de la creme" of privacy law people coming all together, including Jennifer Grannick and people like that but also government officials, quite senior people at the government and then this panic broke out and everybody was together in

<footnote n="183">183. https://www.microsoft.com/en-us/research/people/bal/</footnote>