### 4.2.2 Information flow control

Information flow control consists in monitoring how a piece of information flows through a system or a program. The goal is to ensure either statically or dynamically that a piece of private information is manipulated according to its security property. A typical property is that a private information does not leak towards a public channel.

#### DYNAMIC INFORMATION FLOW TRACKING

Dynamic Information Flow Tracking (DIFT) is a popular and versatile technique for monitoring the flow of information through a system. Typically, data entering the system is tagged according to its security levels and tags are propagated through the system. This technique can be used to monitor a system and ensure that the flow of information abides the security policy. A basic property is that untrusted user-provided data is sanitized before being processed. DIFT is also a powerful tool for penetration testing and vulnerability analysis: a bug, e.g., a buffer overrun, may be considered a security vulnerability if the data is influenced by an untrusted input.

DIFT analysis can be performed at various levels of granularity. At the system level, the challenge is to instrument the operating system with security tags and get a solution that is correct, precise, and requires some limited maintenance. To improve the scalability of DIFT, several hardware solutions have been proposed. For instance, the CrashSafe project proposes a novel processor design. There are also less intrusive proposals where an independent co-processor is dedicated to managing security tags.

DIFT is often limited to direct information flows that can be enforced by monitoring one execution at a time. Remarkably, DIFT can also enforce hyper-properties, such as non-interference, which are trace set properties. As this monitoring stops the execution in case of violation, they turn an information leak into a termination leak that may be acceptable for some applications.

#### STATIC INFORMATION FLOW TRACKING

There is a vast literature of type-systems for information flow. Sophisticated type-systems handle rich language features such as exceptions and method dispatch. The JIF system for Java is probably the most impressive implementation of static information flow control. The traditional property that is enforced by type-systems is non-interference, which basically states that public data does not depend on private data. To accommodate programs that leak a controlled amount of information, e.g., cryptographic primitives, practical type-systems need to include a notion of *declassification*. Declassification may take on various forms but requires some kind of user-provided specification that is trusted by the system. In general, it is hard to predict whether the long-term effect of a declassification corresponds to the user-intent.
## QUANTITATIVE INFORMATION FLOW

An alternative to declassification is Quantitative Information Flow (QIF) where the goal is to quantify either statically or dynamically the amount of information that is leaked by a system, through some observation by the attacker.

To prevent QIF, typical methods of security such as encryption and access control are not applicable: the only way is to obfuscate the link between the secret and the observable. Ideally, we would like systems to be completely secure, but in practice this goal is often impossible to achieve. Therefore, it is important to have a quantitative theory of leakage, so as to measure the vulnerability of a secret, assess whether a system is better than another, or evaluate the effectiveness of a method to prevent leakage. The quantitative aspects stem from the fact that the knowledge of the adversary is typically of a probabilistic nature and that the best methods to prevent leakage are often randomized methods. The most successful approaches to the foundations of quantitative information flow are based on information theory and on the notion of entropy (in various versions: Shannon, Rényi, or guessing). The entropy measures the vulnerability of the secret and the choice between various notions reflects the different operational adversary models that one is interested in.

A first drawback of QIF is that different definitions of leakage offer different, sometimes incomparable, security guarantees. QIF approach captures a large variety of attacks (approximate guess, multiple guesses, or properties of the secret) and subsumes most of the information-theoretic approaches considered in the literature and their corresponding notions of entropy.

A second problem with QIF is that average probabilistic guarantees may not provide an adequate security guarantee in case of an active attacker, i.e., when the attacker controls some of the input data and tries to use it to trigger security vulnerabilities.

## COMPILERS FOR INFORMATION FLOW ENFORCEMENT

Another research trend aims at preserving high-level information flow properties through the compiler chain. The flow of information matters for multitier languages where the compiler has the responsibility to decide which tier stores sensitive information and how to ensure the security of the communications. Information flow is also a stringent matter for critical code, e.g., cryptographic primitives, where any leakage due to the implementation may break the mathematical security guarantee. In this context, the compiler aims at protecting against side-channel attacks, such as timing-attacks or power-analyses.
[Inria teams] Access control and flow control

一是 The **CELTIQUE** team develops certified compiler techniques and analyses for protecting against timing-attacks. Together with the **INDES** team, **CELTIQUE** develops a theory for hybrid monitors that augments a dynamic information flow monitor with a (dynamic) static analysis of non-executed branches computing a symbolic and quantitative form of information flow. Together with the **MARELLE** team, **CELTIQUE** works at providing compiler support for *constant time* programming, a strict programming discipline adopted by cryptographers to limit timing leaks.

二是 The **CIDRE** team has a strong expertise in Dynamic Information Flow monitoring both at the system and hardware levels: the team develops the blare tool, an intrusion detection system (IDS) allowing, on Linux and Android, to dynamically evaluate the legality of information flows. A hardware device has been designed to improve the precision of this evaluation.

三是 The **COMETE** team proposed the theory for grounding security on notions of quantitative information flow and developed the g-leakage framework. They also developed the library Libqif¹, a C++ toolkit implementing a variety of techniques related to g-leakage, quantitative information flow and differential privacy. This team is also investigating an approach based on Game Theory to limit the leakage of information in the presence of an active attacker.

四是 The **FUN** team investigates a way to avoid transmitting all the data collected by the Things to a server. The team uses decentralized storage to avoid reconstructing all data at once. In addition, it identifies malicious relay nodes to bypass them during the data collection process.

五是 The **INDES** team works at defining, comparing, and evaluating the limitations of different information flow policies -including declassification and computational noninterference for cryptography- at the language level. Applications domains mainly include JavaScript, reactive, and multitier languages for IoT. The team also develops static, dynamic, and hybrid sound enforcement mechanisms for information flow security and works on preservation by compilation of information flow guarantees.

六是 The **RESIST** team, in collaboration with the **PESTO** team, works on methods for managing firewall configuration files that automatically reveal anomalies and help the administrator to find an adequate position for a newly added filtering rule.

七是 The **PETRUS** team designs access control mechanisms for the personal cloud that do not require the human user to understand the underlying access control mechanisms for enforcing a given security policy. In the IoT context, the team designs control models relying on embedded data management structures and algorithms for Things, so that data dissemination decisions can be evaluated closer to the data, within the Things that collected them.
The **PROSECCO** team aims at designing formally secure compilers for architecture with support for dynamic information flow control. The team also studies how hardware solutions for dynamic information flow tracking can be exploited to ensure that the running code satisfy security policies.

The **VALDA** team studies both foundational and systems aspects of complex data management, especially human-centric data. They proposed a collaborative access control model in the context of the language Webdamlog (distributed datalog). This model allows individuals to declaratively specify powerful policies governing access to their data, dissemination of their data, and delegation of computation.

a. https://github.com/chatziko/libqif

## 4.3 Trusted computing

### [Summary]

Trusted computing builds on assurances provided by secure hardware. Such hardware can ensure platform integrity and secure boot. It is also possible to provide trust at the application level rather than at the level of the whole platform by executing code in isolated, trusted execution environments, called enclaves. Additionally, these enclaves offer the possibility to attest that results were produced by a given code, opening the possibility for outsourced computation of secure data-oriented tasks.

Trusted computing consists in building secure hardware in order to derive global guarantees about computations made on a platform. The first use of this approach aimed at ensuring the platform integrity by using Trusted Platform Module (TPM) and secure boot. A TPM includes a unique cryptographic key and can compute hash functions. This allows it to authenticate hardware devices and to verify that software has not been changed, and thus to certify the integrity of the whole boot sequence. This permits the user to provide guarantees to external parties that their machine is indeed running a specific OS and specific applications. This approach aims at providing full platform integrity under the assumption that the OS and applications are trusted. Following this approach, TPM-based virtual machine monitors (VMM) have been developed, allowing for isolation of multiple guest OS's and thus isolation of attacks on these OS's. This led to the development of microkernels (e.g., SEL4) and unikernels (e.g., MirageOS), with the aim of minimizing the trust required in the OS and applications.

More recently, with the rise of secure areas in main processors (e.g., Intel's Software Guard Extension (SGX) and ARM's Trustzone), one can guarantee that code and data in memory is protected with respect to confidentiality and integrity. It is
thus becoming possible to ensure trust at that application level rather than at the
level of the whole platform. Indeed, these isolated execution environments offer
the capabilities to execute code in enclaves, whose memory and control flow is
protected from the environment (including the OS). Additionally, these enclaves
provide attestation capabilities, i.e., means for an external party to check that
claimed messages were indeed produced in an enclave running a specific piece
of code. This much more versatile approach leads to many interesting research
directions for securing computations both in the cloud and on home devices.
Typical applications include secure licensing, which leverages guarantees provided
by enclaves to ensure that licensed software is not illegally used.

An important challenge related to this disruptive technology is studying
its applicability in the case of secure outsourcing of (possibly distributed) data
computations. As Trusted Execution Environments (TEE) provide in hardware
security guarantees like confidentiality, integrity, and attestation, the code running
inside a (local or remote) TEE enclave can be considered as sticking close to a
fully honest behavior. This opens up the field to generic, efficient, scalable, secure
data-oriented computation tasks. VC3 is a typical preliminary attempt by Microsoft
towards trustworthy data analytics based on Intel SGX enclaves in the cloud. Two
main research issues must however be overcome. First, the hardware security
properties provided by TEE enclaves cannot be considered as unconditionally
unbreakable and should lead to investigating slightly different threat models.
Second, the most common data-oriented tasks (e.g., private search on the Web,
secure data stream processing in the IoT, privacy preserving machine learning)
must be efficiently transposed in secure counterparts based on TEE's. The challenge
here is to optimize and secure the execution of data-oriented primitives according
to the TEE constraints.

These lines of research are currently emerging. The current trend suggests
that the availability and diversity of TEE technologies will increase in the near
future. New solutions are forthcoming, with multicore platforms in which security/
isolation-oriented cores (à la SGX) would cohabit with other all-purpose cores,
thus allowing for separation of tasks inside the CPU. This suggests that trusted
computing will become ever more prominent in the near future, and Inria will
contribute to the research efforts into this direction.

[ Inria teams ] Trusted computing

↗ The PETRUS team works on new properties to reason about secure hardware computation in the context of distributed database computations, e.g., limiting the amount of data accessible to each agent/enclave. The team also studies database processing using secure elements or more advanced TEEs, like Intel SGX.
## 4.4 Intrusion detection and alert correlation

### [Summary]

Nearly any system contains, generally unintentional, flaws. An attacker can exploit them to bypass existing preventive security mechanisms, e.g., access control. Therefore system monitoring is of crucial importance to identify any violation of the security policy. In order to detect intrusions, the main and largely deployed approach consists in defining malicious symptoms and looking for their occurrences in various information sources (network, OS and application logs, etc.). An alternative consists in defining the normal activities of the monitored system and measuring the potential deviations from this normality. In both cases the challenge is to detect all intrusions but only the intrusions. In practice however, detection is far from being so perfect, leading to numerous false positives (false alerts) or false negatives (attacks not detected). This is why several Inria teams explore different ways of producing alerts.

Intrusion detection leads to a huge number of alerts, many of them being false positives. Thus an additional step is needed: alert correlation. This step consists in applying to the alert flow a series of transformations to improve the content of the alerts progressively (for example by adding information on the success of the corresponding attacks, on the origin of the attacks, on the vulnerability that has been exploited, on related alerts, etc.) and thus increase the "situation awareness" of the administrator.

Numerous flaws (i.e., vulnerabilities) are introduced in any system during its design, implementation or configuration. Generally, this introduction is not intentional, but the malicious introduction of vulnerabilities is still a possibility. These vulnerabilities may be exploited to bypass preventive security mechanisms used to enforce the security policy. In addition to preventive security, a second line of defense, namely intrusion detection, is thus mandatory. It consists in monitoring systems in order to detect any violation of the enforced security policy. By "intrusion" we thus mean "violation of the security policy" which breaks confidentiality, integrity, or availability. Using a malware (virus, worm, logic bomb, etc.) is of course a good way to implement such a violation.

Intrusion detection is a reactive security service that consists in collecting information on the operation of the system under surveillance and analyzing these activities to produce alerts if they are considered to be malicious. As the analyzer is very often prone to generate false alerts, a second step of analysis considers the flow of alerts and tries to eliminate the false alerts from this flow. In addition, alerts are correlated in order to identify multi-step intrusion scenarios. Finally, once the remaining alerts can be trusted, reaction to detected intrusions can be considered.
### 4.4.1 Intrusion detection paradigms

An intrusion detection system (IDS) analyzes data coming from either the network traffic (Network-based IDS, NIDS), the operating system or the applications (Host-Based IDS, HIDS).

The analysis of these two categories of data follows two different detection paradigms: misuse-based and anomaly-based intrusion detection. Anomaly detection consists in defining normal activities of the monitored entity and in identifying any deviation from this normality; while misuse detection consists in modeling malicious activities and detecting occurrences of these activities.

The most classic misuse detection approach, popularized in the late 1990s with the snort NIDS<fnref n="4" />, is to search for traces of known attacks in network packets. It involves constantly updating an attack-signature database; zero-day attacks<fnref n="5" /> are usually not detected. Furthermore, a compromise has to be made between the selectiveness of the signatures and the risk of false positives and false negatives. Very selective signatures increase the risk of missing attack variants, while more generic signatures may result in high false-alarm rate. In practice, signatures have often been very simple, thus very generic, in order to allow real time analysis of every event, usually leading to a higher rate of false positives.

Anomaly detection is less frequently used than misuse detection. Here, defining the normality through a behavior model is of course the critical point. If the model is too precise, the detector raises a high number of false alarms; if it is too lax, it misses attacks. Finding a good balance is difficult, especially when the model is built using statistics or machine learning, because of the possibility of under training or over training. In addition, a good balance for a given activity may later become unacceptable when the activity evolves. In any case, dealing with legitimate but unplanned behavior is problematic for anomaly detection.

The information gathered in network packets can sometimes be too poor semantically to allow for a good detection process. Therefore, intrusion and anomaly detection must also be addressed at both the application level and the operating system level. The most common anomaly detection mechanisms at application level consist in detecting a deviation of the control-flow of a program. A popular method to detect such anomaly is the use of application sequences of system calls. However, such methods can be bypassed by either mimicry attacks [WS02] or attacks against the integrity of the system call parameters.

<footnote n="4">4. https://www.snort.org/</footnote>

<footnote n="5">5. A zero day vulnerability is one that is unknown to those who would be interested in mitigating the vulnerability.</footnote>

<footnote>[WS02] David Wagner and Paolo Soto. Mimicry attacks on host-based intrusion detection systems. In *Proceedings of the 9th ACM conference on Computer and communications security (CCS)*, pages 255-264, New York, NY, USA, 2002. ACM.</footnote>
## [Research challenge 6] Intrusion detection for encrypted networks

Nowadays, intrusion detection is essentially realized at the network level. If, as expected in the near future, the traffic were more systematically encrypted, which would of course be a good practice for security and privacy, the analysis of the network packets would become *de facto* inoperative, apart from the header analysis. Therefore, it becomes important to study and design new mechanisms for monitoring information systems and producing alerts, at the application, middleware, operating system, and even firmware or hardware levels.

### 4.4.2 Alert correlation

We distinguish implicit and explicit alert correlation approaches. Implicit alert correlation uses data-mining paradigms to fuse or aggregate alerts, simply building on the similarity between alert features (e.g., IP addresses of the victim and the attacker), or using more advanced techniques to extract relevant information from alert groups (mining rules between attributes of alerts). Explicit alert correlation relies on security experts to specify logical and temporal constraints between alerts that correspond to complex attack scenarios, which generally require several steps to achieve their ultimate goal. When a complete or partial intrusion scenario is detected, a higher-level alert is generated.

Additional information about the characteristics of the attacks and about the context in which they occur is also useful to the correlation process. This knowledge allows one to take the context into account while processing the alerts, to identify false positives or to evaluate if a given attack has any chance of being successful given the context in which it occurs.

Unfortunately, it often remains difficult to write correlation rules that correctly exploit all the information available and that correctly translate the system administrator's expertise relative to possible attacks against the system. Automating the rule production is thus a current issue for research.

## [Inria teams] Intrusion detection and alert correlation

The CIDRE team has extensively studied intrusion detection (application level and network level) and alert correlation. At the application level, the team has proposed an approach to detect the corruption of data items that have an influence on the system calls. This approach consists in automatically building a data-oriented behavior model of an application by static analysis of its source code that is used to build constraints on data manipulated by the program. The application is then instrumented with executable assertions to check these constraints at runtime. At the OS level, the team proposed an anomaly detection approach in which the behavior model is not learned but given in the form of an information flow policy. The basic idea is to define where each information
(for example, the information contained in each file at the initialization of the system) can be stored, potentially mixed with another information. As for correlation, the team introduced an attack description language allowing one to define alerts to be produced by the attack and the logico-temporal rules between these alerts. These rules are used to configure a correlation engine realized by the team. CIDRE also proposed with its partners a data model to represent a system under monitoring: this model can be used during the correlation process to bring contextual information, for example for false alert identification. As these correlation rules are sometimes hard to define, the team also proposed, in collaboration with the DGA, a comprehensive process to generate such rules as automatically as possible, starting from attack trees (see §3.4) that have usually already been realized by the administrator to evaluate the threats against its system during the risk analysis phase.

The **LACODAM** team tackles the analysis of large network data to identify potential advanced persistent threats (APT) by discovering symptomatic patterns in the metadata of IP-packets. Finding such complex patterns over a large volume of streaming data implies revisiting the existing stream mining algorithms to improve their throughput dramatically, while guaranteeing a manageable false positive rate.

The **MYRIADS** team investigates misuse detection in cloud environment contexts. This is particularly tricky as the information system can be dynamically and automatically reconfigured. Security monitoring mechanisms should then be placed under control of the cloud provider and should follow the dynamics of the environment. In such a context, the team proposed a self-adaptable misuse detection system for IaaS clouds, which monitors changes in the virtual infrastructure of a cloud environment and reconfigures security probes accordingly. In addition, the team proposed a method to enable a cloud customer to verify that a network intrusion detection system located in the operator infrastructure is correctly configured, according to the Service Level Objectives figuring in the Service Level Agreement.

Focusing on network data, the **RESIST** team works at building solutions to characterize and detect unwanted network behavior. The team proposed a clustering and visualization method that allows one to analyze a large number of IP packets in order to make malicious activity patterns easily observable by security analysts. The team also proposed a technique to investigate https (thus encrypted) traffic. The team defined dedicated features for https traffic that are used as input for machine learning algorithms processing full tls sessions. This allows the early identification of encrypted web services in the tls session with a high degree of accuracy, which then enables anomaly detection in the usage of identified services. Another interesting contribution of the team is related to quantification of the number of monitoring nodes required to ensure an acceptable false positive rate for different network topologies. The team has shown that the false positive rate can be reduced by strategic monitoring node placement.
## 4.5 Malware analysis and detection

### [Summary]

Malware (viruses, worms, ransomware, spyware, adware, trojan horses, keyloggers, rootkits, etc.) is of course a major threat to our information systems (OS, applications, and data), especially on the client side (PC's or smartphones).

The goal of malware analysis is to obtain full understanding of a suspected malicious code: identify the targets (e.g., a particular end-user, a machine running a particular OS), attack actions (e.g., leakage of information, or encryption and ransom), techniques to bypass security mechanisms, and its own protection mechanisms to avoid detection. To succeed, the analysis must first defeat the anti-analysis protections put in place by the malware creator (obfuscation).

Malware detection is usually based on the analysis of any information received by a device (a machine, a phone, a firewall) or even on the complete scan of the files contained in a machine. The detection engine compares the data retrieved to a database of known features symptomatic of malware (i.e., malware signatures). The challenge is to maintain an up-to-date signature database since malware authors constantly generate new versions based on the same malicious code in order to escape scrutiny. Therefore, research projects propose detection techniques based on the concrete behavior of the malware, which remains constant across versions.

### 4.5.1 Malware analysis

Malware analysis intends to dissect any piece of code identified as suspicious and potentially malicious, targeting a full understanding of the malicious code in order to enhance existing security mechanisms or to design new counter-measures.

More precisely, malware analysis aims at identifying the malware's targets (a particular end-user, a company, any machine under a specific operating system, etc.), the actions it intends to perform to attack the targets (sensitive information leakage, encryption and ransom, etc.), the way it succeeds in bypassing the security mechanisms protecting the targets, the way it protects itself against malware detection engines.

Given the potential impact of a malware, it is crucial that the analysis takes place as quickly as possible. With respect to this perspective, scientific contributions on malware analysis are twofold. First, some approaches focus on automatic classification. Their goal is to distinguish benign code from malicious code and then to classify the malicious code into one of the known families. Significant efforts have been made in this area that have permitted to decrease the human workload by reducing the number of samples to be analyzed manually. Remaining samples are generally either issued from unknown malware or too protected to be automatically processed. Other approaches thus aim to help the experts to reverse engineer and understand the malicious code.
An analysis can be performed statically, without executing the malware. It offers several benefits. Firstly, static analysis is safe for the host-architecture as the malicious code is not executed. Secondly, static analysis brings insights about all possible executions of the code and thus of about all possible behaviors of the malware.

Unfortunately, since malware authors are aware that their code will probably be confronted with a static analysis, they use various techniques to make static analysis and reverse engineering much harder. They use obfuscation techniques like packing, control flow flattening, or opaque predicates. All these techniques make the control flow graph computed from the malicious code irrelevant for most static analyzing tools.

An analysis can also be performed dynamically, which means that the malicious code is executed as far as possible. The main goal of this kind of analysis is to observe the real and concrete effects of the attack on its target. A difficulty here is to trigger the execution of malicious part of the code.

### 4.5.2 Malware detection

We distinguish here two kinds of malware. Regular commercial-grade malware intends to cause damage to an important population of end users as quickly as possible and can be found on popular platforms such as Windows or Android. By contrast, target-specific malware intends to cause damage to specific platforms, such as the Stuxnet worm that targeted programmable logic controllers of Iranian nuclear plants. Target-specific malware are built with great care and are therefore especially hard to detect<fnref n="6" />. In addition, being more stealthy, they are often only tardily identified, when external symptoms reveal their existence: Stuxnet was revealed in 2010 but it is thought to have been deployed in 2005 or even earlier.

Regular, commercial-grade pieces of malware were initially and mostly developed to target the Windows operating systems and applications, as it was the most widely used operating system. Nevertheless, during the last ten years, Android has also become a popular target for malware authors. A piece of malware may infect its target from different entry points, such as user-downloaded applications, visited websites, or email attachments, and can often spread through different networks. The detection of malware usually occurs during the scan of either network traffic (files attached to mail, for example) or machines. In both cases, the detection engine compares the data to analyze with a database of malware signatures (known features that are symptomatic of malware). The main challenge consists in building and maintaining an up-to-date signature

<footnote n="6">6. The former Carte team lead a deep analysis of Duqu (https://en.wikipedia.org/wiki/Duqu), one of the most sophisticated malware in 2011, and developed a malware detector based on morphological analysis, a technique based on control flow graph comparison. The activity on malware analysis is carried on in the Carbone team of LORIA.</footnote>
<figure><img src="image_18.png" /><figcaption>Detecting unwanted network behavior – © Inria / Photo C. Morel</figcaption></figure>

database. This is a cat and mouse game: on the one hand, malware authors try to avoid detection as long as possible while minimizing the production code effort, and, on the other hand, defenders have to produce malware signatures that are as accurate and complete as possible by deep analysis of known malware, as explained in the previous section.

Malware authors usually build on malware kits and additional armoring techniques to generate new executable files based on the same malicious code. This way, a huge number of variants can be generated from a single, original sample. For instance, new variants of Cerber are generated every 15 seconds. The detection of a new malware or a new variant of a known malware remains an open problem. A recent advance is to use behavioral approaches, instead of static signatures, by monitoring any deviation from a model of the normal behavior, for instance by modeling information flow at the operating system level.

Starting from around 2012, scams relying on ransomware have grown internationally. A ransomware is a specific type of malware that restricts access to a computer system or to its hosted data (classically by enciphering the data). A ransomware requires the user to pay a ransom to the attacker to remove the restriction. Emergence of ransomware are probably linked to the Bitcoin system, which permitted anonymous payoffs and made massive attacks profitable.
[Inria teams] Malware analysis and detection

The CIDRE team works on malware analysis, malware detection and malware de-obfuscation. Regarding malware analysis they target the Android environment. The idea is to use information flow tracking to analyze the malware behavior and, potentially, generate a signature of the malware based on its real activities (i.e., engendered information flows). Like all dynamic analysis, this work is relevant only if the malicious part of the code is really executed. The team thus focuses on the automatic triggering of malicious code. The malware detection tool monitors the usage of filesystem data to check for data deviations with respect to their normal use. Moreover, a postmortem action using an unclassified Machine Learning algorithm provides clues to clearly identify the detected malware. The work on malware de-obfuscation aims at circumventing protections of the malware itself against static analysis.

The LHS Rennes collects ransomware (Malware'O'Matic platform). As this kind of malware has a short lifespan, this requires to periodically renew the database and verify which ransomware are still alive.

4.6 Reaction to detected attacks

[Summary]

Ideally, intrusion and malware detection, as well as alert correlation should lead to the detection of all attacks without false alerts. Therefore, the next obvious step is to respond (possibly automatically) to the detected attacks through appropriate actions: modification of the security policy, new configurations of existing security mechanisms, implementation of new security mechanisms, patch deployment, etc. Of course, it is important to prevent the countermeasures from having similar or even worse consequences than those of the attack itself.

Ideally, alert correlation should lead to the detection of all attacks without false alerts. Therefore, the next obvious step should naturally be to respond (potentially automatically) to the detected attacks. Automatic response is also made necessary by the speed with which an attack may propagate, and the considerable damage it could make before manual response. Considering that the security policy has been violated although preventive mechanisms have been used to enforce this policy, two levels of reaction can be considered: (1) the attack may have succeeded because the policy was incorrect, in which case the policy must be amended, and new configurations of existing security mechanisms or even new security mechanisms must consequently be put in place; (2) the attack may also have succeeded because the enforcement of the policy was incorrect,
in which case configuration errors of the security mechanism must be identified and corrected. Using formal methods can help in guaranteeing that the security properties requested by the policy are effectively insured at the policy level and at the enforcement level. Of course, it is important that the countermeasures (modification of the policy, new configurations, new security mechanisms, patching, etc.) are not similar or worse than those of the attack itself. For example, when trying to stop a DDoS attack, legitimate packets may also be discarded: the service is then unavailable to legitimate users, which was the very objective of the attack.

### [Inria teams] Reaction to detected attacks

The CTRL-A team targets the relatively little-studied topic of models and control techniques for the automated reaction to attacks. The team uses detection information to identify the appropriate defense and repair actions, so that the system can remain operational, entirely or in a degraded mode. In terms of Autonomic Computing, this capacity is referred to as self-protection.

In reaction to DDoS attack, legitimate packets may be discarded. The service may then be unavailable to legitimate users. This is why the RESIST team proposes scattering the DDoS traffic in time and space: by introducing voluntary delays and longer routes, whereas performance can be degraded, service is not discontinued.
5

Privacy and personal
data protection

<figure><img src="image_19.png" /></figure>
Privacy is often defined as the ability for individuals to control their personal data and decide what to reveal, to whom, and under what condition. There is however no single definition since the notion of privacy remains intimately linked with our cultural roots. For instance, the notion of personal data, the cornerstone of privacy, has received several definitions, not always compatible, depending on countries. In France, *personal data* is defined by the "Loi Informatique et Libertés" from 1978, as information that can be directly or indirectly linked to a person, by the data controller (in charge of data processing), or any third party, using any type of means. Therefore, it has a very broad scope, irrespectively of the nature of data. Additionally, *Sensitive Personal Information* is personal data related to such domains as health, politics, religion, or sexual orientation, that cannot be collected and processed except under well-defined situations. These notions and the associated obligations constitute the cornerstone of the French and European regulations (e.g., through the European General Data Protection Regulation, GDPR), as will be discussed in this chapter. The notion of *Personally Identifiable Information* (or PII), widely used in the U.S.<fnref n="1" />, is close to that of personal data as defined above, but is not equivalent.

Privacy considerations have become a central topic in our connected world. Several domains, as yet unaffected by this trend, will soon generate huge amounts of personal and sometimes sensitive data, without leaving users any option to opt out. Privacy is therefore a key question, as important as security.

Being naturally multi-faceted, the work on privacy encompasses several dimensions:

* part of it is *legal*: harmonized rules are needed, that apply in the largest geographic area possible, in order to favor good practices and ban the others. In order to be applicable, guidance may be needed to implement these harmonized rules and this can be really challenging;

* part of it is *technical*: advanced privacy tools are needed in both theoretical and applied areas. They can help analyzing and improving existing systems or help designing privacy preserving systems from scratch;

* part of it is *economic*: understanding the underlying ecosystem is essential, since it often determines the practices in terms of personal data collection and processing. A sustainable ecosystem respectful of European regulation in terms of data protection is needed;

* part of it is *cultural*: the peoples of different geographical areas can have different approaches to privacy, due to their cultural roots, and these differences impact the local regulation in terms of personal data;

* finally, part of it is *sociological*: the end user is often inclined to declare herself concerned by privacy while at the same time behaving in an opposite manner.

<footnote n="1">1. http://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-122.pdf</footnote>
This well-known “privacy paradox” highlights the need for sociological studies to better understand human behaviors in this domain and potentially improve awareness and practices.

This chapter essentially covers the technical aspects and the legal dimensions up to a certain point, since the other aspects are out of Inria's research scope. We first present the high-level principles and regulations around privacy. We then present tools and privacy enhancing technologies. Finally, we discuss privacy leaks on existing systems, where personal data is either deliberately communicated by the user or collected without the user's knowledge.

<figure><img src="image_20.png" /></figure>

# Privacy principles and regulation

[ Summary ]

To take into account the major changes that took place during the last decade in terms of collection and use of personal data, the European Union adopted the General Data Protection Regulation (“GDPR”) that came into effect in May 2018. The main change is the emphasis put on the responsibility of the data controllers, i.e., the organizations processing personal data, as well as their sub-contractors, if there are any. Any data controller must conduct data protection impact assessments, implement privacy by design and be accountable. If the impact assessment indicates that the processing is likely to severely impact the rights and freedom of physical persons, the measures taken will have to be strengthened. The rights of a data subject are also strengthened with better information and control over her data, following the user empowerment philosophy.

However, the GDPR provides very little guidance about the effective implementation of these concepts. An interdisciplinary work is needed to reduce this gap between legal and technical instruments, for instance by defining rigorous privacy risk analysis and privacy by design methods, or by defining techniques to strengthen accountability, transparency and enhance the user's control over her personal data. In any case, privacy comes with a price since tensions exist with several other considerations and privacy is sometimes regarded as a limiting factor.

A certain number of fundamental principles and legal considerations govern privacy. This section discusses them, in particular those associated to the new European regulation (GDPR) and other legal texts (e.g., the ePrivacy regulation that particularizes and complements the GDPR).

5.1.1 Tensions between privacy and other considerations

Privacy comes with a price. In a context where many commercial services depend on personal data (e.g., many free services are supported by targeted
advertising), where big data can offer highly valuable services (e.g., the study of the flu virus propagation through the analysis of medical acts), where several countries deploy mass-surveillance systems meant to help fight terrorism, where a certain form of user traceability is needed in order to enable respectful relationships between citizens, privacy can be regarded as a limiting factor. This is the sign of a fundamental tension between privacy, that requires to minimize personal data collection, and other considerations like utility, security, or accountability, where the higher the volume and accuracy of data, the better. A compromise is therefore needed and the idea of finding an appropriate balance is central for instance to data protection regulations. This balance of course heavily depends on cultural aspects, hence the importance of having a European regulation in the domain in order to preserve our sovereignty. The following sections will address and illustrate this tension according to several angles.

## 5.1.2 Evolution of the regulatory framework

The notion of privacy is complex and multifaceted. In addition, its perception evolves through time and space and is affected by the adoption of new technologies. To take into account the major changes that have taken place during the last decade in terms of collection and use of personal data, the European Union adopted in 2016 the "General Data Protection Regulation" (or GDPR) taking effect in May 2018, throughout the State Members in a uniform manner.

The biggest shift introduced by the GDPR is the emphasis put on the responsibility of the data controller (i.e., the private or public organization or association processing personal data) as well as its sub-contractor (if there are any). Any data controller must:

* conduct data protection impact assessments;

* implement privacy by design;

* and comply with the accountability principle.

If the impact assessment indicates that the processing is likely to impact the rights and freedom of physical persons severely, the measures taken will have to be strengthened. The rights of the data subjects are also strengthened in order to improve their information and control over their personal data, following a user empowerment approach. For instance, the data controller must be in position to prove they have obtained explicit user consent, users have the "right to be forgotten" on search services provided in Europe, a new data portability right is added in order to enable a user to switch to a different platform while reusing their data, and children under the age of 16 are better protected.

However, the GDPR provides very little guidance about the effective implementation of these new provisions and some of them raise a number of technical challenges. Before describing research on privacy enhancing technologies in
§5.2, we provide an overview of the challenges raised by legal requirements and interdisciplinary works to reduce the gap between legal and technical instruments.

5.1.3 Data Protection Impact Assessment (DPIA)

Data Protection Impact Assessments (DPIA), or simply Privacy Impact
Assessments (PIA), are used by organizations to assess any privacy issue that
might arise when developing new products or services that involve the processing
of personal data. Conducting a DPIA is made mandatory in Europe by the GDPR
for certain categories of personal data processing. Beyond legal requirements,
conducting a DPIA is in the interest of any organization to ensure that privacy
risks are properly understood and addressed before deploying any new product
or service. There is already a large body of contributions on DPIAs and a number
of DPIAs for specific products have been published. The Commission Nationale
Informatique et Libertés (CNIL), the French data protection agency, has also
recently released a tool to help data controllers prepare a DPIA<fnref n="2" />.

All these contributions are very useful to define the DPIA process (including
planning, stakeholders consultation, resource allocation, or audits) and its main
goal (evaluating the likelihood and severity of privacy threats). However, they
do not define very precisely how the technical part of the DPIA, the privacy risk
analysis, should be performed.

There are of course a number of commonalities between privacy and security risk analyses. However, privacy is a more complex and multifaceted concept aiming at the protection of people (i.e., individuals, groups, and society as a whole) rather than resources or organizations. These dimensions must be considered in a privacy risk analysis, especially the notion of privacy harm that has been extensively discussed by lawyers.

Even if privacy risk analysis frameworks are being defined, work remains to
be done. First, dedicated privacy risk analysis frameworks for specific application
areas would be very useful in practice. On the theoretical side, it would also be
interesting to establish formal links with privacy metrics. Last but not least, beyond
legal compliance, a great benefit of a privacy risk analysis should be to provide
guidance for the design of a new product, following the privacy by design approach.

5.1.4 Privacy by Design (PbD)

The philosophy of privacy by design, made mandatory by the GDPR, is that
privacy should not be treated as an afterthought but as a first-class requirement
in the design of any system. However, from a technical standpoint, it remains a
challenging endeavor:

<footnote n="2">2. https://www.cnil.fr/fr/PIA-privacy-impact-assessment</footnote>
* privacy includes a variety of dimensions (such as collection limitation, data quality, purpose specification, use limitation, or security) which are generally not defined very precisely;

* then, these requirements may seem to be in tension with other requirements such as functional requirements, ease of use, performance, or economic viability of the product or service.

In order to implement privacy by design, a wide array of Privacy Enhancing Technologies (PETs) are available, as discussed below. Each PET provides different guarantees based on different assumptions and therefore is suitable in different contexts. As a result, it is quite complex for a software engineer to make informed choices among all these possibilities and to find the most appropriate combination of techniques to solve their own requirements. Solutions have been proposed in different application domains such as smart metering, pay-as-you-drive, or location-based systems but the next challenge in this area is to go beyond individual cases and to establish sound foundations and methodologies for privacy by design.

A formal framework, for example based on epistemic logic, can be helpful to express data minimization requirements as properties defining for each stakeholder the information that she is allowed to know or not. But conflicting requirements often have to be met simultaneously, for instance guarantees about the correctness of a computation's result. In fact, the tension between data minimization and correctness is one of the delicate issues to be solved in many systems involving personal data. A formal framework is also useful for reasoning about architectures. Its axiomatization can be used to prove that a given architecture meets the expected privacy and integrity requirements.

### 5.1.5 Accountability

In line with previous regulations (e.g., the 1980 OECD Guidelines on the Protection of Privacy and Transborder Flows of Personal Data), the GDPR integrates accountability as a principle that requires that organizations put in place appropriate technical and organizational measures to demonstrate their compliance with the regulation.

A distinction can be made between three main types of accountability:

* Accountability of policy can be seen as the first level of accountability: the organization should be able to demonstrate that it has defined a clear and properly documented privacy policy;

* Accountability of procedures, which refers to the demonstration of organizational mechanisms such as documented processes to cope with user consent or to address complaints or personal data requests;

* Accountability of practice is the *a posteriori* demonstration of the effectiveness of the procedures' accountability. This is a proof that the privacy policies have been effectively met.
The first type of accountability is purely declarative and provides at best a form of legal guarantee (binding commitment). The second type adds guarantees at the organizational level but only the third type can deliver the full promises of accountability. However, focus has been placed on the first and second types of accountability so far, resulting in superficial guarantees.

For accountability to contribute effectively to the enhancement of privacy protection, it is necessary to be able to translate its general principles into practical measures and to consider its various dimensions. For instance, collecting and keeping detailed processing logs can contradict privacy if those logs contain personal data. An appropriate balance between these requirements must be found.

### 5.1.6 User empowerment through control and transparency

A service that is compliant with the legislation and for which the data controller is accountable, is not necessarily in line with user expectations. Empowering the user is required in order to gain user acceptance.

#### EMPOWERMENT THROUGH USER CONTROL

Privacy is increasingly seen as the ability for a user to control their personal data. However, even if this notion is predominant in the privacy literature and plays a central role in the GDPR, clear definitions are still missing. The word “control” is usually used in a very vague way in this context, both by lawyers and by computer scientists. For example, such notions as “access control” or “usage control” in computer science do not really encapsulate the intuition underlying the notion of control over personal data. This lack of precision may lead to misunderstandings and makes it difficult to check compliance. A multidisciplinary study of the notion of control as used by lawyers and computer scientists has led to identify three dimensions, corresponding to the capacities for an individual:

* to perform actions on their personal data;

* to prevent others from performing actions on their personal data;

* to be informed of actions performed by others on their personal data.

On the practical side, two main conditions must be met to make it possible for data subjects to exercise control over their personal data:

* the user must be properly informed about the collection of their data, its purpose, the entity collecting the data, and for instance the retention delay (transparency);

* the user must be able to express their choice to have their data collected or not for a given purpose and have guarantees that this choice is actually followed (consent).

In fact, the explicit consent of the data subject, which is a cornerstone of most data protection regulations, is a typical example of a legal requirement that is very difficult to put into practice. This is the case with Internet of Things (IoT)
where many data communications occur without user notice, or social networks for instance, and several proposals have been made to help a user to exercise their control.

## EMPOWERMENT THROUGH TRANSPARENCY

Transparency is an essential concept to design privacy preserving systems and services. The data controller should provide clear and complete information on what information is collected, how often, for what purpose, how data is processed, how data is stored (where, how long, with what security), and whether data is likely to be communicated to third parties. The same questions recursively apply to third parties to whom data may be transferred.

Transparency is a challenge in various domains where data collections happen in an invisible manner. This can be the case with Internet of Things (IoT) systems that measure and collect continuously personal data. The information provided to the data subjects should be as visible and intelligible as possible (which excludes simple signs on walls that generally remain unnoticed). A new design space for effective privacy notices is required, which is an active research topic in the privacy area.

Beyond the transparency of data collection, a new major challenge is the transparency of algorithms. This need for more transparency is another illustration of a legal requirement (in the GDPR and also in the law for a digital Republic<fnref n="3" /> adopted in France in October 2016) which raises many technical challenges and can be a source of interdisciplinary research topics.

Algorithm transparency is central in many automated systems with societal impacts (e.g., with an automated assignment system, once all candidates have posted ordered preferences): the question of a possible algorithmic bias, whether intentional or not (e.g., bug), is unavoidable. Transparency of algorithms is necessary to enable third parties (including citizens) to analyze their internal behavior. This is the role of the TransAlgo⁴ French initiative led by Inria.

Since transparency requires detailed information to be communicated to the user, the way it is achieved is non-trivial. The usual approach, through the General Terms and Conditions of Use, is usually far from satisfying, being non-user-friendly: this legal document is often meant to protect the company rather than informing the user. Research is in progress on analyzing practices, on trying to have a standardized format understandable by everybody (like Creative Commons did in a different domain) and perhaps amenable to mechanized treatment, and on the impacts of transparency on user behavior.

<footnote n="3">3. Loi pour une Republique numerique
https://www.economie.gouv.fr/republique-numerique.
4. https://www.inria.fr/en/news/news-from-inria/transalgo</footnote>
## [Research challenge 7] Understanding privacy and deriving practical tools

Understanding privacy principles and regulations is the foundation of any activity in privacy. Although this is not a new domain (e.g., the “Loi Informatique et Libertés” was adopted in 1978), this area has recently experienced major evolutions with the new GDPR European regulation and at the same time new opportunities to collect personal data. As a consequence, understanding the concepts and the regulation is a first necessity. Being able to derive practical tools is another one: even though the GDPR promotes several concepts and goals, it provides little guidance about the effective implementation of these new regulatory provisions.

In particular the GDPR introduced the right to data portability whereby a user can retrieve their data in a human readable and machine portable format. This right opens new research areas around individualized management and control over one's personal data. The goal is to empower citizens to leverage their personal data for their own good, which calls for secure, extensible, and sovereign personal cloud platforms, three conflicting goals that open new research challenges (see e.g., §5.2.4).

## [Inria teams] Privacy principles and regulation

* The **CIDRE** team works on privacy policies and the right to be forgotten in collaboration with lawyers.

* The **INDES** team works on the relationships between web tracking and the ePrivacy Regulation, in relationships with lawyers. In particular the team evaluates the privacy impacts when the Regulation doesn't require user consent for tracking and builds tools that detect violations of the Regulation.

* The **PETRUS** team works on control over personal data and data minimization and has strong collaborations with research groups in other disciplines such as economy, law and social sciences.

The team also contributes to enhance individuals' control over their personal data from an architectural point of view. This is the case of PlugDB, a secure personal server that allows individuals to exercise control over their personal data, while preserving durability, availability, and sharing.

* The **PRIVATICS** team works on most subjects, with a strong emphasis on interdisciplinarity through collaborations with lawyers and economists. For instance, the team has contributed to a framework and methodology for conducting a privacy risk analysis in a rigorous and systematic way, compatible with most DPIA recommendations. Concerning privacy by design, the team has proposed a formal framework based on epistemic logic that enables one to express data minimization requirements. This framework has been used to formally compare different architectures for biometric access control.
Concerning accountability, the team has defined a set of practical measures to be taken at each phase of the personal data life cycle, from collection to deletion, including storage, usage, and forwarding to third parties. On the formal side, the team has proposed a framework based on privacy friendly logs, showing that compliance can be checked based on logs that do not contain personal data.

Concerning user control, after a multidisciplinary study of this notion as used by lawyers and computer scientists, a formal model has been derived to formally characterize each type of control.

On the particular case of IoT and user control, the team has proposed an architecture based on "Privacy Agents" implementing the choices of the data subject, expressed in a restricted natural language that can be easily understood by non-experts.

<figure><img src="image_21.png" /></figure>

## 5.2 Privacy tools

### [Summary]

Tools for different types of publics, from a data protection officer down to an end-user, are needed in order to transition from the high-level principles of GDPR to privacy compliant products and services. Some of them focus on such principles as data protection impact assessment, privacy by design, and accountability.

Other tools are meant to anonymize a database before releasing it for open data access. However, this is a complex task that requires an appropriate trade off to be found: increasing privacy usually reduces the utility of an anonymized database. From this point of view, the differential privacy concept turned out to be a key tool in order to provide provable privacy guarantees.

The advent of personal clouds, meant to give users a full control over their own data, is another major tool for user empowerment.

Finally, techniques have been designed to provide unlinkability, i.e., the guarantee that no one can link several uses of a service by a given user. This is particularly important for systems that embed an RFID token, like electronic passports. Concerning communications, the Tor system tries to guarantee anonymous communications (no one can identify the source of a packet), although in practice this is less usable.

### 5.2.1 Tools related to DPIA, privacy by design, and accountability

Several tools have been designed in order to address the needs for data protection impact assessment, privacy by design, and accountability. Some of them are respectively discussed in §5.1.3, §5.1.4 and §5.1.5.
5.2.2 Database anonymization: a necessity for open-data and big-data

A strict regulation applies to any data controller in charge of a database containing
personal data. Evading this regulation is possible by anonymizing this database, since
the resulting database no longer contains any personal data. For instance, this is a
prerequisite for releasing a public dataset in the context of an open-data initiative.

Data anonymization consists in altering the dataset in order to remove any
information that could be used to re-identify any participant of the dataset or to
infer personal attributes. This is not an easy task. A first reason is that anonymization
is intrinsically complex and domain dependent: there is no universal solution. Certain
types of data, such as mobility traces, are highly unique and therefore identifying. For
instance, the knowledge of four spatiotemporal points may be enough to uniquely
identify 95% of the individuals in a large cell phone operator mobility database [dMHVB13].

A second reason is that anonymizing a database is one thing, preventing re-identification through side information (the “inference” problem) is another one. The AOL de-anonymization of pseudonym No. 4417749 [BZ06] is famous from this point of view. By analyzing the detailed records of searches she made, her identity has quickly been found. In general, replacing names by pseudonyms is now known to be highly vulnerable to re-identification attacks.

Last but not least, an appropriate trade-off between privacy and utility is
needed (see §5.1.1). Guaranteeing meaningful privacy requires the distortion of the
original dataset which mechanically yields imprecise, coarse-grained knowledge
even about the population as a whole.

Data anonymization can be achieved through various types of approaches. In
2014 the G29 group, gathering the European data protection agencies, published
a technical document on the topic [Art14]. This document discusses the effectiveness
and limits of various techniques: permutation, differential privacy (see §5.2.3),
aggregation, k-anonymity, l-diversity, and t-closeness. In terms of privacy
guaranties, these techniques can be classified into two categories: syntactic and
semantic privacy models. Syntactic models focus on syntactic requirements of
the anonymized data (e.g., with k-anonymity each record should appear at least
k times in the anonymized dataset), without any guarantee on what sensitive
information the adversary can exactly learn about individuals. Semantic models,
on the other hand, are concerned with the private information that can be inferred
about individuals using the anonymized data as well as perhaps some prior (or
background) knowledge about them. A higher level of privacy can be expected
from the semantic models. This is the case of differential privacy that is discussed
in the following section.

[dMHVB13] Yves-Alexandre de Montjoye, Cesar A. Hidalgo, Michel Verleysen, and Vincent D. Blondel. Unique in the crowd: The privacy bounds of human mobility. *Scientific Reports*, 3:1376, 03 2013.
[BZ06] M. Barbaro and T. Zeller. A face is exposed for aol searcher no 4417749. New York Times, August 2006.
[Art14] Opinion 05/2014 on anonymisation techniques, April 2014.-
### 5.2.3 Differential privacy

Differential privacy [DMNS06] was originally proposed in the area of statistical databases and it is nowadays one of the most successful approaches to privacy. The goal is to protect an individual's data while publishing aggregate information about the database. This is obtained by adding controlled noise to the query outcome in such a way that the data of a single individual will have a negligible impact on the reported answer. Differential privacy has several advantages: (1) it is independent from the side-information of the adversary, meaning one does not need to take into account the context in which the system will operate; (2) it is compositional, i.e., if we combine the information that we obtain by querying two differentially-private mechanisms, the resulting mechanism is also differentially-private; and (3) differentially-private mechanisms usually provide a good trade-off between utility and privacy.

A successful variant, called local differential privacy, has the advantage of requiring no trusted third party: users obfuscate their personal data, adding noise by themselves, before sending it to the data collector. Local differential privacy is particularly suitable when data is collected for statistical purposes, as it usually achieves a good trade-off between privacy and utility. Local differential privacy has the same advantages as differential privacy (independence from side-knowledge and compositionality). It had a considerable impact after large companies like Apple and Google adopted it for privacy preserving data collection systems (e.g., Google uses a particular implementation in the RAPPOR crowdsourcing technology).

There are many cases in which the data domain features a notion of distance (e.g., location, energy consumption in smart meters, or age and weight in medical records). Then the privacy/utility trade-off can be greatly improved by exploiting the concept of approximation intrinsic in the notion of distance. The idea is to allow two values to become more and more distinguishable as their distance increases so that more accurate statistics can be made.

#### [Research challenge 8] Open data and anonymization

Open data initiatives may sometimes mean releasing databases that contain sensitive, personal information. To ensure privacy of the individuals, data need to be anonymized. Robust anonymization, that effectively resists de-anonymization attacks, is an active and hot research topic. If differential privacy has become a key scientific tool to achieve provable anonymization guarantees, challenges remain on its application, for instance in order to improve the privacy/utility trade-off.

<footnote>[DMNS06] C. Dwork, F. McSherry, K. Nissim, and A. Smith. Calibrating noise to sensitivity in private data analysis. In *Theory of Cryptography (TCC'06)*, 2006.
https://ec.europa.eu/justice/article-29/documentation/opinion-recommendation/files/2014/wp216_en.pdf</footnote>
### 5.2.4 Empowering users with personal clouds

The time of individualized management and control over one's personal data is upon us. Thanks to smart disclosure initiatives and the right to data portability in the GDPR, we can access our personal data from the companies or government agencies that collected them, thus allowing the creation of cross-domain personal data repositories. Concurrently, Personal Cloud solutions, also called Personal Information Management System (PIMS) or Personal Data Server (PDS), are flourishing (see, for instance, the French startup Cozy Cloud<fnref n="5" />). Their goal is to empower us to leverage our personal data for our own good, opening the way to new value-added services by crossing data issued from different data silos or by sharing our data for social/societal benefits (e.g., contribute to an epidemiological study, compute queries based on shared data within communities of users).

However, managing our own personal data constitutes a considerable burden. We must ensure the security of the data we gather and manage the disclosed data and control its usage. We inherit the combined responsibility of an information security expert and a database administrator. Therefore, personal cloud providers propose solutions to manage personal data on behalf of their customers, thus creating genuine honeypots (centralizing huge amounts of personal data belonging to millions of individuals). Thus, paradoxically by empowering users, smart disclosure and personal clouds push them towards even more delegation over even more data, thus exposing them to a larger risk than ever.

<figure><img src="image_22.png" /></figure>

Dedicated personal cloud prototype – © Inria / Photo C. Morel

<footnote n="5">5. https://cozy.io/en/</footnote>
To escape this paradox and truly achieve the empowerment of the user, a set of expectations should be fulfilled: (1) sovereignty: the system must offer the user the ability to exercise her data disclosure decisions under her own authority and without any form of delegation; (2) security: the system must provide tangible guarantees about the enforcement of these decisions whatever the type of attacks or misuse the system could face not only to the personal cloud holder but also to the other users and to third parties; and (3) extensibility: the system must not impede the development of new services using the data of a single individual ('Personal Big Data') or of large groups of individuals ('Big Personal Data'). However, these statements introduce two forms of tension, a first one between sovereignty and security: to reconcile the absence of delegation to central IT experts or administrators with high security guarantees; and another one between security and extensibility: to reconcile security, which calls for closed systems, with the need for extensibility to support new data services, which by definition are not fully trusted. Deriving an architecture to fulfill these three requirements altogether is an intrinsically difficult issue.

Another important problem linked to users' empowerment is to allow for novel big personal-data applications (e.g., participatory sensing, epidemiological studies, and personalized recommendation systems). Distributed personal-data processing is not a new issue. However, the personal cloud is distributed at the individual level and is expected to scale up to nationwide populations. In this context, the first issue is to prevent or minimize the effect of personal data leaks during computations performed at such scale. Also, decentralization emphasizes the central role of the individual in the architecture. This calls for the emergence of new forms of decentralized computations where individual profiles and individual privacy settings are integrated by construction.

Beyond the privacy and security perspective, many important research problems around personal clouds remain to be addressed: foundational and systems aspects of complex data management, especially with human-centric data, and orchestration of queries to the various services.

### 5.2.5 Privacy preserving protocols and communication technologies

#### UNLINKABILITY PROTOCOLS

According to the ISO/IEC 15408-2 standard, unlinkability, also referred to as untraceability, "ensures that a user may make multiple uses of resources or services without others being able to link these uses together." Of course, this notion of unlinkability applies both in the physical world (i.e., tracing a person) and in the virtual world (i.e., linking transactions of a person).
Unlinkability is getting increasingly important with the widespread use of RFID tokens for authentication. If the authentication protocol guarantees unlinkability, it should indeed be impossible to place an unauthorized RFID reader that can decide whether a given token is the same as a previously seen token. For instance, since RFID tokens are used in electronic passports, an attacker who eavesdrops a first legitimate protocol execution for a target, say Alice, should not be able to setup a reader that distinguishes Alice's passport from others. Similarly, the authentication protocols implemented in 3G/4G/5G mobile telephone networks prevents an eavesdropper, other than the operator, to link different communications made by the same device.

Similarly, digital transactions may require unlinkability. A typical example is a blockchain based electronic currency where the ledger is completely public. Such systems generally provide pseudonymity (i.e., use of a pseudonym rather than a real identity). However, by linking several transactions made by the same user, user profiling is trivial and then reidentification may be possible with side information. That said, Zerocash 6 is an example of an electronic currency that provides unlinkability guarantees.

More generally, authentication and authorizations are sometimes required while guaranteeing anonymity and unlinkability. In that case, Direct Anonymous Authentication (DAA) protocols provide anonymous authentication tokens, allowing remote anonymous authentication.

Finally, Fingerprinting is a direct threat to unlinkability. Browser fingerprinting is discussed in §5.3.4. However, even end devices may be fingerprinted: for example, the length of the encrypted message containing personal data, in particular the JPEG photo of the electronic passport can be effectively used for tracking 7.

## ANONYMIZED COMMUNICATIONS

Tor is an anonymizing system that aims at preserving the anonymity of Internet users. This is achieved thanks to so-called "onion routing": instead of direct connections from a host to a destination, an application-level virtual circuit is created that goes through a certain number of relaying nodes spread throughout the world and chosen randomly. Traffic is encrypted by the client as many times as there are relays in this virtual circuit. Then each node decrypts the outer part of the packet received, thereby revealing the IP address of the next relay, and forwards the packet to this relay. Thereby, Tor prevents any eavesdropper to identify the source and destination hosts by looking at the packet source and destination IP addresses.

However, even if theoretically interesting, this solution is severely flawed for the general public because of the amount of knowledge required to correctly operate on top of Tor. Indeed, several types of de-anonymization attacks against Tor are regularly found.

<footnote n="6">6. http://zerocash-project.org/</footnote>

<footnote n="7">7. https://www.inria.fr/centre/nancy/actualites/securite-des-donnees-les-passeports-biometriques</footnote>
## [Inria teams] Privacy tools

The **COMETE** team works on differential privacy and its variants, in particular when a notion of distance exists, as well as techniques to measure the utility of the result. For instance, the team has implemented $d^x$-privacy and works on a tool to retrieve as accurately as possible the original distribution from the noisy one. The team also develops a tool to measure the result's utility in terms of the quality of the true distribution's approximation. Finally, the team also works on the unlinkability definition and its formal verification.

The **DIANA** team has worked on de-anonymization attacks on Tor.

The **PESTO** team works on the unlinkability definition and its formal verification.

The **PETRUS** team works on solutions and architectures for personal cloud systems and on trusted environments for privacy preserving decentralized computation. In particular the team works on a new reference architecture where potentially complex manipulations of personal data rely on Trusted Execution Environments (TEE). The objective is to limit the side effects in terms of data leaks since only expected results are declassified to third parties, without any direct access the raw data.

The **PRIVATICS** team works on various tools for privacy risk analysis, privacy by design, accountability, and user control. The team also works on different aspects and models for data anonymization. The team developed several anonymization schemes for mobile traces and set-value datasets under the k-anonymity or differential privacy models. The team also worked on a novel technique for privately releasing high-dimensional datasets using generative neural networks. The idea is to produce a synthetic dataset that resembles the original training data as much as possible while complying with privacy requirements. Differential privacy has been applied to protect the user's privacy in several domains, including smart meters, sequential data publication, generative neural networks, and data storage in Bloom filters.

The **VALDA** team studies both foundational and systems aspects of complex data management, especially human-centric data, in the context of personal clouds. They focus on personal clouds under the angle of data integration and service orchestration.

The **WIDE** team works on the use of gossip protocols for privacy preserving decentralized computations and seeks to extend this work to the protection of privacy for decentralized machine learning systems.
## 5.3 Privacy analysis of existing systems

### [Summary]

Our connected world is at the origin of many privacy leaks, and as time passes, domains that are today leak-free will also be impacted. Some of the leaks are deliberate. This is the case with social networks where users massively share (sometimes overshare) personal data. The consequences are numerous for the user's privacy. For instance, the profiling it makes possible enabled the rise of dedicated companies that commercialized services meant to influence users (e.g., their votes).

Geolocation information is another type of information that is shared with either the tacit or the informed consent of the user. However, the record of a user's location over time is particular in the sense that a lot can be inferred, from home and work locations to sensitive personal information (e.g., religion if she regularly goes to a place of worship). In order to benefit from geolocalized services without leaking too much information, several solutions have been designed. For instance with spatial obfuscation, the position precision is reduced by reporting a zone. Geo-indistinguishability, which leverages on differential privacy, is a promising technique to achieve spatial indistinguishability with additional good properties.

The user can also be invited to provide biometric features, such as fingerprints, in order to permit their identification and authentication, in case of access control systems or secure national identity documents. However, these highly discriminating and stable features (they cannot be changed) create major security and privacy risks. Biometric systems must be carefully designed and, in order to investigate design options, various frameworks have been developed to define privacy architectures, to formally reason about them and to justify design choices in terms of trust assumptions.

But very often, privacy leaks happen without the user's consent. This is the case when browsing the web. Every visit to a web site can trigger a wide variety of hidden data exchanges across multiple tracking companies. Information can then be used for targeted advertising, but also to discriminate users (e.g., through customized prices) or worse. Several privacy solutions have been proposed, from protective regulations (e.g., with the GDPR) to client-side mechanisms like advertisement and tracker blocking tools. However, the domain is in constant evolution, new techniques appearing to better protect users and, simultaneously, to improve tracking within a browser or across devices.

With the advent of smartphones and IoT, privacy leaks have reached an unprecedented level in volume and precision, both within the digital and physical worlds, and often without the user's knowledge. This trend will be reinforced in the coming years, encompassing new domains. The goals of research in this domain are to analyze these systems, to give transparent information of hidden behaviors, to highlight good and bad practices, to propose methods likely to improve transparency and user control, and to encourage certain stakeholders to change practices.
As a common denominator of our connected world the Internet can also be the source of hidden privacy leaks. A first example is the wireless access network used by most devices. We have seen the rapid growth of cyberphysical tracking systems that analyze the Wi-Fi frames sent by a smartphone looking for a known access point. Research is needed to analyze these technologies and propose privacy preserving versions whenever possible. Another example is the DNS service used to map host names to IP addresses. A motivated attacker could eavesdrop DNS traffic (under certain circumstances) and potentially identify websites of interest. Unfortunately, the road is long before a privacy friendly version of DNS is standardized and deployed.

Finally, the safe browsing techniques used to detect blacklisted URLs (e.g., known to contain a malware) are potentially intrusive: storing the whole list of blacklisted URLs on the host is impossible and relying on an external service too intrusive since this latter would collect all the visited URLs. Intermediate solutions are used that need to be carefully analyzed from a privacy point of view.

This section focuses on privacy leaks in various existing systems. It starts with leaks for which there is either a tacit or informed consent of the user and then presents data collections that take place without the user knowledge.

### 5.3.1 The visible side: the case of social networks

Social networks are places where users are incited to share personal data massively. Very often, personal information, and even sensitive personal information, is shared to a group of people that is significantly larger than what the user would think. Several reasons can account for this situation. First of all, the “Privacy Paradox” is a well-known phenomenon whereby users explain they are concerned by their online privacy but at the same time behave in the opposite way. Many users may not realize that the risks apply to them, particularly the youngest, for whom the risks are either theoretical or for those “who have something to hide”. Or perhaps they consider that the reward obtained by sharing personal information is superior to the risks. Second, users may be too confident in the protection offered by the social network, omitting the small detail that significantly enlarges the group of people with whom information is shared. Third, the default settings and behaviors with respect to third parties may also be much more permissive than one could expect (e.g., Facebook changed default audience to “Friends only” in 2014, before that posts were “Public” by default). And finally, mistakes are always possible (e.g., the audience setup in Facebook posts is sticky, meaning that if a post is tagged for public audience, so will all the following ones until the user changes the audience back to the previous setting).

The consequences are numerous: identity theft, social pressure on users (in particular teenagers), sexual predation, unexpected consequences of shared
information (e.g., to a potential employer), targeted surveillance, or at the other extreme massive surveillance of citizens. Users are generally unaware of the precision of the story they tell about themselves. On 2009, *Le Tigre* published the life of Marc L*<fnref n="8" />, the story of several years of this person's life, solely collected from various social networks. More recently, information extracted from social networks has been used by the *Cambridge Analytica* company to profile users and influence their vote via personalized messages<fnref n="9" />.

Research is needed to better understand the tendencies, but also protect the users, in a context where data mining and machine learning have become extremely powerful tools.

### 5.3.2 The visible side: the case of geolocation information

Geolocation information is another type of information that is shared with either tacit or informed consent of the user. However, collecting the geolocation of a person over time raises major risks: additional personal data can easily be inferred (e.g., their home and work locations, or their habits), but also Sensitive Personal Information (e.g., health concerns if they go to an hospital or religion if they regularly go to a place of worship). Protection is therefore needed, for instance in order to benefit from geolocalized services without leaking too much information.

<figure><img src="image_23.png" /><figcaption>Geolocalisation anonymization — © Inria / Photo H. Raguet</figcaption></figure>

<footnote n="8">8. http://www.le-tigre.net/Marc-L.html</footnote>

<footnote n="9">9. https://ca-political.com/casestudies/casestudydonaldjtrumpforpresident2016</footnote>
The main methods for the privacy protection of location data are divided in two classes: spatial cloaking and spatial obfuscation. In spatial cloaking, the goal is trace-anonymity in order to prevent the identification of an individual. Most of the methods of this class are based on group-anonymity, a very popular approach in the anonymity literature. The general idea is to make the traces of an individual indistinguishable from those of other individuals; this is typically achieved by reporting a cloaked area that is large enough to contain the group size necessary to meet the intended anonymity constraint. In order to limit the size of the cloaked area, some proposals have combined spatial cloaking with temporal cloaking as well. When the system uses pseudonyms, a necessity for certain applications, there is the risk of linkability between points belonging to the same user's trajectory. To solve this problem researchers have proposed the so-called mix-zones, which are zones in which many users meet and can get their pseudonym renewed, without the danger of being traced. All the above measures related to spatial cloaking need, of course, the intervention of a trusted party that acts as an *anonymity server*.

In the second class, spatial obfuscation, the goal is to address the problem of identifying the user's position. In general, privacy is preserved by reducing the position precision. This is done by reducing the granularity of the location information: the user reports a zone rather than the exact coordinates. An important advantage of this approach is that it can be done without the intervention of a trusted third party. This method however is not very robust, being subject to trilateration attacks: a user sending two consecutive signals from two different zones reveals that they are close to the border between them and with three consecutive signals from different zones would reveal their position quite accurately. People have therefore investigated more effective solutions for spatial obfuscation.

Geo-indistinguishability [ABCP13] is one of them. It extends local differential privacy to arbitrary metrics with the idea that the protection of the user's location increases exponentially as the distance from the real location decreases. Therefore, an attacker can determine that the user is in Paris rather than London, and be reasonably confident that they are in the Quartier Latin, but cannot tell where exactly in the Quartier Latin. Geo-indistinguishability inherits appealing properties: it is independent from the side knowledge of the adversary, it is robust with respect to composition, and it does not rely on any trusted third party. It can be implemented at the user's end simply by adding noise to the real location. A low-cost planar Laplace function can be chosen for that, which enables its use in computationally limited devices such as smartphones. Thanks to the above properties, geo-indistinguishability via the Laplace mechanism has been adopted in several tools for location privacy (e.g., LP-Guardian, LP-Doctor, and SpatialVision QGIS plugin).

<footnote>[ABCP13] Miguel E. Andres, Nicolas E. Bordenabe, Konstantinos Chatzikokolakis, and Catuscia Palamidessi. Geo-indistinguishability: Differential privacy for location-based systems. In *Proceedings of the 2013 ACM SIGSAC Conference on Computer, Communications Security*, CCS '13, pages 901-914, New York, NY, USA, 2013.</footnote>
### 5.3.3 The visible side: the case of biometry

Biometrics is a powerful technology to identify or to authenticate a person. Biometric features, such as fingerprints or iris, are stable over time and highly discriminating; these are key advantages for applications such as security or access control. However, from a privacy point of view, these advantages turn into drawbacks: because of their stability over time and because an individual cannot easily change their biometrics, the leak of biometric traits to a malicious entity gives rise to serious privacy risks, including tracking and identity theft.

Many techniques (like encryption, homomorphic encryption, or secure multi-party computation) and architectures have been proposed to take into account privacy requirements in the implementation of privacy preserving biometric systems. Some solutions involve dedicated cryptographic primitives such as secure sketches and fuzzy vaults, others rely on adaptations of existing cryptographic tools or the use of secure hardware solutions. The choice of particular techniques and the role of the components (like the central server, a secure module, terminal, or smart card) in the architecture have a strong impact on the privacy guarantees provided.

Considering the variety of options available and the complexity of these techniques, general frameworks have been proposed to define privacy architectures, to specify different options, to reason about them in a formal way and to justify their design in terms of trust assumptions.

Biometric systems are also increasingly used by states to secure identity documents. For example, the French government authorized in October 2016 the creation of a centralized file of "secure electronic documents" (TES). The main motivation put forward was the fight against identity fraud. However, the decree also authorized access to the database by a variety of police and officers. Several criticisms have been voiced concerning the risks that such a centralized file could pose to individual freedom and privacy. The strengthening of the means to fight fraud and criminality, and the requirement to protect privacy should not necessarily be incompatible. However, in order to be able to reach a decision on the benefits and weaknesses of a secure electronic document system, it is necessary to [CM17]:

* clearly define the desired functionalities and the advantages that can be expected from them;

* describe the technical solutions in a sufficiently precise way to enable their analysis;

* and rigorously analyze the risks of privacy breaches with regard to the expected benefits.

<footnote>[CM17] C. Castelluccia and D. Le Metayer. Titres electroniques securises: la centralisation des données biométriques est-elle vraiment inévitable? Note d'analyse Inria, Février 2017.</footnote>
### 5.3.4 Hidden privacy leaks: the case of web tracking

The massive deployment of the Internet has quickly been accompanied with personal data leaks. Every visit to a web site can trigger a wide variety of hidden data exchanges across multiple tracking companies that each collect large amounts of data on users' preferences and habits. Information can then be used for targeted advertising, but also to discriminate against users (e.g., through customized prices) or surveillance.

Web tracking has been made possible thanks to the discreet addition of small components, called trackers, to webpages. Each tracker is owned by a third party, usually distinct from the web site owner, and it enables this third party to recognize users across the different websites that embed it. These technologies are roughly divided into stateful and stateless types. Stateful tracking techniques store information on the user's computer that can be retrieved later to recognize them. Third-party cookies are the most prevalent stateful online tracking technique. Advanced features, such as the ability to spawn cookies deleted by a user or the synchronization of cookies among different third parties, are often implemented in order to map and exchange user's profiles.

On the other hand, stateless tracking techniques allow third parties to recognize users, just by *fingerprinting* without storing anything. Collecting various pieces of information about the user's browser and operating system is sufficient to uniquely identify each browser. In 2010, Eckersley first demonstrated that the technology was highly effective through the Panopticlick project<fnref n="10" />. Recent works have demonstrated that today fingerprinting is as effective on mobile devices as it is on computers, in particular thanks to recent web technologies (e.g., HTML5 brought highly discriminating attributes) or websites where a user is connected.

These practices, source of major privacy leaks, raise two complementary and complex questions: how to detect them and how to protect oneself?

With stateful web tracking, explicitly blocking third-party cookies within the browser configuration or adding ad blocker extensions help a lot. Although not perfect, these techniques correctly detect and disable the most common and intrusive trackers [MHB+17].

Concerning protection against stateless web tracking (fingerprinting), disabling JavaScript is efficient but not practical. The automatic tracker detection question is complex and there is no precise methodology today. Most detection methodologies ignore many trackers because they only check access to fingerprinting-related APIs in the browser or apply very basic static analysis. Language-based security techniques can be used to go further: a monitoring is put in place to analyze how

<footnote n="10">10. https://panopticlick.eff.org/
[MHB+17] Georg Merzdovnik, Markus Huber, Damjan Buhov, Nick Nikiforakis, Sebastian Neuner, Martin Schmiedecker, and Edgar R. Weippl. Block me if you can: A large-scale study of tracker-blocking tools. In *2017 IEEE European Symposium on Security and Privacy*, EuroS&P 2017, Paris, France, April 26-28, 2017, pages 319{333, 2017}.</footnote>
much information is actually leaked from fingerprinting-related APIs and whether
this information is sufficient to uniquely identify a user. Another approach consists
in adding browser diversification at the virtual machine and API levels: by intro-
ducing enough noise during the fingerprinting process, trackers can be fooled.

The previous protection techniques were at the client initiative. Website owners may also be interested in protecting their users from web tracking (e.g., the GDPR makes website owners liable for the third-party tracking present on their websites). This is the goal of newly proposed architectures where additional servers automatically intercept and modify web requests thereby preventing third-party tracking.

Finally, it should be noted that the business model of web sites is often based
on advertisements and blindly blocking everything harms content providers.
Hence the question of user control and responsible choice: rather than trying
to block all trackers, authorize some of them, for instance those present on web
pages considered as less sensitive.

5.3.5 Hidden privacy leaks: the smart world

Beyond web tracking, the advent of intelligent and connected devices considerably expended the opportunities to collect personal data, both in volume and precision, encompassing domains that were up to now out of reach.

Smartphones have played a key role from this point of view. These personal assistants, easily personalized with applications, always connected, equipped with a large variety of high precision sensors, contain and generate a lot of information about our activities and centers of interest, both in the cyber world (Internet) and the physical world. Smartphones therefore have become ideal targets. The smartphone ecosystem consists of many actors, from application developers to advertisers. However, personal data collection is mainly orchestrated by the Advertising and Analytics (A&A) companies<fnref n="11" />, also known as “third parties.” In a world where free applications represent a large part of the offer, the A&A companies have developed small tracking software that developers are invited to integrate within their applications in order to monetize them. These trackers collect personal data from several billions application sessions every day and send them to the A&A companies in order to create user profiles whose accuracy keeps increasing every day.

Arguably, collecting personal data in exchange for free applications or
services could be acceptable if this collection was documented in a privacy
policy notice, respectful of the laws of the country where the user resides, and
if it was implemented in a "privacy by design" manner with data minimization
and accountability guarantees. But many studies report that the opposite is
taking place. A&A companies tend to collect as much personal information as

<footnote n="11">11. http://www.mobyaffiliates.com/guides/mobile-advertising-companies/</footnote>
technically possible<fnref n="12" />. However, major differences exist between the various operating systems (OS), depending on the decisions taken by the OS editor with respect to user tracking possibilities and user control (e.g., in 2013 Apple has been the first company to ban the access to stable identifiers and to replace them by a dedicated Advertising Identifier, under full user control).

Today the craze for "quantified self" wearable devices, smart home appliances, smart cities, and connected cars, or more generally "Connected Devices," enables the collection of personal data in new domains<fnref n="13" />. This is all the more worrisome as part of data being collected is Sensitive Personal Information and many connected devices remain highly vulnerable.

If the business model of companies selling connected devices is different from those of smartphone application developers, little is known about the actual practices in terms of confidentiality and data exchanges. Also, the privacy notice that should be provided with each connected device in order to inform the user is often missing or is unreadable for non-lawyers. The end-user is therefore prisoner of a highly asymmetric system.

Academic work is needed in these domains to understand the various facets of the problem, including the underlying business models. By giving transparent information of hidden behaviors, by highlighting good and bad practices, by proposing methods to improve transparency and user control, the ultimate goals of these works are to reduce the information asymmetry of the system, to empower users, and hopefully to encourage certain stakeholders to change practices.

### 5.3.6 Hidden privacy leaks: the case of the Internet

The Internet, as a complex assembly of diverse technologies, also features privacy risks for the users. This section discusses risks associated to the access network, to the Domain Name System (DNS) that is central to almost every activity on Internet, and finally to phishing protection services that could be turned into privacy intrusive tools.

#### WIRELESS ACCESS NETWORKS

In wireless access networks the traffic carried over the wireless link is in general protected by security mechanisms like WPA in IEEE 802.11 (Wi-Fi) networks. However, the headers and the content of management frames is not protected and is thus available to snoopers. The exposure of this information poses serious privacy threats that are made critical by the massive adoption of portable devices and the development of wireless networks.

<footnote n="12">12. For instance the InMobi company has been condemned in 2016 by FTC (https://www.ftc.gov/news-events/press-releases/2016/06/mobile-advertising-network-inmobi-settles-ftc-charges-it-tracked) because of their unfair misuse of the ACCESS WIFI STATE Android permission to track the users' geolocation.</footnote>

<footnote n="13">13. https://www.cnil.fr/fr/enceintes-intelligentes-des-assistants-vocaux-connectes-votre-vie-privee</footnote>
<figure><img src="image_24.png" /><figcaption>Picking up IoT waves – © Inria / Photo C. Morel</figcaption></figure>

More precisely, Wi-Fi enabled devices scan for nearby access points by sending probe requests. These probe requests can include the name (SSID) of the network to which the device has been associated in the past. In that case, the SSIDs emitted by a device reveal a lot of personal data, like travel history and identity, and one can also infer social links between users. Another aspect with 802.11 frames is the MAC address, a globally unique identifier tied to the device. Using this identifier, it is possible to detect the presence of people and track them in the physical world.

The opportunity of tracking smartphones users has been seized by corporations, leading to the rapid growth of cyberphysical tracking systems. For instance, they are deployed in commercial centers to measure and analyze client movements and habits, and attempts have been made to deploy such technologies within public equipment in order to display targeted advertising. Measures taken by these corporations supposedly to anonymize data and reduce privacy risks have proven to have a limited impact.

Solutions to enable privacy preserving analytics for this kind of applications are thus needed. A promising approach is to use probabilistic data structures based on Bloom-Filters that include a perturbation mechanism to enforce strong privacy guarantees while allowing the accurate estimation of the number of detected device identifiers. In response to tracking issues, Wi-Fi vendors started
implementing MAC address randomization, a technique in which a random and
temporary pseudonym replaces the real MAC address in the IEEE 802.11 frames.
Despite the adoption of this Privacy Enhancing Technology, studies have shown
that it may still be possible to track users through active attacks that force
devices to reveal their real MAC address and through fingerprinting attacks based
on content and probe request timings that could be used to single-out devices.

A lot remains to be done for Wi-Fi and similar wireless technologies in order
to reduce the privacy risks associated with their use; this remains an active
research topic.

CORE INTERNET SERVICES

Beyond the access networks, the DNS protocol, central to the Internet (see §2.2), does feature privacy threats that remain unsolved. The progressive transition to the "DNS Security Extensions" (DNSSEC) version solves some of the related security threats, however it does not address confidentiality requirements. For instance, even in the presence of DNSSEC, an eavesdropper present between a client and its DNS resolver will be able to analyze DNS traffic and identify most of the visited web sites, even in case of encrypted HTTPS web traffic<fnref n="14" />. This leaked information is considered as personal data (it is associated to a physical person) and tells a lot on the user centers of interest. Other risks are discussed in [Bor15].

Several directions are under consideration in the context of the “DNS PRIVate
Exchange” (DPRIVE) IETF working group¹⁵, leveraging on encrypted communications
(e.g. over TLS or DTLS). Research is still needed as practical considerations make
the real-world situation more complex than it may seem.

MALICIOUS ACTIVITIES DETECTION SYSTEMS

Browsing the web can lead users to visit malicious websites. The Safe Browsing techniques have been setup in order to detect blacklisted URLs (e.g, websites known to be involved in phishing attacks or to contain malware). Although very useful to the end-user, these services are also potentially dangerous from a privacy point of view. Storing the whole list of blacklisted URLs on the host is impossible, and relying on an external service to check visited URLs is too intrusive (the service provider would be in position to collect all the URLs visited by a user). Therefore, intermediate solutions are used in practice, and there is a need to carefully analyze them from a privacy view point.

<footnote n="14">14. 4DNS traffic between clients and their DNS servers traditionally goes through UDP communications, incompatible with TLS protection.
[Bor15] Stephane Bortzmeyer. DNS privacy considerations. RFC, 7626, 2015.
https://tools.ietf.org/html/rfc7626
15. https://datatracker.ietf.org/wg/dprime/about/</footnote>
The Google Safe Browsing service, in turn reused by most web browsers, has been studied from the privacy perspective. Although the company probably did their best to anonymize data and be privacy compliant, this Google service lacks transparency and accountability. More generally opening and discussing their technology with independent, trusted third parties would be highly beneficial.

[Research challenge 9] Towards a privacy preserving smart connected world

Our connected world experiences an unprecedented growth in terms of personal data collection, with practices that are increasingly intrusive for the citizen's intimacy. Surfing the web, using smartphones and other smart devices, driving a connected –and soon to be autonomous– car are activities that generate personal data leaks. The lack of transparency (many services and devices behave as black boxes) and lack of user control (how to express consent or opposition when there is no information, nor user interface) are major issues.

Identification of such hidden behavior is hindered by the number and complexity of underlying technologies specific to each domain. For instance, identification of tracking practices in a web page requires advanced JavaScript execution analyzes, while monitoring of smartphone applications needs dedicated frameworks, and monitoring of certain wireless communication technologies remains mostly unsolved. The analysis of these data flows is required to assess potential privacy leaks, e.g., in a smart home. Such challenging and diverse research activities are essential to bring transparency, highlight good and bad practices, and enable regulators to enforce data protection laws. As such, this research directly helps in the shaping of our future smart connected world.

[Inria teams] Privacy analysis of existing systems

The CIDRE team works on user control in the context of social networks. The team has proposed a classification of existing social networks based on the type of implementation (centralized versus distributed) of their functionalities (e.g., communication, search, or storage).

The team also worked on location privacy, producing the GEPETO tool<sup>a</sup> whose goal is to enable a user to design, tune, experiment, and evaluate various sanitization algorithms and inference attacks and evaluate the resulting trade-offs between privacy and utility.

the COMETE team works on location privacy, proposing the notion of geo-indistinguishability [ABCP13], that extends differential privacy to distance metrics: the precise location of a device is protected through the addition of controlled noise to the reported position. The team has also developed a tool based on geo-indistinguishability, called Location Guard<sup>b</sup>, a browser extension that allows to protect the user's location while accessing location-aware websites, by adding controlled noise to it.
[-] the **DIVERSE** team works on web browser fingerprinting. The *Am I unique* website demonstrates how the situation evolved with the most recent web technologies (e.g., HTML5 brought highly discriminating attributes) and shows that fingerprinting is today as effective on mobile devices as it is on computers, albeit for different reasons. In order to protect against fingerprinting, the team also works on browser diversification, with the FPRandom proof of concept⁴.

[-] the **INDES** team works on analyzing various forms of web tracking in web applications. In a joint work with **PRIVATICS**, the team also demonstrated that web browsers can be fingerprinted through the extensions⁵ that the user installs and the websites where they are logged. Finally, the team proposes a server-side technique to protect against web tracking: two additional servers respectively rewrite and redirect the original web application requests to prevent third-party tracking automatically⁶.

[-] the **PESTO** team works on privacy in online social networks, with the goal to inform users of latent information that can be inferred from their published information.

[-] the **PRIVATICS** team works on privacy threats introduced by the information society, in particular for biometry, web tracking, smartphones, smart world (IoT), wireless access networks, or malicious website detection systems. The goals are to understand the situation, analyze the threats, and if applicable design privacy preserving solutions to prevent or mitigate them.

For instance, the team proposed a general framework for the specification and formal reasoning of biometric architectures, and applied it to several architectures for biometric access control. The team also contributed in [CM17] to the secure electronic document systems (TES) debate. Through the MyTrackingChoices project, the team proposed a new type of ad blocker that lets the user choose which website categories can track them or not. The team also worked on the subject of smartphone and smart world (IoT) privacy, following a multidisciplinary approach with lawyers and economists, and on the questions of transparency and user control inside a smart city. Finally, the team pioneered the problem of privacy leaks within wireless access networks and developed the Wifiscanner⁸ and Wombat⁹ Wi-Fi scanning/tracking tools.

[-] the **SPIRALS** team works on analyzing web browser fingerprints, in particular its inconsistency and evolution over time.

<footnote>a. https://gforge.inria.fr/projects/gepeto/</footnote>

<footnote>b. https://github.com/chatziko/location-guard</footnote>

<footnote>c. https://amiunique.org/</footnote>

<footnote>d. https://github.com/plaperdr/fprandom</footnote>

<footnote>e. https://extensions.inrialpes.fr/</footnote>

<footnote>f. https://www-sop.inria.fr/members/Doliere.Some/essos/</footnote>

<footnote>g. https://github.com/cunchem/gtk-wifiscanner</footnote>

<footnote>h. https://github.com/Perdu/wombat</footnote>
6

Critical infrastructures,
systems, and applications:
use-cases for security

<figure><img src="image_25.png" /></figure>
In this chapter we look at cybersecurity under the angle of applications. We focus on a few selected domains where cybersecurity is likely to have a strong impact. We selected in particular security sensitive domains:

* infrastructures: the Cloud, Software-Defined Networks (SDN), as well as the blockchain;

* critical and cyber-physical systems: the Internet of Things (IoT) and industrial systems;

* application areas: medicine, robotics (including connected autonomous vehicles), and machine learning (ML).

This list is of course not exhaustive, but it illustrates well that many systems and application areas are security critical and that security considerations must necessarily be included in the design process. The categorization into critical infrastructures, critical systems, and critical applications areas is also not absolute — for instance, medical implants are cyber-physical systems — and critical applications may of course build on critical infrastructures or systems.

## 6.1 Critical infrastructures

Communication infrastructures are primarily designed to offer a service, often with usability and efficiency as their primary goals. However, security is also crucial as these infrastructures may be used for storing and manipulating sensitive data. Loss of availability (or simply efficiency) and integrity of data may also have strong economic impact. We will focus on three examples of such critical infrastructures: the Cloud, software-defined networks, and the blockchain.

### 6.1.1 Security and privacy in the Cloud

**[Summary]**

Security remains an obstacle to a wider adoption of Cloud services by organizations for their critical services. Privacy has also become an important issue for Cloud users in the IoT era. Hence, a major challenge for Cloud providers is to provide their users with both security services with associated guarantees in SLAs and privacy preserving computing and data storage services. Security and privacy issues are exacerbated in the context of distributed Clouds, AKA. edge and fog computing. Trust and other security assumptions change over time owing to changes in the technology and the discovery of new vulnerabilities. These evolutions have led necessarily to the appropriate evolution of the threat models in Cloud security considering not only attacks on the tenant operating system but also attacks on the hypervisor. Securing hypervisors and operating systems and proving security properties on them appear to be timely and important research topics.
<figure><img src="image_26.png" /><figcaption>Autonomous vehicles – © Inria / Photo G. Scagnelli</figcaption></figure>

Security is one of the main concerns in the adoption of the Cloud model<fnref n="1" />. The hardware and low-level software infrastructure is owned and governed by the Cloud provider while customers outsourcing their information system have the control on the services and the OS running in their virtualized infrastructures. Server virtualization enables the execution of different OS's and/or applications from different tenants on the same server in a datacenter. This multi-tenancy situation entails specific security and privacy threats.

Cloud environments face multiple security threats originating from different privilege levels (application, network, and operating system levels) in the infrastructure. In an IaaS (Infrastructure as a Service) Cloud environment, the attack surface is expanded with the addition of the highly privileged hypervisor, as the building block of a Cloud infrastructure, as well as its web-exposed management API. In such a context, security concerns two different actors: tenants, and providers.

Tenants are concerned with the security of their outsourced assets, especially if they are exposed to the Internet. Attacks targeting traditional information systems could also target applications running inside virtual machines in an outsourced infrastructure.

The provider is also concerned about the security of the underlying infrastructure especially since it has no insight regarding the hosted applications and their workload. In a Cloud environment, security threats originating from corrupted tenants against other legitimate tenants and their resources, threats against the

<footnote n="1">1. http://www.infosecbuddy.com/download-cloud-security-report</footnote>
provider's infrastructure as well as threats towards the provider's API should be considered. Cloud providers need to assess security risks taking into consideration the hosted virtualized infrastructures and vulnerabilities in virtualization technologies. They need to know which virtual machines can interact via network protocols.

In IaaS Clouds, the Cloud provider manages the Cloud's security monitoring infrastructure while Cloud tenants manage their outsourced information system. Security monitoring is essential in Clouds. A specific challenge is to ensure that the security monitoring infrastructure is automatically reconfigured when Cloud dynamic events (e.g., VM migration) happen. Tenants are incited to trust the provider's claim (e.g., infrastructure availability) thanks to the assurance given by Service Level Agreements (SLA's), making a trade-off between tenant's disclosed private information and the monitoring service offered.

Although multi-tenancy maximizes efficiency for the Cloud provider's resources, it also offers the possibility that a tenant's VM can be located in the same physical machine as a malicious VM. This in turn engenders a new threat: breaking the resource isolation provided by the hypervisor and the hardware and gaining access to unauthorized data or disturbing the operation of legitimate VMs. One of the most prominent attacks that illustrates this threat is the side channel attack where an adversary with a collocated VM gains access to information belonging to other VMs (e.g. passwords, cryptographic keys). For example, the attackers could use shared CPU caches as side channels in order to extract sensitive information from a collocated VM.

Cloud providers are usually assumed to be "honest but curious": given their privileged position with respect to their tenants either a malicious provider or a provider whose system becomes compromised could threaten their tenants' privacy. Virtual machine introspection mechanisms can be used by the Cloud provider to monitor the tenants' virtual infrastructure. Outsourcing data in the Cloud means delegating control over data to the provider. Cloud customers have little to no control on where or for how long data is stored and to which third parties it is forwarded. Moreover, the Cloud provider may provision resources located in different countries with different data regulations resulting in data protection depending in where data is stored and this is often transparent to Cloud customers.
## [Inria teams] Security and privacy in the Cloud

The **AVALON** team is interested in the modeling of application security properties and their automatic enforcement for applications executed in Clouds. They proposed a specification-driven approach where the security is expressed as properties in a mechanism agnostic language to facilitate the expression of the user's requirements for both the application and its security resulting in the Sam4C toolbox. They also developed mechanisms to provide automatic application deployment and automatic enforcement of its security, with proven properties.

The **CASCADE** team works on privacy for the Cloud, designing a new generation of secure multi-party computation protocols enabling computation on encrypted data.

The **CIDRE** team investigates security assessment in Clouds considering virtualization technologies, hypervisors and SDN, in connectivity extraction in Cloud infrastructures and Cloud-specific vulnerabilities in attack graph generation.

The **MYRIADS** team aims at integrating security monitoring terms in IaaS Clouds' SLA's and designing a self-adaptable Cloud security monitoring infrastructure. The team has designed one such framework that is able to alter the configuration of its components and adapt the amount of computational resources available to them depending on the type of dynamic event that occurs in a Cloud infrastructure.

The **STACK** team has proposed a compositional approach to the declarative and correct composition of privacy-preserving applications in the Cloud. The proposed approach provides language support for the composition of three techniques: symmetric cipher, vertical data fragmentation, and client-side computations to make Cloud applications' privacy preserving. The team also studies isolation threats in containerized environments investigating side-channel attacks in the context of edge Clouds.

## 6.1.2 Security of Software-Defined Networks (SDN)

### [Summary]

The softwareization of networks is the ongoing evolution in the networking field. The goal is to enhance flexibility and reduce costs. However, these evolutions centralize the control of a network, providing a single point of failure for the attacker to target. Moreover, the network softwareization enables a better coupling between the network and applications thanks to diverse API's. While previously restricted to a limited number of actors, these API's significantly ease the development of applications, but also create a new attack surface.

Major evolutions have recently occurred in networking technologies. A shift towards the softwareization of networks deeply changed the network architectures and operations. The most notable emerging paradigms are SDN (Software-Defined
Networking) and NFV (Network Function Virtualization). SDN advocates a logically centralized and powerful controller to replace distributed algorithms (the former panacea) leaving only forwarding purposes to network devices. NFV enables any kind of network function to be run as a virtual machine, supposing some central controller or management plane to orchestrate the function's deployment. It is thus well aligned with the cloudification of everything. Although core networks will continue to rely on high-end specific network hardware, the softwarization of networks naturally relies on standard servers in a datacenter-like architecture. The final objective is to enhance flexibility in network operations and to reduce the costs substantially. However, there are inherent security-oriented challenges related to this paradigm shift.

Centralizing the control of a network reduces the attacker's focus and increases vulnerability: the attacker may efficiently disrupt a network by compromising a central controller, while distributed algorithms are more robust against individual attacks. Targeted denial-of-service against a single controller may provoke a severe impact. New technologies, new frameworks, and new protocols come along with these new paradigms to integrate as many functionalities as possible, thus increasing the attack surface. Guaranteeing the security and detecting the misuse of these protocols is of paramount importance.

In addition, the softwarization of networks will enable a better coupling between the network and applications thanks to diverse APIs. Development of applications will no longer be restrained to a very limited number of actors as is currently the case with hardware appliances produced by very few manufacturers. Strict quality control will need to meet the market's usability and flexibility demands. Hence, runtime application behavior can mistakenly or intendedly lead to open security breaches in network. Moreover, such breaches may be the consequences of a set of applications that are only problematic when used co-jointly. Verifying policies and detecting configuration anomalies in softwarized networks must take into account that configurations are dynamic, using only static analysis is thus not appropriate. Service Function Chaining (SFC) consists in chaining multiple virtualized network functions (VNF) to deliver innovative services, including for instance security services (firewall, intrusion detection, deep-packet inspection, proxy, etc.); several chains could even share some VNF's.

NFV enables a higher flexibility and should make network functions more elastic. However, network functions can be collocated on the same physical machines. As the core idea is to take network functions off-the-shelf, knowing or predicting exact behavior is challenging. Moreover, expected short development and deployment cycles can be the source of dangerous behaviors, mistakenly or intendedly introduced. Checking and evaluating the output of a network function from a security perspective is essential. Several approaches can be leveraged: static code analysis if possible, dynamic analysis, or testing. Furthermore, isolation
between collocated network functions is primordial. Whereas VM isolation has been largely studied in the past in order to render sustainable the Cloud model, the game changes when it comes to virtualizing network functions. Indeed, achieving line-rate operations leads to relaxing strong isolation properties and thus can lead to security problems. A good trade-off between security and performance needs to be found and might be adjusted depending on the criticality of the VNFs. Detecting and preventing a misbehaving VNF (Virtual Network Function) that impacts the operation of a collocated VNF is necessary. Similarly, the VNF deployment may integrate security constraints rather than just being seen as a pure resource allocation problem.

## [Inria teams] Security of Software-Defined Networks

* The **COATI** team focuses on optimizing resources used by forwarding devices in SDN. The goal is to be able to store more forwarding rules in a compressed manner while maintaining acceptable performance and limited control overhead.

* The **DIANA** team contributes to the definition of new solutions for the SDN and NFV paradigms that take into account security considerations. In addition, the team evaluates the performance of related technologies and so, indirectly, their ability to resist to heavy loads created by denial-of-service attacks.

* The **RESIST** team works on the definition and testing of new network programming abstractions. Runtime and in-switch execution of VNF's need a new data plane abstraction. Since isolation between VNF's may not be guaranteed, it becomes necessary to introduce safeguards and methods to predict execution time and dependencies.

* The **RESIST** and **VERIDIS** teams both aim at checking SFC's by translating complex policies into a formal representation. Verification is needed to detect potential inconsistencies due to the distributed deployment of multiple SFCs.

## 6.1.3 Blockchain

### [Summary]

By proposing a trusted, append-only and immutable ledger with various models for writing and managing it (fully decentralized, permissionless, permissioned, consortium, etc.), blockchains enable many applications relying on this new security feature and its related infrastructures. Yet, their real security and level of trust need to be properly asserted with analysis both from the cryptography and distributed systems communities. Also, as with any Internet system, blockchains can suffer for instance from low-level network attacks, software bugs, or failures, that can appear when blockchains are used for higher-level applications (smart contracts). Additional features such as strong privacy and anonymity may also conflict with security requirements from public and legal bodies.
## A SECURITY PROMISE

The quite trendy and emerging blockchain technology, first put in place by the Bitcoin protocol, is a security promise. A blockchain, in any of its variants, implements a secure electronic ledger, a functionality similar to existing ledgers managed by banks, notaries, states, etc., providing the main security property that data registered in the ledger cannot be removed or modified. Indeed, the blockchain provides integrity of past history, relying on the notion of cryptographic hash functions: the hash of the last trusted block certifies the integrity of the whole ledger since its inception.

## SECURITY OF THE LEDGER

Such a ledger can be guaranteed by a single, centralized entity, trusted for certifying the last block of data on a regular basis [HS91] and for regularly publishing chained integrity checksums of the ledger. Decentralizing the role of certifying the blockchain is the main innovation of Bitcoin. The decentralization could follow a peer-to-peer model, with unregistered, unknown, and untrusted nodes (i.e., permissionless as in Bitcoin or Ethereum). The decentralization could also follow a permissioned model, where either some participants are known to have the role of signing the last block while controlling each other (e.g., Hyperledger, industrial blockchains), or participants are unknown stakeholders that are believed to certify the ledger honestly to protect their stake.

As a consequence, the security analysis of a blockchain depends on the social and political model underlying the blockchain: the Bitcoin blockchain can be attacked by a powerful adversary controlling 51% of the computing power of the Bitcoin network (Namecoin has exhibited this weakness [ANSF16]); consortium blockchains can be completely and quickly rewritten if signing keys are compromised; and blockchain with stakes encounter the risk a non rational stakeholder acting against their own financial interests.

## DYNAMICS OF THE PROTOCOL

Besides the integrity properties of the ledger, the problem of dynamically determining the current data block to be appended to the ledger brings into focus the domain of distributed algorithms, where issues of liveliness and correctness are central. Since the blockchain is replicated between participants, it is crucial that all have the same view of the blockchain: failing to do so allows an attacker to profit from the divergent views of the blockchain. While most blockchain protocols establish that the views are the same after a certain period of time, there is always a period of instability before the agreement is reached. Attacks

<footnote>[HS91] Stuart Haber and W. Scott Stornetta. How to time-stamp a digital document. *Journal of Cryptology*, 3(2):99-111, January 1991.</footnote>

<footnote>[ANSF16] Muneeb Ali, Jude Nelson, Ryan Shea, and Michael J. Freedman. Blockstack: A global naming and storage system secured by blockchains. In *2016 USENIX Annual Technical Conference (USENIX ATC 16)*, pages 181-194, Denver, CO, 2016. USENIX Association.</footnote>
can thus arise (for instance double spending) and non-cryptographic issues from
distributed algorithms need to be addressed.

NETWORK ATTACKS

Network issues are often not considered, but blockchains can suffer from
low-level network attacks. Their security cannot be solely guaranteed by the
cryptographic protocol itself. Recently, it has been proven that the mining power
of bitcoin networks is very unbalanced and that a network-level attack is possible
through the partitioning the overlay network (BGP hijacking) [AZV16]. No exhaustive
assessment of network-level attacks and their impact on the blockchain security
and performances has as of yet been realized. A threat assessment model needs to
be established and adapted to each blockchain technology. Security can be improved
by adding either security mechanisms by design (network-level requirements, such
as topology constraints) or counter-measures at runtime (coupling between the
network and the blockchain). First, a monitoring scheme is needed to assess the
security of a blockchain. Second, dynamic mechanisms such as blacklisting nodes
or strengthening the consensus model can be dynamically applied (programmable
networks, with flexible configurations will be helpful). Blockchain-specific network
policies could be defined, deployed, and verified automatically, for example assuming
cooperation between autonomous systems exchanging BGP route information.

SOFTWARE FAILURES

A blockchain system has to be implemented in software, as must be applications
using blockchains. These implementations can encounter the same problems as
any system, such as bugs, attacks, undefined behavior, and so on. These issues are
much more critical for blockchains, however, because their applications handle
currency and securities. This means any issue will have direct financial impact,
giving cyber-criminals strong economic incentive to find or create such issues.
Furthermore, applications must cope with notoriously error-prone features
such as concurrency, distribution, authority, and secrecy. While there have been
no major issues with the Bitcoin or Ethereum systems per se, both clients (the
MtGox exchange) and applications (the DAO) have suffered devastating breaches.

BLOCKCHAINS AS BUILDING BRICKS FOR HIGHER LEVEL PROTOCOLS

A blockchain can be used as a building block providing a ledger, on which
higher level programs and protocols can be implemented. For instance, Ethereum
envisions the blockchain as storage with integrity for data and programs, where
programs can be activated for writing and updating data on the blockchain,
while not erasing past versions. Similarly, Bitcoin has a weak, yet useful, notion

<footnote>[AZV16] Maria Apostolaki, Aviv Zohar, and Laurent Vanbever. Hijacking bitcoin: Large-scale network attacks on cryptocurrencies. CoRR, abs/1605.07524, 2016.</footnote>
of programmable money that allows for instance off-chain transactions and cross-chains atomic swaps. Even when considering the underlying blockchain as a perfect cryptographic ledger primitive, developing higher level programs and protocols has the same pitfalls as developing high level cryptographic protocols given simple basic blocks. Because privacy is lacking by default in permissionless blockchain, some efforts have been made to ensure privacy using zero-knowledge proofs and other advanced cryptography (Monero, Zcash). At this point, monitoring requirements or legal constraints, e.g., KYC (Know You Customer), are conflicting with the privacy provided by strong cryptography, but this conflict can sometimes be dealt with, depending on the blockchain model.

## [Inria teams] Blockchain

➤ The **ANTIQUE** team is conducting research on the semantics, abstraction, formal proof, and automatic analysis of systems, together with verification of security properties, on distributed data-structures and consensus protocols.

➤ The **AVIZ** team develops visualization tools and exploration methods for the interactive analysis of complex and large data sets, in particular the Bitcoin blockchain.

➤ The **CASCADE** team is doing cryptography with rigorous security proofs and has an expertise in protocols for e-cash (centralized and decentralized "cryptocurrencies") and privacy tools for blockchains.

➤ The **CIDRE** team is designing distributed agreement algorithms to improve security, performance, and scalability of permissionless blockchains by focusing on both the underlying peer-to-peer networks and the structuration of the information in blockchains.

➤ The **COAST** team is doing research on safe and secure peer to peer data sharing and service composition for collaborative environments without a central authority.

➤ The **DELYS** team does research on distributed systems, from theory to algorithms to implementations, including blockchain as a core mechanism where they work to improve its performance and scalability by reordering operations without compromising safety and by interoperation between blockchains.

➤ The **GRACE** team studies privacy and secure multiparty protocols and how they can be used in the context of blockchains, as well as coding theory for distributed storage.

➤ The **PROSECCO** team works on the analysis, design and implementation of security protocols based on blockchain technology, e.g., smart contracts, through the use of formal methods, software languages and tools.

➤ The **RESIST** team is designing novel approaches to monitor and configure the blockchain infrastructure (network, Clouds, etc.) in order to enhance or guarantee performance and security. In addition, the team also investigates blockchain-based management to address the evolution of the Internet ecosystem.

➤ The **SPECFUN** team does formal proofs of mathematical theories, as well as formal analysis of Ethereum environment and other blockchain enabled execution models.

➤ The **TOCCATA** team promotes formal specification and computer-assisted proof in the
development of software that requires a high assurance of its safety and correctness.

↗ The **VERIDIS** team studies mechanized verification techniques applied to concurrent and distributed algorithms and systems. It is interested in precise formulations of the algorithmic problems that arise in the context of blockchains and corresponding verification problems.

## 6.2 Critical and cyber-physical systems

Critical systems are systems that require very high reliability, because failure could have extremely harmful consequences, such as endangering life, causing severe damage to an infrastructure, or cause important economic loss. However, most safety-critical systems are also security critical and must resist a cyberattack.

A related concept is that of Cyber Physical System (or CPS for short). Wikipedia defines a CPS as a mechanism that is controlled or monitored by computer-based algorithms, tightly integrated with the Internet and its users. In cyber-physical systems, physical and software components are deeply intertwined, each operating on different spatial and temporal scales, exhibiting multiple and distinct behavioral modalities, and interacting with each other in a myriad of ways that change with context. CPS's are often part of critical systems, which must be secured. Composed of many interconnected subsystems, running different

<figure><img src="image_27.png" /><figcaption>Security of embedded systems (fuzzin) – © Inria / photo C. Morel</figcaption></figure>
protocols for communication at different scales, they offer a large attack surface. Besides, some of them are exposed targets because a successful attack, such as shutting down part of a state infrastructure, may have a huge economic and political impact. Hence, cybersecurity is now often an essential issue for CPS.

While cybersecurity of CPS may reuse traditional approaches, methods, and techniques for securing systems and networks, new approaches will also have to be developed to cope with dynamicity and to analyze and ensure their safety and security in this context. Interestingly, these might be inspired by reactive security, where monitoring and reaction to abnormal situations will take a more important place – and machine learning is likely to take an increasing important role that will, of course, introduce its own security issues.

### [Note] CPS vs. embedded systems

CPS generalizes the much older concept of embedded systems in many ways. An embedded system also implies an interface between the digital and the physical world and emphasizes the temporal issues, but it focuses on individual components rather than the whole system and the interactions between its many components—and, often, humans as well. For example, the ABS (Anti-lock Braking System) is an embedded system—and thus also a CPS, while a power plant controller is a CPS, involving many other subsystems. A large majority of computer devices in use today are actually parts of embedded systems or CPS.

## 6.2.1 Security of Internet of Things (IoT)

### [Summary]

The IoT revolution is extending the Internet at a rapid speed and is changing the way the world interacts with physical devices. Security and privacy considerations put this revolution at stake. IoT applications are not just yet another instantiation of previously known distributed systems: their specificities, in particular the resources-constrained nature of devices, present new challenges for security that need to be addressed and classical security solutions are not always applicable. Several research directions need to be explored, in particular: secure operating systems and firmware update capabilities, lightweight and hardware-assisted cryptography, wireless technologies security and privacy, IoT dedicated security and privacy policies, IoT intrusion detection methods, secure programming languages and compilers for IoT applications, and dedicated authentication protocols.

The Internet of Things (IoT) is the network of embedded physical devices, equipped with sensors, actuators, processing, storage, and connectivity that allows these things to connect and exchange data (Wikipedia). IoT applications belong to a wide variety of domains, from patient health (e.g., smart pacemakers),
quantified-self devices (e.g., activity tracking watches), to home appliances (e.g., smart thermostats, plugs, or bulbs), cars (e.g., vehicle-to-vehicle emergency systems), and industrial systems (e.g., SCADA). Because IoT enables a more direct integration of the physical world into computer-based systems, the IoT has been labeled<fnref n="2" /> the Fourth Industrial Revolution.

However, concerns about IoT security, or the lack thereof, and privacy, since personal and sometimes sensitive personal data is exchanged, put this revolution at stake. For instance, the Mirai botnet (§1.3) involved a large number of consumer devices (including vulnerable IP cameras) to launch massive Distributed DoS attacks. Similarly, users' privacy is at risk when companies treat security as an afterthought (as was the case with some Internet connected sex toys <fnref n="3" />).

A first IoT specificity is the resources-constrained nature of devices. Microcontrollers in IoT devices are architecturally very different from a typical PC in terms of storage and CPU capabilities<fnref n="4" />. Certain IoT devices or components will be stringently constrained not only by low energy consumption but also limited resources. These ultra-low-power and ultra-low-cost devices operate on batteries with limited capacity or even harvest energy from the surrounding environment (e.g., light, heat, or vibration).

These specificities led to the design of dedicated operating systems to handle different embedded architectures. Popular IoT operating systems include Contiki<fnref n="5" /> (initiated at the Swedish Institute of Computer Science), Mbed OS<fnref n="6" /> (ARM), RIOT<fnref n="7" /> (initiated by Inria, Freie University of Berlin, and Hamburg University of Applied Sciences), and Zephyr<fnref n="8" /> (Wind River Systems, Linux Foundation, Intel, NXP Semiconductors, et al.).

Another specificity is the heterogeneous nature a global IoT application. Covering the spectrum from microcontrollers to the Cloud, an IoT application potentially includes code that runs in web clients, on servers, as well as in embedded devices. IoT applications must handle a wide variety of asynchronous events: queries to distant services, answers, timeouts, and errors. In turn, each event may launch calculations that trigger a cascade of new events.

Bringing security and privacy to IoT is challenging, the attack surface being important and classical security solutions not always applicable. Among the research directions to be addressed, we can mention the following:

<footnote n="2">2. Source: the World Economic Forum</footnote>

<footnote n="3">3. https://blog.trendmicro.com/penetration-testing-researchers-effectively-hack-a-vibrator/</footnote>

<footnote n="4">4. For example, RAM and Flash storage are 106 times smaller in an Arduino Uno microcontroller and its processing power is around 16 MIPS (Million Instructions per Second) to be compared with a few 100,000s MIPS for recent desktop processors.</footnote>

<footnote n="5">5. http://contiki-os.org/</footnote>

<footnote n="6">6. https://www.mbed.com/en/platform/mbed-os/</footnote>

<footnote n="7">7. https://www.riot-os.org/</footnote>

<footnote n="8">8. https://www.zephyrproject.org/</footnote>
## SECURE SOFTWARE UPDATE CAPABILITIES

The secure deployment of software updates on IoT devices, and in particular the firmware itself, is a mandatory feature of any secure operating system. Unfortunately, this key feature is often compromised by the desire to keep device complexity and energy consumption as low as possible, or simply by bad practices. In addition to academic work, the Software Updates for Internet of Things (SUIT)<fnref n="9" /> IETF working group investigates this question, focusing on an architecture that is agnostic to the communication technologies and protocols (e.g., CoAP or HTTP).

## LIGHTWEIGHT CRYPTOGRAPHY AND HARDWARE-ASSISTED CRYPTOGRAPHY

Regarding the cryptographic building blocks, embedded device constraints can require the use of lightweight cryptographic primitives. This topic is discussed in §3.1.3.

IoT devices will also benefit from the design of specific hardware architectures and software optimizations of lightweight cryptography, including protection against side-channel attacks and fault injection. These objectives require an efficient platform based on:

* hardware accelerators (crypto-processors) for security functions (e.g., symmetric and asymmetric cryptography, hashing, authentication, signature, or random number generators), with a specific focus on energy efficiency and ultra-low-power;

* specialized crypto-processor designs, such as randomization, that protect against attacks;

* compiler optimizations targeting resource-constrained crypto-processors;

* hardware accelerated dynamic binary translation (DBT) as a mean to enhance software protection;

* and new techniques for efficient hardware protections against side-channel attacks and fault injection, both in software and hardware.

The global aim is to propose an energy-efficient crypto-processor with an instruction-set architecture that can be configured at runtime.

## WIRELESS TECHNOLOGIES SECURITY

IoT communications leverage both on standard wireless technologies (e.g., Wi-Fi and Bluetooth) and on low power technologies (e.g., Bluetooth Low Energy (BLE) and Zigbee), as well as low-power wide area networks in unlicensed bands (e.g., LoRa<fnref n="10" /> and SigFox<fnref n="11" />) and in licensed bands (the promising 5G that also aims at connecting IoT devices). Work is in progress with respect to the security and privacy risks (§5.3.6) associated to these communication technologies.

<footnote n="9">9. https://datatracker.ietf.org/wg/suit</footnote>

<footnote n="10">10. https://en.wikipedia.org/wiki/LoRa</footnote>

<footnote n="11">11. https://en.wikipedia.org/wiki/Sigfox</footnote>
## IOT DEDICATED SECURITY AND PRIVACY POLICIES

Another challenge is the design of IoT dedicated security policies that prevent attacks exploiting the exposure of the system to physical hazards or resulting in physical consequences.

## IOT INTRUSION DETECTION METHODS

Another challenge for IoT security is the design of IoT intrusion detection techniques to stop malware and other attacks, for instance based on the device activities in IoT networks (§4.4).

## SECURE PROGRAMMING LANGUAGES AND COMPILERS FOR IOT APPLICATIONS

Programming an IoT application is complex because of the mix of technologies involved and the languages used, and taking security and privacy into account adds another level of complexity. Several frameworks meant to simplify IoT development exist, however they are not always designed with security in mind and can present serious security risks. Therefore, there is a need to develop a secure and privacy-preserving programming language and framework for the IoT. The formal verification of IoT software, starting with its cryptographic library, is another possible research direction.

## DEDICATED AUTHENTICATION PROTOCOLS

Finally, the combination of resource-constrained devices and IoT heterogeneity requires novel authentication protocols to provide a simple, secure, and privacy-friendly way of seamlessly authenticating individuals and their smart devices online.

### [Inria teams] Security of Internet of Things

* The **ANTIQUE** team is working towards analyses based on abstract interpretation for IoT programs.

* The **CAIRN** team tackles the question of improving energy efficiency through the use of flexible hardware accelerators. It proposes to rely on morphable hardware, whose structure can be efficiently reconfigured at runtime, for example to specialize the hardware to the application it executes, adapt the accelerator parallelism for performance or resource usage. The team also explores hardware accelerated dynamic binary translation.

* The **CELTIQUE** team is working on the formalization of the Hop.js semantics in Coq as well as on formal verification of program analyses for the IoT.

* The **FUN** team is working on the verification of the Contiki system as well as on intrusion detection techniques for the IoT.
The **GRACE** team participated in the design of several targeted cryptographic primitives for microcontrollers, notably qDSA, a signature scheme for IoT, now included in the RIOT operating system.

The **INDES** team is working on the development of a secure compiler and on the enforcement of privacy policies for Hop.js, a programming language for the IoT.

The **INFINE** team leads the RIOTª open-source IoT operating system development along with the Freie University of Berlin and Hamburg Univ. of Applied Sciences. Work is under progress on supporting secure firmware update facilities for RIOT.

The **KAIROS** team is working on formal model-based co-design techniques for real-time and security aspects of the IoT.

The **PETRUS** team is working on adapted strategies to stock data in IoT objects instead of sending the data to the servers.

The **PRIVATICS** team is working on privacy considerations and privacy preserving systems for the IoT.

The **PROSECCO** team is building HACL*, a verified cryptographic library that is already included in RIOT and work is underway on a lightweight version for resource-constrained IoT devices.

a. https://www.riot-os.org/

## [ Research challenge 10 ] Securing the Internet of Things (IoT)

Security in IoT is a major challenge: attacks are still relatively easy (many devices have not been designed with security in mind), invasive (e.g., pervasive in our lives), and potentially with a major impact due to the multiplication factor made possible by the large number of devices available and to the direct implications some of them have in the physical world (e.g., connected cars). The research directions are manifold, with for instance the ability to securely update embedded devices' software, the design of lightweight cryptographic primitives adapted to limited resources, the analysis of the security of new low-power wide area wireless technologies, the detection and mitigation of intrusions or misbehaving devices, and the need for secure-by-design frameworks, protocols, and operating systems.
6.2.2 Security of industrial systems

[ Summary ]

Cybersecurity of Industrial Systems is an emerging topic and recent cyberattacks on industrial systems show that the problem is open. One of the main difficulties in dealing with industrial systems is that they fail the security-by-design principle: since they were not intended to be exposed on Internet at the origin, the protocols used are not secure; sometimes specifications are not publicly available; general-usage firewalls and intrusion detection devices are not handling industrial protocols. End devices are built with slow processors unable to use standard cryptographic protocols and thus require dedicated ones.

Industrial Control Systems (ICS) or Industrial Automation and Control Systems (IACS) include a large variety of industrial digital data acquisition, control, and monitoring systems together with their underling communication networks. Two system architectures are currently used: DCS (Distributed Control System) and SCADA (Supervisory Control and Data Acquisition), although SCADA tends to supersede DCS.

<figure><img src="image_28.png" /><figcaption>Understanding attacks on industrial control systems – © Inria / Photo C. Morel</figcaption></figure>

Cybersecurity of Industrial Control Systems is a relatively recent application field. Communications in traditional industrial systems were achieved through small size networks using proprietary protocols. They were submitted to hard real-time constraints and not interconnected with IT systems or the Internet. Therefore, traditional cybersecurity in industrial systems was achieved by
isolation and obscurity, using proprietary protocols. Modern and large-scale industrial systems such as power grids, nuclear plants, or hydraulic dams have global control optimization needs that require interconnexion with supervisory control applications, distributed data-base systems, and, consequently, long-range communication, standard data-exchange formats and interoperability. The paramount communication paradigm in the modern industrial systems is the convergence of the Information Technology (IT) and Operation Technology (OT), or said otherwise, the interfacing between TCP/IP networks and real-time proprietary protocols. The counterpart of IT pervasion in the OT area is that the OT is now exposed to cyberattacks in the same way as the IT. Moreover, as the OT still heavily relies on legacy protocols they are even more vulnerable. However, for almost a decade (1995-2003), the cybersecurity of the industrial systems was largely ignored.

The initiating event of the industrial systems cybersecurity research was the Northeast blackout of America in 2003. Although not the result of a cyberattack (for memory, the blackout was due to a wrongly calibrated sensor and a chain of IT bugs), this event demonstrated that a false data injection may exploit several vulnerabilities to shutdown 256 nuclear plants and blackout 55 million people in the US and Canada. Little progress was made until 2010 while the reality of the threat was not formally proved. After the occurrence of Stuxnet in 2010-2011 important research programs were started. Further events like the string of attacks on the Ukrainian power grid in 2015-2016 increased the importance of industrial cybersecurity programs.

The IT cybersecurity solutions do not apply directly, due to the specificities of the industrial systems. Below is a list of soft spots of the ICS communication.

* *Insecure and unsecurable protocols:* legacy protocols were not designed for security; moreover, they are also intended to be used with very low speed processors, which rules out the use of cryptographic protocols.

* *Unavailable specifications:* some legacy protocols are still proprietary and the full specifications are not disclosed.

* *Many versions:* some legacy protocols are intended to be extended by constructors with their own messages.

* *Large attack surface:* the end devices (e.g., programmable logic controllers) often act as network gateways between several legacy networks.

* *Process-oriented attacks:* the legality of a network packet depends on the state of the underlying physical system or the frequency of the packet. For example, opening a feeding tap of a tank will be harmless if the tank is at an intermediate level but may damage the plant if the tank is full. Similarly, Stuxnet only modified the value of an otherwise legal control command. Indeed, repeatedly starting and stopping an actuator may eventually damage it.

Process-oriented intrusion detection is an important topic in the ICS cybersecurity area.
## [Inria teams] Security of industrial systems

The CIDRE and CTRL-A teams developed an approach based on monitoring the process specifications. Process specification are safety properties expressed in Linear Temporal Logic (LTL). They are automatically mined from execution traces, then monitored by runtime verification techniques. Alert correlation between process monitors, a white list network Intrusion Detection System (IDS) and a pattern IDS are used to reduce the number of the false positives. The approach is validated on an ICS testbed.

The CTRL-A team has also developed a testbed and demonstrators of attack/defense in industrial systems and smart-grids. On the protocol vulnerability research side, the CTRL-A team worked on a vulnerability of the real-time protocol in IEC 61850 networks, demonstrating the attack, and proposed an IDS module in BRO.

## [Research challenge 11] Secure Industrial Systems

Industrial systems rely more and more on software mechanisms that can be attacked. Their security has thus become a major issue, especially since the consequences of an attack against such systems can be dramatic. Although traditional security approaches seem applicable to the case of industrial systems, their specificities require reviewing the traditional security mechanisms to adapt them to this new context. In particular, the communication protocols used in this context cannot be modified overnight. There must be a transition during which legacy communications should be embedded in secure protocols. In addition, real-time control of the system is usually required. Security must thus also be applicable in real time. Finally, it is often impossible to modify industrial devices. Therefore, preventive security mechanisms cannot be used. It is then mandatory to use reactive security and thus extremely important to study how effective attack detection mechanisms could be deployed in this context.

<figure><img src="image_29.png" /></figure>

## 6.3 Critical application areas

Some application areas are particularly security sensitive. Medicine is one example, as medical data is extremely sensitive and attacks on medical apparatus and implants could endanger life.

Finally, machine learning has recently become extremely popular in many different areas. Even though it is not an application area in itself, it is an interesting case study for security: machine learning techniques may be misled by so-called adversarial examples and present privacy risks when trained on sensitive data.
## 6.3.1 Medicine

### [Summary]

Medicine is being significantly transformed by the digital revolution, and is therefore also increasingly exposed to cybersecurity threats. One threat is the leakage of sensitive, medical data. As this data is extremely useful for research, the compromise between privacy and utility is of particular importance. Medical apparatus, such as surgical assistant robots, are increasingly connected. Similarly, medical implants offer wireless connections to avoid unnecessary surgeries, increasing the attack surface in a similar way as for CPS and IoT.

Medicine, as many other domains, is being significantly transformed by the digital revolution and is therefore also increasingly exposed to cybersecurity threats. While cybersecurity in medicine is not much different from cybersecurity in other domains, many aspects of cybersecurity are actually more sensitive in medicine, because medicine ultimately involves human lives, for which there is no price and thus no acceptable risk. This of course applies when an attack could endanger our lives, but also when it could leak our medical records that belong to the class of Sensitive Personal Information and thus are subject to very protective regulations (see chapter 5).

Therefore, looking at medicine as an application domain is instructive. Although privacy has long been a serious concern in medicine, it is still a challenging issue; other aspects of cybersecurity that could endanger our lives due to malicious actions are often underestimated.

In fact, while safety has always been part of the medicine culture, where all possible precautions are taken to save our lives, security is usually not much of a concern, because hospital and care centers are normally considered a sanctuary where there is no place for crime. This may be changing with the digitization of medicine, as attacks may be conducted remotely, from outside of the hospital.

There are mainly three kinds of threats: privacy leaks, malicious actions on medical apparatus, and attacks on implants.

### PRIVACY LEAKS

Our concern —and thus our awareness—for privacy leaks is much higher than for malicious actions, probably because even before digitization, we have always treated health information as very private and personal. However, medical data is being more and more centralized and stored for very long periods, hence, despite our awareness, this remains a critical issue that has not yet found an acceptable compromise.

Although medical data is highly sensitive and should not be leaked, the increasing amount of medical data over a person's lifetime and on a very large
population constitutes precious information for medical research. Indeed, a
medical study based on a few tens or hundreds of patients is far from statistically
sufficient. We will soon have the possibility of analyzing not only thousands or
even millions of records, but also records containing extremely rich information
for which machine learning techniques could reveal interesting correlations
between pathologies and physio-biological factors. Even though keeping medical
data private and decentralized would be a good answer to privacy preservation,
it could be a hindrance to medical progress, given the considerable potential
impact access to this data could have for medical discoveries.

MEDICAL APPARATUS

Health care is an important user of digital equipment. If it has long been the
case with imaging equipment, it has recently spread to all kinds of monitoring, as
well as robotic surgical assistants. All these systems are of course interconnected,
which increases the attack surface. There are no particularities to mention about
the apparatus except that in general cybersecurity threats and sometimes even
the safety of these devices are not given sufficient attention.

IMPLANTS

More worrying are medical implants (running the gamut from pacemakers, insulin implants, or audio implants to artificial hearts) that nowadays can be controlled remotely, allowing for the adjustment of parameters without a surgical intervention. This automatically exposes them to cybersecurity attacks as for Cyber Physical Systems (see §6.2). However, these implants are usually under important size and energy-consumption constraints, often making normal encryption techniques inapplicable, as is often the case in IoT devices. Manufacturers then choose ad hoc solutions that are likely to be insufficiently secure.

[Inria teams] Medicine

↗ The work of the **PETRUS** team on private Cloud (§4.2.2) was originally motivated as a way to decentralize medical data, allowing everyone to hold their personal medical records and keep control on what, how, and by whom pieces of information can be accessed when necessary.

↗ The **PRIVATICS** and **TYREX** teams are jointly working on better compromises between privacy and utility (§5.1.1), anonymization techniques (§5.2.2), and the privacy of machine learning (§6.3.3).
### 6.3.2 Robotics and connected autonomous vehicles

**[Summary]**

We can partition robotics into four major application areas: self-operating robots (e.g., factory robots, robots to assist the elderly), remotely operated robots (e.g., surgical robots, drones), robotic-based transportation (e.g., terrestrial vehicles with or without people inside), and large systems of robotic systems (e.g., smart cities). At Inria most research around cybersecurity and robotics focuses on connected autonomous vehicles (CAV's). Connected cars are communicating with each other and also with external objects. Due to legacy issues, architectures do not isolate critical parts from non-critical ones. This makes CAV's vulnerable to both internal and external threats. Internal threats are generally due to software bugs or unsecured channels. External ones are due to remote attacks exploiting communication channels. Current directions for avoiding these attacks tend to isolate the critical subsystems from non-critical ones.

Systems of connected robots are instantiations of the cyberphysical systems. Early examples can be found in factories and hazardous sites (e.g., nuclear or chemical plants). The closeness of humans and robots is getting tighter every day. Schematically, one may distinguish four application areas, which intersect partially: self-operating robots (e.g., factory robots, robots to assist the elderly), remotely operated robots (e.g., surgical robots, drones), robotics-based transportation (e.g., terrestrial vehicles with or without people inside), and large systems of robotic systems (e.g., Smart cities). Of course, there is a non-empty intersection between systems of connected robots and IoT. Cyberthreats may lead to undesired, possibly catastrophic and irreversible, outcomes in the cyberspace and in the physical space.

A recent study<fnref n="12" /> from TrendMicro and Polytechnic University of Milan showed that industrial robots connected to the Internet are vulnerable and not secured. This security problem is also known for tele-operated surgical robots<fnref n="13" />. Roboticists are paying increased attention to cybersecurity issues, due to emerging low-cost hardware and SoC's that are changing the economic side of the equation. The possibility of having numerous affordable “smart nodes” (having low energy and computationally efficient cybersecurity services implemented on them) within networks of robots or IoT opens up new perspectives.

Currently, within Inria, the robotics-related domain where cybersecurity issues are addressed is connected autonomous vehicles (CAV's), in particular cars—see the

<footnote n="12">12. https://www.trendmicro.com/vinfo/us/security/news/internet-of-things/rogue-robots-testing-industrial-robot-security</footnote>

<footnote n="13">13. http://www.washington.edu/news/2015/05/07/uw-researchers-hack-a-teleoperated-surgical-robot-to-reveal-security</footnote>
dedicated white book on this subject <fnref n="14" />. In addition to onboard robotics, CAV's will be equipped with radio devices enabling the exchange of data with other vehicles (Vehicle-to-Vehicle, or V2V, communications) as well as mobile access to Internet-based services (Vehicle-to-everything, or V2X, communications). Furthermore, CAV's will be equipped with internal sensors able to monitor passengers, in particular their drowsiness. However, in current vehicles, all devices share the same communication bus and most actuators (e.g., steering, brakes) are accessible via the onboard diagnosis port. Moreover, current onboard systems are monolithic. Thus, existing CAV's are vulnerable to both internal and external cyberthreats.

## INTERNAL CYBERSECURITY ISSUES

Internal issues are due to native attackers (faulty software, back doors) and to intrusions via unsecured channels (viruses, malware). Intrusions may lead an otherwise honest vehicle to behave maliciously. This has been illustrated with the takeover of a Jeep Cherokee using a Trojan horse placed in an MP3 CD.

Cameras and diverse sensors installed within CAV's raise specific privacy concerns. Who owns the collected data? What happens if the data is stolen? Also the underlying rationale where a passenger should be able to retake control whenever necessary is not widely acknowledged, nor accepted, since human reaction delays are usually considered to be too high.

## EXTERNAL CYBERSECURITY ISSUES

External attacks can be performed remotely via V2X communications. Cybersecurity issues raised with V2X technologies are similar to those arising in any network services. And since V2X communications depend on intermediate relays (e.g., road-side units and telecommunication network nodes), without specific protection, they favor man-in-the-middle attacks and hence, message falsification, suppression, or spoofing.

In France, state authorities and insurance companies require authentication and non-repudiation for proper identification of responsibilities (e.g., in the event of an accident). Thus a state-issued certificate is stored in each vehicle in a tamper-proof device. For improving VAC safety, periodic beaconing is being considered. This implies the broadcasting of unencrypted position, speed, and direction several times per second. The real identities would be obfuscated using pseudonymity, i.e., reversible anonymity based on asymmetric cryptography <fnref n="15" />. Unfortunately, it has been shown that frequent changes of pseudonyms do not prohibit linkability of CAV paths. This is a real privacy concern and beaconing has come under question.

<footnote n="14">14. Véhicules autonomes et connectés — les défis actuels et les voies de recherche. Available at https://www.inria.fr/institut/strategie/vehicules-autonomes-et-connectes</footnote>

<footnote n="15">15. https://research.utwente.nl/en/publications/pseudonym-schemes-in-vehicular-networks-a-survey</footnote>
To avoid the above attacks scientists and manufacturers are defining partitioned architectures comprised of two subsystems. A Safety Critical subsystem is in charge of critical V2V messaging, processing, and cooperative driving. A non-safety critical subsystem, isolated from the critical part, is in charge of services based on V2X communications (e.g., infotainment, traffic and road conditions, free parking places, personal messaging). The safety critical subsystem will be endowed with the capability of inspecting V2X messages received by the non-safety critical subsystem, prior to importing them for further processing.

### [Inria teams] Connected autonomous vehicles

The CIDRE team currently works on the definition of an intrusion detection system (IDS) that could be integrated on the next generation vehicle architecture. Several paradigms (network vs host-based IDS, anomaly vs detection) should be explored in the CAV context.

The RITS team is conducting research on safety critical subsystems and protocols for vehicular communications. The goal is to enable the prevention of attack, their immediate detection, and that render eavesdropping and tracking unfeasible and useless. In addition to pseudonymity, anonymity (non-reversible obfuscation of identities) of message senders has been shown to be feasible by forming trusted ad hoc vehicular sub-networks spontaneously. Onboard systems come with a "stealth mode" option (no sending of V2X messages), for increased privacy.

### 6.3.3 Machine learning based technologies

#### [Summary]

Machine learning is used in an increasing number of applications, including e-commerce and recommendation systems, advanced language translation mechanisms, spam or parental filtering tools, as well as self-driving cars and cyber physical systems in general. The deepest impact so far is certainly in speech and image recognition, but other areas are also significantly impacted. Machine learning techniques suffer from two main threats in relation to cybersecurity. The first one, called adversarial machine learning, consists in adding carefully designed noise (barely visible to human eye) to an image, leading to misclassification. The second one is related to privacy and consists in extracting information about training data from a trained network.

Machine learning (ML) based technologies now form the backbone of a quickly growing number of organizations and services. These include e-commerce and recommendation systems, advanced language translation mechanisms, spam or parental filtering tools, as well as self-driving cars. Millions of users interact on a daily basis with such systems, transparently, without even noticing.
There is one domain where convolutional neural networks and deep learning strategies have had a profound impact: computer vision. Traditional problems such as precise image classification or fine grain object detection and labeling in images have recently made enormous progress, with state-of-the-art approaches outperforming by far older methods and even surpassing human capabilities for specific domains.

ADVERSARIAL MACHINE LEARNING

Recent research has shown that such machine learning approaches can be subverted when a small amount of carefully-designed, imperceptible adversarial noise is added to the input data. Such a perturbed image, called an adversarial example, is in turn typically misclassified, although the perturbation is barely visible to human eyes. Adversarial perturbation generalizes surprisingly well across different input images, classifiers, and models, even when trained on diverse learning sets; worse, this phenomenon is not particular to deep learning and can be observed even with simpler classifiers. In addition, given some input data it seems to be relatively easy to compute a small perturbation of that input that will be misclassified. These alarming observations have many practical implications in an area where machine learning technologies are ubiquitous.

Ongoing research at Inria focuses in part on convolutional neural networks and follows several directions. It is necessary to gain a better understanding of the weaknesses of deep learning strategies, to analyze what kinds of attacks are possible and propose mechanisms for protecting convolutional neural networks against adversarial attacks. It is important to gain insight on why adversarial perturbations generalize so surprisingly well. Another approach is to monitor the internal activations that flow between the layers of a deep network, in order to observe where spurious artefacts take place, whether or not this is related to the dimensionality of the intermediate vector representations. It is certainly worth observing the effects of defensive strategies such as distillation on these flows to see why it makes the overall process more robust. Furthermore, it seems essential to connect adversarial perturbations to the subspace that corresponds to natural images, the subspace where classifiers are trained. This may facilitate the identification of unnatural images that in fact belong to different subspaces. Yet, adversarial natural images do exist and succeed in subverting the detection systems. Considering subspaces is therefore not the ultimate solutions for making systems more robust.

Finally, while most research currently focuses on images, other modalities such as adversarial text or adversarial audio are likely to raise a whole new set of difficulties which also have to be addressed. Domains that are outside multi-media are likely to face similar sensitivities to adversarial behaviors as they also heavily rely on machine learning, e.g., applications for spotting intruders from the network analysis or applications dealing with biometric traits.
# PRIVACY ISSUES AND MACHINE LEARNING

Using machine learning techniques also raises privacy problems. In addition to the use of machine learning techniques to infer possibly sensitive data, machine learning also raises the question as to whether an attacker who has access to the trained network can gain information about the training data. Different scenarios can be distinguished according to whether an attacker is granted white-box (giving access to the neural network internals) or black-box access to the network, and whether their aim is to extract training data or simply decide whether a given input was part of the training data. These privacy properties have been stated in terms of differential privacy, introduced in the area of database anonymization (discussed in §5.2.2). In the worst case, the attacker may gain access to the stored training data itself. This raises the additional questions on how to transform the data prior to storage, so as to discard any private information that is useless for the task at hand, and how to train in a distributed online fashion, in order to avoid storing all data in a single place which increases the risk of a security breach.

## [Inria teams] Cybersecurity and machine learning

While adversarial machine learning is often considered as a cybersecurity issue, as it can be used to attack critical systems that use machine learning techniques, it is actually quite different from traditional cybersecurity attack techniques, and therefore, mostly tackled by teams working in machine learning. Conversely, the privacy issues raised by machine learning are typically cybersecurity ones.

☞ The **COMETE** team is interested in the privacy aspects of machine learning, in particular through the use of differential privacy.

☞ The **LACODAM** team, working in machine learning, is also considering applications to cybersecurity.

☞ The **LINKMEDIA** team investigates adversarial machine learning issues that go beyond adversarial computer vision and considers for instance adversarial audio, adversarial video, adversarial Automatic Speech Recognition (ASR), or adversarial Natural Language Processing (NLP). Adversarial machine learning is also considered from the perspective of each modality, as well as considering truly multimodal inputs. The team also considers a wide range of applicative goals that go beyond classification and include adversarial retrieval.

☞ The **MAGNET** team works on privacy considerations in machine learning, in the context of decentralized learning where several actors collaborate to improve the model without leaking personal data, or in datamining of mobility traces.

☞ The **MULTISPEECH** and **MAGNET** teams work on privacy-friendly speech recognition. The goal is to train a speech recognition system on users' speech data without disclosing information about the identity, the traits (e.g., gender, age, or ethnic origin), or the states (e.g., health or emotional state) of individual users. To do so, the teams
investigate adversarial learning (which is seen here as a solution rather than a problem),
decentralized training, and formal privacy frameworks such as differential privacy.

esk The ORPAILLEUR team is interested in the privacy aspects of machine learning.

→ The PRIVATICS and TYREX teams are jointly working on privacy preserving decentralized or federated machine learning, in particular in the context of large medical databases.

The SEQUEL team, working in machine learning, is also considering applications to cybersecurity.
7

Cybersecurity
in France

<figure><img src="image_30.png" /></figure>
## [Summary]

France is one of the major European actors in cybersecurity. Its academic workforce consists in almost one thousand people, including researchers, faculty, postdoctoral fellows, Ph.D students, and research engineers.

Almost 25% of the French academic activity in cybersecurity is conducted at Inria within joint teams with other academic institutions, primarily CNRS and universities, where on average only half of the staff is from Inria. This gives Inria a leverage effect and makes it one of the main French academic actors in cybersecurity. Inria's forces in cybersecurity represent about 7% of its total activity.

A large and visible effort, both at Inria and in France, is dedicated to cryptography and formal methods applied to cryptographic protocols and privacy. Hardware security is also well represented in France, but with very few forces are Inria. Work on network security, system security, and, more generally, on reactive security are, however, under-represented both at Inria and in France given the increasing challenges of cybersecurity for critical infrastructures and the arrival of the Internet of Things.

Education in cybersecurity is becoming a key challenge, at all levels. Inria contributes to education via the advising of Ph.D students, many of whom will later be working in state bodies or private companies. Transfer via the creation of startups does exists, but is not as of yet an important vector.

French academic forces in cybersecurity are rather well organized and coordinated, in particular via the dedicated pre-GDR and Allistene working group. However, there is still insufficient interaction and coordination between the industry and academic forces.

Although cybersecurity is usually well recognized as a priority in France, in Europe, and worldwide, it is important that France also reinforces its financial support to maintain its leading position and its capacity to seize economical opportunities.

# 7.1 Academic forces at Inria and in France

## Research organization and forces

Research at Inria is organized into small teams sharing a common research project, which are often joint teams with other academic institutions such as CNRS, Universities, specialized engineering schools, or other research institutes (INRA, INSERM, etc.). On average, about half of the research staff in Inria teams are from partner institutions, giving Inria a leverage effect. Hereafter, research at Inria always means research in Inria's teams including partners. The average size of a team is 18 people—counting researchers and faculties, postdoctoral fellows, Ph.D students, and research engineers—but with a large variation, ranging from 3-5 persons for the smallest teams up to 45-50 for the few largest ones.
Cybersecurity has been one of Inria's research priorities for the last fifteen years<fnref n="1" />. Cybersecurity now covers 7% of Inria's activity, with about 30 teams working in this field<fnref n="2" />, two thirds of which have cybersecurity as their unique or primary research topic. Altogether, this amounts to about 200 full-time positions. This represents a fourth of the French academic forces in cybersecurity. The other main academic forces are scientists from CNRS and faculty members from universities and engineer schools hosted in CNRS UMRs<fnref n="3" /> but outside of the Inria teams, faculty members from the Institut Mines Telecom (IMT), and CEA scientists. French academic forces in cybersecurity have roughly doubled in the last decade. This growth is continuing but at a smaller pace. While there are some new hires, the growth is mostly due to researchers and faculty members moving to cybersecurity from other fields of computer science (see §1.1).

## Research domains

Figure 8.1 describes the proportion of research activities between the main domains of cybersecurity at Inria, in red, and in all French academic entities taken together, in blue. The comparison is instructive. Cryptology is a strength of Inria, with a third of its workforce. This results from Inria's long involvement in number theory, computer algebra, cryptanalysis, and coding theory. Indeed, Inria plays a key role worldwide in the design of new cryptographic primitives or protocols and the cryptanalysis of cryptographic primitives.

<figure><img src="image_31.png" /><figcaption>Figure 8.1: Main topics in cybersecurity by % of French academic forces<fnref n="4" /></figcaption></figure>

<footnote n="1">1. Security was extensively discussed in Inria's Strategic Plan for the period 2003-2007.</footnote>

<footnote n="2">2. All Inria teams working in cybersecurity are listed in Appendix A.</footnote>

<footnote n="3">3. Mixed (organization) Research Units.</footnote>

<footnote n="4">4. Source: cartography of French academic forces in cybersecurity done by the Allistene Alliance's working group on cybersecurity, see https://www.allistene.fr/files/2018/03/VFCartographie_2017-06-13.pdf.</footnote>
<figure><img src="image_32.png" /><figcaption>Figure 8.2: Geographic breakdown of academic cybersecurity forces</figcaption></figure>

The next most important research domain at Inria is formal methods applied to security and privacy. As explained in §1.1, this is largely due to the transfer of competences coming from formal methods combined with the fact that Inria, and France in general, are very well-positioned in the area of formal methods. Conversely, hardware and system security are underrepresented at Inria, but this area is still well-covered elsewhere in France at CEA, CNRS, and IMT. Notice that there is very little research at Inria on forensics. Nevertheless, in a collaboration with tracIP<fnref n="5" /> the **RESIST** team works on the development of a forensics platform dedicated to industrial control systems. Outside of Inria, research on forensics does exist in France<fnref n="6" />.

Figure 8.2 describes the geographic breakdown of academic research forces in France. The unit of measure is ETP, which amounts to a full time research activity. This shows that besides Paris the main forces in cybersecurity are in

<footnote n="5">5. https://www.tracip.fr</footnote>

<footnote n="6">6. The activities in forensics in France are probably underestimated, since the gures do not cover research in humanities.</footnote>
Brittany (Brest and Rennes) and in the Rhone-Alpes region (Lyon and Grenoble). Next comes Nancy and the Côte d'Azur (Nice and Sophia Antipolis). For Inria, the forces are in the Paris region (Ile de France) and then comes Rennes and Nancy.

## Non academic forces

There are also important non-academic research forces. Indeed, research is also conducted in private companies, although it is usually more applied. Moreover, there are institutional bodies, such as DGA<fnref n="7" />, which is part of the French ministry of armies, and the ANSSI<fnref n="8" />, the French cybersecurity agency, which is part of the SGDSN<fnref n="9" /> placed under the Prime Ministry, which have toplevel cybersecurity expertise, including some research activity, and play a key role in defining and conducting the French cybersecurity policies. There are also some laboratories depending on the Ministry of the Interior, such as the CREOGN<fnref n="10" />. Research regarding privacy issues and regulations are also conducted in the LINC<fnref n="11" /> innovation lab of the CNIL<fnref n="12" />, the French data protection authority. Inria teams regularly have scientific collaborations on some specific projects with most of these organizations.

## Community animation

There are also a number of associations that play an important role in the animation of the cybersecurity community. The CNRS has created a pre-GDR<fnref n="13" /> on cybersecurity <fnref n="14" /> in 2016, whose purpose is to animate the French academic community in cybersecurity, in particular via the organization of workshops or summer schools.

The Allistene alliance <fnref n="15" /> has created a working group on cybersecurity <fnref n="16" /> where Inria and the main academic actors have representatives and whose aim is to exchange information, to build a shared vision, to conduct general-purpose studies such as the cartography of academic forces described above, and to coordinate actions like the participation of the Allistene members to the FIC<fnref n="17" />.

Finally Inria is part of the working group devoted to research and innovation of the CoFIS (Comité de la Filière industrielle de sécurité), whose main role is to foster the French security industry by proposing targeted actions to increase both competitiveness and security at the national and European levels.

<footnote n="7">7. Direction Générale de l'Armement.</footnote>

<footnote n="8">8. Agence Nationale de la Sécurité des Systèmes d'Information.</footnote>

<footnote n="9">9. Secrétariat Général de la Défense et de la Sécurité Nationale.</footnote>

<footnote n="10">10. Centre de recherche de l'Ecole des officiers de la gendarmerie nationale</footnote>

<footnote n="11">11. Laboratoire d'Innovation Numérique de la CNIL</footnote>

<footnote n="12">12. Commission Nationale de l'Informatique et des Libertés</footnote>

<footnote n="13">13. A GDR (Groupement De Recherche) is a structure lead by CNRS to animate the French scientific academic research community in a particular area.</footnote>

<footnote n="14">14. http://gdr-securite.irisa.fr/index.html</footnote>

<footnote n="15">15. http://www.allistene.fr/</footnote>

<footnote n="16">16. https://www.allistene.fr/organisation-allistene/groupes/groupe-cybersecurite/</footnote>

<footnote n="17">17. Forum International de la Cybersecurité</footnote>
Inria is also part of the ACN<fnref n="18" /> professional association whose role is to federate and represent the main industrial actors in cybersecurity. HEXATRUST is another important association composed of 29 SME's in cybersecurity, where Inria (and academic research) is however not represented.

## 7.2 Education

Education is an important issue, as there is a huge lack of expertise in cybersecurity at all levels, but high-level expertise remains the crux. This situation has been known for a few years, thus some engineering schools and universities have created new cybersecurity programs. Although Inria is not a university, education and knowledge transfer are still one of its important missions, and Inria contributes to education in several ways.

Many Inria researchers are teaching advanced cybersecurity courses at engineering schools and universities, especially in master-level courses, and in dedicated summer schools. Inria also created a couple of MOOCs on cybersecurity.

More importantly, most researchers are advising PhD students. Those that do not continue in academia will be hired by industry or other institutions; this is one of the most efficient ways of transferring expertise. The number of Ph.D students is about the same as the total number of researchers and faculty members, a proportion that is roughly the same for cybersecurity and other research domains. Since the industry is lacking cybersecurity experts, it is important to increase the number of students in cybersecurity at all levels, including Ph.D's. Inria could increase its number of Ph.D's, but only if there is a corresponding increase in Ph.D funding for cybersecurity and in good Ph.D candidates. Hence, the main challenge remains to attract more young students to computer science in general and to cybersecurity in particular (see §8.2.4).

## 7.3 Inria's impact in cybersecurity

France is one of the worldwide leading countries in cybersecurity. This strength is due to its academic forces including those at Inria. Besides strong research results and maintaining the highest level of expertise in most subdomains of cybersecurity, Inria's research is also a key contribution to the community.

France and Inria have long been participating in cryptanalysis challenges, holding several factorization records. This effort is necessary to check the state of the art in cryptanalysis continuously, both in terms of algorithmics and computing power, and to recommend adjustments of cryptographic keys accordingly or a change of cryptographic primitives. This is a key contribution to the community.

<footnote n="18">18. Alliance pour la Confiance Numérique</footnote>
Researchers are also often part of standardization committees, such as IETF. This is often a time-consuming long-term effort but it is important to raise the quality of the standards.

## Inria's startups

In comparison with its research impact in cybersecurity, only a few startups in cybersecurity are directly emerging from Inria teams.

### CRYPTOSENSE

Founded in 2013 and located in central Paris, Cryptosense<fnref n="19" /> is an academic spin-off of Inria and Ca'Foscari University of Venice. They develop software based on the academic research carried out by Graham Steel and his colleagues from the Prosecco team and the former Secsi teams.

Their main product, Cryptosense Analyzer, discovers security flaws in cryptographic systems using a variety of techniques. Cryptographic hardware is treated as a black box and tested using fuzzing techniques. The fuzzing results are used to infer a logical model of the device that is then analyzed by model checking. Cryptographic software is tested by instrumenting calls to cryptographic libraries. A model is inferred from these calls which is analyzed against a database of cryptographic usage rules. The libraries themselves are tested for a suite of cryptographic attacks.

Cryptosense software is used by a number of international banks, payment providers, technology companies, government agencies, and hardware manufacturers, in Europe and North America.

### CYBER-DETECT

The Cyber-Detect<fnref n="20" /> start-up was created from research on malware carried out in the former Carte team and continued in the Carbone LORIA team. Since several years, they developed a solution, dubbed morphological analysis, to analyze binary codes and detect malware. Morphological analysis is a method that consists in abstracting the control flow graph of a binary code and fully automatically build signatures from this abstraction. The recombination of signatures allows one to identify malicious functionalities. The resulting prototype is now commercialized by Cyber-Detect as a solution to help reverse analysis and forensics.

### LYBERO.NET

The Lybero.net<fnref n="21" /> start-up, issued from the Inria Nancy - Grand Est research center, proposes two commercial services. The first one is a quorum-based digital

<footnote n="19">19. https://cryptosense.com/</footnote>

<footnote n="20">20. https://www.cyber-detect.com/</footnote>

<footnote n="21">21. https://lybero.net/</footnote>
distributed escrow that enables the recovery of a forgotten key (or any digital
content) if a pre-defined quorum of administrators has been gathered.

The second service is CryptnDrive, a secure cryptographic driver that enables a
secure and easy exchange of files within and outside an organization. Through a
regular web browser, users can securely exchange important documents without
ever exchanging any password with the remote destination, confidentiality being
guaranteed by end-to-end encryption. Additionally, this service leverages on the
quorum-based escrow in order to recover from lost credentials.

MALIZEN

Although many tools are used to secure our information systems proactively, we regularly see that attackers find ways to circumvent them or exploit flaws. Monitoring solutions allows one to detect, characterize, and respond to these intrusions in the majority of cases. However, it sometimes becomes necessary to handle exceptions or verify that these systems are functioning well. Faced with the masses of data collected from monitoring, security experts are often poorly equipped and have difficulties to respond to security incidents.

Fuelled by the promising results of collaborative research projects between Inria and DGA-MI, Malizen<fnref n="22" /> is a startup whose goal is to equip experts in cybersecurity with hardware and software bricks to help them better respond to security incidents.

By using graphical user interfaces and data visualization, especially designed for cybersecurity, the experts are reintegrated into the analysis process. Those responsible for analyzing intrusions can explore their security data more intuitively and better understand critical situations.

Malizen is supported by Inria using research from the **CIDRE** team.

7.4 High Security Laboratories (LHS)

Inria has also two *High Security Laboratories* (LHS) located in the research
centers of Nancy and Rennes that are shared with Inria's local partners, CNRS
and universities, and also supported by the Grand-Est and Bretagne regions, and
DGA in Rennes.

Both LHS are in secured rooms. The Nancy-LHS has a network telescope that captures malware code and attack logs and permits experimenting with Internet probes. It has a closed network for sensitive experiments such as malware code analyses and an isolated room, not connected to the Internet, where highly sensible information can be processed and confidential hardware and software experiments can be conducted. The Rennes-LHS hosts three platforms: one for experimenting

<footnote n="22">22. https://malizen.com/</footnote>
with malware and ransomware infection and repair, which contains a database
of computer viruses and ransomware; one for electromagnetic observation and
analysis to experiment with side-channel attacks; and the last for electromagnetic
hardware fault injection.
8

Conclusions
and recommendations

<figure><img src="image_33.png" /></figure>
While cybersecurity was not an issue for the general public two decades ago, everybody is concerned nowadays: states, industries, and citizens. Cybersecurity is a hot topic with many economic, societal, political, or geopolitical security issues at stake and will very likely remain so in the next decades. We conclude this white book with the list of research challenges that we have identified and a few general recommendations related to the organization of research in cybersecurity and the interaction between researchers in cybersecurity and the society.

## 8.1

# Research challenges

While the challenges identified throughout this white book are not the only important topics, we believe them to be of particular importance and we recommend they be treated as a priority. Therefore, and for convenience, we recall them below. Each challenge can be put in context in the section where it was first introduced.

### 8.1.1 Hardware-targeted software attacks (see 2.1)

Attacks against information systems do not usually involve the hardware layer but exploit a software vulnerability. However, recent attacks, such as Rowhammer, Spectre, or Meltdown, have shown that attacks implemented in software can exploit performance optimizations of the hardware. This new type of attack is especially dangerous as it makes hardware attacks possible at a distance, as opposed to classical side channel attacks. It is not yet completely clear how the current proof of concept attacks can be "industrialized," but they pave the way to a new class of serious attacks. Therefore we need to get a better understanding of how such attacks could possibly be deployed, propose a clear typology of this new kind of attack, and propose countermeasures, both at the hardware and the software levels. This task requires expertise at the hardware, firmware, and operating system levels. The countermeasures can also be difficult to design as they may require to revisit crucial optimizations used for years, such as speculative execution.

### 8.1.2 Security and usability (see 2.3.2)

Very often, when users request a service, they are willing to sacrifice security, and bypass an annoying security mechanism, if it prevents them from using the service. In order to avoid this problem, security must be as transparent as possible. Even though complete transparency is not always possible, security services must be as simple as possible to use. Work is needed to propose interfaces and security mechanisms that are suitable for nonexpert users, that ensure the user is well aware of the consequences of their actions, and that prevent users from making errors that compromise security. Designing such usable security mechanisms calls for interdisciplinary research typically including experts in cognitive sciences.