MACTA: A Multi-agent Reinforcement Learning Approach for Cache Timing Attacks and Detection
Authors: Jiaxun Cui
Xiaomeng Yang
Mulong Luo
Geunbae Lee
Peter Stone
Hsien-Hsin S. Lee
Benjamin Lee
G. Edward Suh
Wenjie Xiong
Yuandong Tian
Venue: ICLR 2023 poster (2023)
Keywords: multi-agent reinforcement learning
security
game theory

Abstract: Security vulnerabilities in computer systems raise serious concerns as computers process an unprecedented amount of private and sensitive data today. Cache timing attacks (CTA) pose an important practical threat as they can effectively breach many protection mechanisms in today’s systems. However, the current detection techniques for cache timing attacks heavily rely on heuristics and expert knowledge, which can lead to brittleness and the inability to adapt to new attacks. To mitigate the CTA threat, we propose MACTA, a multi-agent reinforcement learning (MARL) approach that leverages population-based training to train both attackers and detectors. Following best practices, we develop a realistic simulated MARL environment, MA-AUTOCAT, which enables training and evaluation of cache-timing attackers and detectors. Our empirical results suggest that MACTA is an effective solution without any manual input from security experts. MACTA detectors can generalize to a heuristic attack not exposed in training with a 97.8% detection rate and reduce the attack bandwidth of adaptive attackers by 20% on average. In the meantime, MACTA attackers are qualitatively more effective than other attacks studied, and the average evasion rate of MACTA attackers against an unseen state-of-the-art detector can reach up to 99%. Furthermore, we found that agents equipped with a Transformer encoder can learn effective policies in situations when agents with multi-layer perceptron encoders do not in this environment, suggesting the potential of Transformer structures in CTA problems.


Published as a conference paper at ICLR 2023
MACTA: A MULTI-AGENT REINFORCEMENT LEARNING APPROACH FOR CACHE TIMING ATTACKS AND DETECTION
Jiaxun Cui
Xiaomeng Yang 22
Mulong Luo Geunbae Lee
Peter Stone
Hsien-Hsin S. Lee Benjamin Lee
G. Edward Suh
Wenjie Xiong Yuandong Tian
1 The University of Texas at Austin "Meta AI
'Cornell University
4 Virginia Tech
5 "Sony AI Intel Corporation "University of Pennsylvania
cuijiaxun @ uuexxas.edu, yangxm @ meta.com, ml2558 @ cornell.com, geunbac @ vt.edu, pstone @cs wtw.ustexxccuu linear@aem acm. leebec seass seas. www.uueeeen.eeee edu. edsuh @ meta.com, wenjiex @ vt.edu, yuandong @ meta.com
ABSTRACT
Security vulnerabilities in computer systems raise serious concerns as comput- ers process unprecedented amount of private and sensitive data today. Cache- timing attacks (CTA) pose an important practical threat as they can effectively breach many protection mechanisms in today's systems. However, the current de- tection techniques for cache timing attacks heavily rely on heuristics and expert knowledge, which can lead to brittleness and the inability to adapt to new attacks. To mitigate the CTA threat, we propose using MACTA, a multi-agent reinforce- ment learning (MARL) approach that leverages population-base -based training to train both attackers and detectors. Following best practices, we develop realistic sim- ulated MARL environment, MA- AUTOCAT,, which enables training and evalu- ation of cache-timing attackers and detectors. Our empirical results suggest that MACTA is an effective solution without any manual input from security experts. MACTA detectors can generalize to a heuristic attack not exposed in training with a 97.8% detection rate and reduce the attack bandwidth of RL-based attackers by 20% on average. In the meantime, MACTA attackers are qualitatively more effec- tive than other attacks studied, and the average evasion rate of MACTA attackers against unseen state-of-the-art detector can reach up 99%. Furthermore, we found that agents equipped with a Transformer encoder can learn effective poli- cies in situations when agents with multi-layer perceptron encoders do not in this environment, suggesting the potential of Transformer structures in CTA problems.
INTRODUCTION
With increasingly sensitive data and tasks, security in modern computer systems is recognized as one of the 14 grand challenges for engineering (National Academy of Engineering (007). As
concrete example, cache-timing attacks (CTA) in processor caches have been shown to leak private encryption keys YYrrom& & Falkner 2014 Liu et al. 2015), break existing security isolation (Kocher) cct al., 2019). cause privilege escalation (Lipp et al. 2018), and break new hardware security features in the latest processors (Ravichandran et al. 2022) In CTA. the attacker is able to gain such access to private information (e.g., via memory access patterns) from the victim who shares a cache with the attacker. Over decades, the attack and defense policies in CTA have been explored manually by computer architecture experts. To defend against such attacks, statistical analysis and machine learning models with static strategies have been proposed for CTA detection, e.g., CC-Hunter (Chen) & Venkataramani] 2014) uses auto-correlation and Cyclone (Harris et al.] 2019) uses an SVM clas- sifier. Yet, new CTA attacks are still being reported (Xiong & Szefer] 20200] Briiongss et al. 2020 Saileshwar et al., 202T; Guo et al. 2022b a showing higher leakage rates or the ability to bypass existing defensive mechanisms.
Equal second-author contribution. Equal supervision. Code available https://gghu...com//ccchtttrrree

---

Published as a conference paper at ICLR 2023
Computer security can be seen competitive game between the attackers and the defenders, and teeneetheoreticccc approaches that analyze strategy (policies) for both sides have been proposed (Anwar et al. 2018 Elderman et al. 2017 Eghtesad et al. 2020). These methods highly abstract the attack and defense strategies, usually based on known attacks and defenses, and analyze simplified games in the limited strategy spaces. For example, (nwar et al. (20T8) studies CTA-llie attack scenarios where the attacker decides when to terminate its attack and the defender decides an abstract security level. However, real-world CTA has large action and state spaces for different agents, sparse reward, and long game horizons, making the game analysis hard without exploring all possible policies.
In this work, we use multi-agent reinforcement learning (MARL) to jointly explore and optimize complex attack/defense policies in CTA. We take an integrated approach of reinforcement learn- ing and game theory. First, we build multi-agent gym environment, MA-AUTOCAT, that closely models a realistic CTA setting and allows efficient learning for both attackers and defenders. Specif- ically, we study a aeteet-and--errinaaee defense. Second, we introduce and evaluate a MARL ap- proach, named MACTA, to automatically find both attacker and detector policies through self-play, similar to past successes in games with large state/action spaces (e.g., StarCraft (Vinyals et al.) 2019), Go (Silver et al.. 2016), and Poker (Brown & Sandholm 2019 ). MACTA adopts Fictitious Play (FP) (Brown, 195 1), population based training in MARL Vinyals et al. 2019) and Proximal Policy Optimization (PPO) (Schulman et al., 2017) to learn the best response policy pool of di- verse opponents, to avoid cyclic behaviors of the attacker/defender policies. Finally, MACTA uses Transformer architecture to parameterize the policy/value function so that an important subset of actions can be picked quickly during training, yielding fast policy learning.
We performed extensive experiments in a representative setting of cache-timing attack. The ex- periments show that learned policies trained with MACTA can generalize to detectorssattackers that they were not exposed to during the learning phase (henceforth referred to as "unseen de- rectors/attackers"). The MACTA detector exhibits a 97.8% detection rate on an existing human- designed attack without training on it and can lower the number of attacks per episode (bandwidth) of adaptive attackers by 20% on average. The MACTA attacker can bypass previously unseen detector, Cyclone, with a more than 99% success rate.
While there has been increasing interest and effort in using machine learning for computer system security recently, our work is the first to introduce the hardware timing attack problem as a promising application of MARL and show that MARL can be effectively applied to detect simulated CTA attacks with strong generalization.
Our main contributions are follows:
We contribute simulated multi-agent environment MA-AUTOCAT that models realistic CTA and allows learning in both cache timing attacks and defenses.
We introduce and evaluate MACTA, multi-agent learning approach for CTA, and show the resultant detector acquires interesting high-level patterns that can generalize to novel attackers and make the cache less exploitable to bigh-bandwidth attacks.
Our study the neural architecture of learning agents indicates that the CTA is one case where Transformers are significantly better for retrieving state information than multi-layer perceptrons.
2
THE CACHE TIMING ATTACK CHALLENGE
The cache timing attack challenge is a fundamental problem to address as such kinds of attacks are stealthy but powerful. We leave the detailed reasons for studying the problem in Appendix A. and introduce the domain knowledge and problem formulation in this section.
2.1 DOMAIN DESCRIPTION
A cache is small and fast on-chip memory commonly used in modern processor designs to reduce latency of memory accesses. Accessing memory addresses whose data are available in a cache is fast (called "cache hit"). If the data is not in the cache, data has to be retrieved from the main memory, which is much slower (called "cache miss").
Surprisingly, this timing difference in memory accesses due to caching could leak information across different programs/processes executing with a shared cache, a vulnerability known as cache timing attacks (CTA). As shown in Figure [ a), CTA involves the attacker process and the victim process
2

---

Published as a conference paper at ICLR 2023
both sharing the same cache. An example (Prime+Probe CTA (Luu al. 2015) is given in Fig- ure bb). The victim's 's memory access will evict the attacker's ' cache line from the cache, causing latency changes in the attacker's future memory accesses. Thus, the attacker can infer whether the victim made access to specific memory address by observing its own memory access latency, and thus be able to infer the victim's private information.
Attacker Process
(b)
Attacker Prime:
Virtual Mamory Space
A0
A1
A2
A3
Cache
Attacker accesses AO, and occupies the whole cache
+
(2) Victim access:
2
3
Cache
A3
Cache
Victim accesses one of the secret address example; victim accesses
(3) Attacker Probe:
V9
Vi
yiS
A0
A1
A2
A3
Cache
Attacker accesses AD, Cache Hit (Fast access)
Memory Space
Attacker accesses A1, Cache Miss (Slow access) Victim's secret address must be V1!
Victim Process
Hit
Miss
Hitt
Hit
Figure 1: (a) Cache timing channel attack is formed when the attacker process the victim process use the same locations of shared cache for their memory accesses. (b) An example of Prime+Probe CTA in a 4-set direct-mapped cache. The attacker process can infer which memory address the victim process accesses by observing the latency.
2.2 PROBLEM STATEMENT
In this work, our goal jointly find novel attackers and robust detector policies that can generalize to unseen opponents, leading to insights for future cache design. The problem of joint learning can be formulated as a general-sum Partially Observable Markov Game (POMG), where the attacker and detector have limited observations and optimize for their own cumulative return. Given the finite set of policies, the resultant attacker is the best response to a mixture of all detector policies explored, and the resultant detector is the best response mixture of all attacker policies explored.
Partially Observable Markov Games (POMGs) Formally, an n-player episodic POMG can be described using tuple (I, T. {A] (O) {R] 1 7). where I is the finite set of play- ers, T is the episode length, S is the true state space, P is the state transition probability. A; is the action space of player i, and the joint action space of all agents is {A) Similarly, O is the observation space of player i, and R, is the reward function for player i. Lastly, y [0, 1] a reward discount factor. In POMGs, each agent only has access its own observations and actions, and its goal is to maximize the cumulative episodic reward for itself given the oppo- nents' policies, J'(m ) E E 7'r2/00, at T/' (s1), a, (s))] A Nash Equilibrium (NE) is one solution concept to POMGs. Formally, a NE is defined as a saddle point where for any player's policy n¹. we have J' .. J J'T'', ,,,), Vi e M. Namely, given all other agents' equilibrium policies प.¹ there is no motivation for agent i to unilaterally deviate from its current policy œं achieve higher returns.
A A2... An-
MA-AUTOCAT
To study the learning dynamics of the attackers and the detectors in CTA, we develop MA- AUTOCAT, a gym (Brockman et al., 2016) environment that models realistic multi-agent CTA interactions. We build the environment based on cache simulator, which faithfully models cache state changes, following practices in prior works on CTA detection schemes (Harris et al. 2019, Mirbagher-Ajorpazz et al. 2020). Note that experimenting detectors on real processors requires hardware modifications, which is prohibitively expensive. Figure demonstrates the environment components and game mechanism.
In AA-AUTOCAT. each agent plays different role, and each role has a specific goal (i.e., reward), different level of privileged accessibility (i.e., observation) to the information of the environment, and a different way to take actions (i.e., action space), listed as below:
Benign Program (B) accesses memory in regular way, implemented by replaying offline log
of memory accesses from regular programs (e.g., a standard benchmark suite such as SPEC (Bucek
et al. 2018)). It has observation and policy needs to be learned.
Victim (V) accesses memory with addresses that depend on a secret. Studies have shown that such secret-dependent memory accesses common in real-world applications (e.g., HTTP parser), li-
3

---

Published as a conference paper at ICLR 2023
Correct Alarm:
Detector receives reward Detector receives penalty
Negative:
Observe memory accesses by both programs
Observe latency
Detector Agent
Cache
Attacker Program
Benign Program
a: Detector alarms: Terminate the programs
simulator
Memory accesses by both programs
Attack Scenario
Viclim Program
Benign Program
Benign Scenario
Successfful Attack without alarm: Attacker receives reward Unsuccessful Attack:
Attacker receives penalty
We propose MA-AUTOCAT, multi-agent environment to jointly explore and optimize the policies of the attacker and the defender processes in CTA. In this environment, multiple agents can play different roles and learn from each other. The goal is to learn policies that can generalize to deal with previously unseen opponents (c.g., those designed by human heuristics).
Figure 2:
braries (e.g., OpenSSL), and Linux kernel (ohhnnesmmeyrr et al. 2022; Qi et al. 2021; Oleksenko al. 2020). In CTA, victim's secrets usually contain multiple bits, and attackers target bit at a time; after guessing one bit of secret, the attacker moves to the next bit. To model this in our environment, the secret bit is reset after the attacker's attempt to guess the secret and the victim accesses an address depending on the secret when triggered.
Attacker (A) aims to obtain the secret memory address of the victim process, by checking the pat- terns of latency of memory accesses. An attacker may learn a policy to pick which memory addresses access, and observes the binary latency signal (slow/fast). The attacker can also trigger the victim process to execute, regain control after its execution, and guess the secret address of the victim if it confident do so. importantly, the attacker can only see the latency of its own accesses.
Detector (D) aims to raise the alarm as soon as possible when an attacker is present while avoiding false alarm for benign programs. As system process, we assume that the detector can observe memory accesses to the cache sets of all running processes in the environment. The detector will terminate an episode if alarm is raised.
See Appendix A.2 for detailed specifications of the observations, actions, and rewards.
In each episode, we may pick multiple agents different roles be in the environment and let them interact. In this work, we mainly test the following two possible scenarios:
Attack Scenario (DAV). The environment contains a detector, an attacker, and a victim. The attacker aims to obtain the secret address of the victim. The detector aims to detect the presence an attacker and terminate processes as soon as possible.
Benign Scenario (DBB). The environment contains detector and two benign programs with no malicious intent. In this case, the detector should not raise any false alarms.
We leave more complicated settings, such as scenarios with both victims and benign programs (e.g., DAVB) as future work.
4
METHOD
The CTA that we consider is a POMG with three fundamental characteristics: 1) Partial Observ- ability. In CTA, the attacker knows which program attack but only see the attacker's own actions and latencies, while the detector does not know if there any attacker nor which program the attacker is targeting. 2) Oparse-RRward Markov Game. The CTA game have long episode length. and agents have to come up with good action sequence before receiving the reward. Es- pecially, the attacker must learn both low-level skills perform attacks and high-level strategies to avoid defenders. 3) Environment Randomness. Such randomness comes from randomized victim secret addresses and the random trajectory sampling of benign programs. We propose our method based on the three crucial features.
4.1 OUR APPROACH: MACTA
In this paper, we introduce our approach, MACTA (Figure Right), as an initial solution to the
CTA challenge using MARL. MACTA adopts Transformers Vaswani et al. 2017b) the neural
encoder of policy nets, Proximal Policy Optimization (PPO) (Schulman et al. [017] as the policy learning algorithm, and Fictitious Play (FP)
(Brown) 1951 the game-theoretic tool.
4

---

Published as a conference paper at ICLR 2023
IBR-PPO
MACTA
Detector
Detector
Detector
Datector
Detector
Detector
Attacker
Attacker 2
Attacker
Attacker
Attacker
Attacker 3
learns policy against using PPO
Figure 3: Method. Iterated Best Respouse PPO (IBR-PPO) learns the best response to previous opponent only, while MACTA learns the best response to a uniform mixture of all historical opponents.
To deal with history-dependent partial observations and sparse rewards, both the attacker and the de- tector are equipped with policy nets with Transformer encoders. The Transformer encoder mainly composed of scaled dot-product attention and multi-head self-attentions. It can effectively integrate information from long time horizons and large-scale data while not suffering from vanishing or exploding gradients in recurrent neural networks (RNNs) (Parisotto et al. 2020
The attacker and the detector optimize their policies by the PPO algorithm to effectively learn a policy in the Markov game. Although independent reinforcement learning, where all agents are up- dating their policies simultaneously, is notoriously known for the instability issue training (Tan) 1993). if we only train agent at time and keep others stationary, then other agents can be taken as a part of the environment, and PPO can effectively optimize the policy for higher cumulative rewards. Iterated Best Response PPO (IBR-PPO) (Figure 3 Left) is the most naive way of imple- menting the above idea. It alternates the training of the attacker and detector so that they learn the best policy against the most recent opponent. However, may fall into the cyclic policy learning and never converge to any Nash Equilibrium (Roughgarden) 2010).
As widely accepted method in MARL, creating diverse pool of opponents and learning the best response to mixture of them can alleviate the eyclic issue and help with generalization. Similar to fictitious play in game theory, we create a pool for each agent and add their historical policies the pool. Concretely, for each iteration т, we denote the set of policies explored until T of agent by our method as IT; the opponents' joint policy set as IT,'. Then we learn the best response (BR), Ti (UIT,")) to the uniform mixture of the opponents' policy pool using best response learner (e.g. PPO), and add the best response to the policy pool. Mathematically, for each iteration
Vi II + II, U { (+((UU()))))
where -ř represents all players except for player i, and U is the uniform distribution. There are more advanced meta game frameworks like Double Oracle (McMahan et al.) 2003) and Policy Space Response Oracle (Lanctot et al. (017), which measure the meta game payoff matrix among different explored policies and solves the matrix for the best opponent mixture. In our case, since the environment contains some randomness, it is inefficient to precisely estimate the payoff matrix. We thus leave exploring more advanced game frameworks as future work.
The above components constitute our approach (Algorithm in Appendix A.7 MACTA alternates the training of attacker and detector every E epochs and adds one deterministic policy checkpoint of the learning agent the agent's policy pool every N epochs. During one agent's training, the agent faces uniform mixture of all opponents' past deterministic policy checkpoints. Note that we create such mixture by uniformly sampling policies from the opponent's policy pool each action step.
4.2 IMPLEMENTATION DETAILS
Specifically, we start with empty policy pools for both agents, first train the attacker for 50 epochs (each epoch contains 3000 training steps) to gain the basic skills of obtaining information from the victim program, and add one policy to the attacker's policy pool every 10 epochs. Then we stop the attacker's training and switch to train the detector against the pool of the first 5 attacker policies for 50 epochs. Similarly, the detector will have 5 policies by the end of this training iteration (50 epochs). The above process is repeated until the target training iterations (1800 epochs). We adopt an Actor-Criti implementation of PPO for both the attacker and the detector, and both the policy net and the value net are 1-layer 8-head Transformer encoders with different output heads. We leverage the MLMeta (Yang et al. 2022) learning framework for the PPO implementation, which is
5

---

Published as a conference paper at ICLR 2023
an asynchronous version of PPO with sampling and learning in parallel, and construct multi- agent learning framework on top of it. For stabilizing the self-play process, we also apply dual- clip PPO (Ye et al,, 2020). Refer to Appendix A.7 for more detailed description of training and environment hyper-parameters. Our code is available at http://ghhhbbbcmmcccooooosscaaas
5
EXPERIMENTS
5.1 EVALUATION SETUP AND METRICS
To evaluate the proposed MARL method, we compare with a few attacker and detector baselines. For attackers, we consider textbook attack Prime+Probe (Algorithm (2, an RL-based attacker (Au- toCAT) (Luo et al. 2023), , and the PPO with Iterated Best Response Oracle (IBR-PPO) attacker. For detectors, include our implementation of CC-Hunter (Chen & Venkataramani 2014) (Appendix A.8) and Cyclone (Harris et al. 2019) (Appendix (A.8), and IBR-PPO Detector.
In this work, we employ episode return and intuitive metrics including Attack Correct Rate, Attacks per Episode, Detection Rate, Episode Length, and False Alarm Rate. Details are listed in Table
Table Evaluation metrics.
Metries
Object
Description
Attack Correct Rate
Attacker
Measures the ability of attacker to infer a secret correctly (at- tack successfully). is the percentage correct guesses among all guesses aggregated over episodes.
Attacks per Episode
Attacker, Detector
Measures speed of attacker or the attacker's ability to by- pass detection or the detector's 's ability to prevent attacks. It is the average number of correct guesses per episode.
Detection Rate
Attacker,
Delection rate is the percentage of DAV episodes alarmed by the detector within the time limit the evaluated DAV episodes.
Detector
Episode Length
Attacker,
Measures how fast the detector can find out the existence of the attacker.
Detector
False Alarm Rate
Detector
Measures the false positive (terminate episode before time limit) rate of detector given all benign agents.
5.2 BENIGN DATASET
We the Standard Performance Evaluation Corporation (SPEC) 2017 benchmark suite (Bucek ett al. (2018) to represent benign programs, and obtain their memory access traces using the gem5 simulator (Binkert et al. 2011). We then generate benign traces by combining the memory cesses from two programs based on the simulation timestamps. We introduce the details of the Train/Val/Test dataset Appendix A.3
5.3 RESULTS
All the experiment results below reported an 8set-lway L1 cache. The attacker's memory address range is 8-15 and the victim's secret address is randomly chosen between 0-7. The episode length is 64 steps. To evaluate different methods, report the statistics based on three indepen- dent training instances for each learning- based method and control the final policies from different instances of method undergoing the same number of optimization steps.
5.3. ATTACKER PERFORMANCE
Attacker performance. Evaluation of the attacker's correct rate and number of attacks in 64-step episodes without detectors. Statistics are reported on three independent evaluations of 10,000 episodes.
Table 2:
Attackers
Prime+Probe
AutoCAT
IBR-PPO Attacker
MACTA Attacker
Metries
Attack Correct Rate (%) f
100.0 0.0
100.0
99.9 + 0.1 5.2 0.1
100.0 - 0.1
Attacks per Episode f
3.0 +
5.2 + 0.1
4.3 0.3
We first evaluate the attacker agent's performance in terms of attack correct rate and the number of attacks in an episode, to validate that the attacker agent is conducting effective attacks. shble that every attacker evaluated can achieve decent attack correct rate, indicating the agent acquires

---

Published as a conference paper at ICLR 2023
effective attack policies. In addition, the MACTA attacker has the smallest number of attacks per episode among the learning-bast based methods, because it learns to obfuscate itself as a benign program. Example attack sequences demonstrating the strategic attack behaviors can be found in Section A.4
5.3.2 HEAD-TO-HEAD EVALUATIONS
In this head-to-head evaluation, we have attacker play against detector from different training instances for 10,000 episodes and report the mean detection rate and the mean episode length for all attacker and detector pairs. The head-to-head evaluation results can be found in Table 3 and Table We also report the mean false alarm rate and the mean episode length of the detectors on unseen Benign agents in the last column of the table.
Table 3: Mean detection rate (%). Head-to-head evaluations with unseen opponents from different training instances. The higher the better for detectors when the opponent is an attacker, and the lower the better when the opponents are benign programs. '0' as Cyclone (SVM) is trained on Prime+Prooh
Opponents
Prime Probe
AutoCAT
IBR-PPO Attacker
MACTA Attacker
Benign J
Detectors
CC-Hunter (thold=0.45)
37.7 + 0.6
13.7 1.3
12.1+ 0.4
16.4 =
27.6 + 0.9
Cyclone OOns-Class SVM)
0.0 = 0.0
55.8 4.3
33.6 12.8
9,0
19.3 + 0,9
Cyclone (SV
(99.5 0.1)
0.0 = 0.0
0.0 = 0.0
0.1 0.1
1.4 0.2
IBR-PPO Detector
0.9 = 0.7
7.3 33+:20.55
6.4 15.6
8.4 21.9
0.5
MACTA Detector
97.8 + 0.9
99.9 0.2
99.6
31.2 18.5
0.2
Table 4: Mean episode length (steps). Head-to-her evaluations with unseen opponents from different training instances. The lower the better for detectors when the opponent is an attacker, and the higher the better when the opponents are benign programs. Cyclone and CC-Hunter both require a fixed episode length of 64 steps.
Opponents
Prime+Probe J.
AutoCAT +
IBR-PPO Attacker J
MACTA Attacker J
Benign
Detectors
IBR-PPO Detector
63.4 1
0.4
59.6 1 12.4
60.1 9.5
58.9 1 12,2
63.7 = 0.3
MACTA Detector
16.4 = 1.1
12.0
12.5 2.2
50.5 + 8.7
63.4 = 0.1
We find that the heuristic detector CC-Hunter cannot effectively discriminate the RL attackers from benign agents since the episodes are too noisy and too short to compute meaningful auto- correlations. Tuning the auto-correlation threshold only returns either high false alarm rate low detection rate. The anomaly detector, Cyclone (One-Class SVM), is more effective at detecting high-bandwidth attackers such as AutoCAT and IBR-PPO attackers, yet it struggles with detect- ing low-bandwidrs attackers like Prime+Probe and MACTA attacker and has a high false alarm rate. The SVM detector with Cyclone features is able to perform well (99.5% detection rate) on the heuristic attack (Prime+Probes that it is trained on, but has low detection rate on RL attackers. An- other drawback of these previous methods that they require fixed-length observation that longer than the steps needed complete attacks (usually 12 steps in this cache configuration). IBR-PPO falls into the cyclic policy learning issue; the detector is able to react well (98.3% detection rate) to the attacker that it is trained against but fails respond well to other attackers.
MACTA, however, is able to generalize to unseen attacks such as Prime+Probe and the IBR-PPO attacker. At the same time, MACTA also has low false positive rate and fast detection speed which prevents further information leakage. We hypothesize that MACTA can abstract the general pattern of the attackers from interacting with diverse attacker strategies during training.
On the other hand, since the detector is trained to block all the previous attack policies, the attacker had explore new policy space evade detection. The MACTA attackers are able evade a variety of unseen detectors. The above findings highlight the benefits of using MARL solution concepts in learning the detectors.
5.3.3 EXPLOITARILITY EVALUATIONS
We measure how a detector can be exploited by adaptive attackers, by fixing a detector strategy and training an RL exploiter (i.e., an attacker) against the detector by dual-clip PPO from scratch. The training curve of the exploiters of MACTA detectors can be found in Figure 4 As the training time of the MACTA detectors increases. it becomes more difficult for an RL attacker to bypass the detectors. Specifically, it will take the RL exploiter attacker longer to find a meaningful attack strategy. And even though the RL exploiter attacker can learn to attack eventually, the number of
7

---

Published as a conference paper at ICLR 2023
attacks per episode decreases from around 5.0 attacks per episode (No Detector) to about 4.0 attacks per episode (MACTA-18th) leading to about 20% reduction in a learning attacker's 's bandwidth. The decrease in the number of attacks can come from the slower attack speed (to reduce the chance of detection) or faster detection speed so fewer attacks can be performed.
Avainge Eptanic Niorkkr Cornect Rate
1.0
Numbor of Attacks par Eptsods
0.8
50
0.4
No Detector MACTA 8th MACTA-183
MACTAV99h MACTA-180
0.0
SOth
Epach
Epoch
Figure 4: Exploitability evaluation. We fix the detector policies (No Detector, detector of 9th and 18th fictitious play iterations in MACTA (MACTA 9th, MACTA- 18th)) train an RL attacker against the detectors from scratch. Left:: Average Episodic Attacker Correct Rate. Right: Attacker's Number of Attacks per episode.
5.4 NEURAL ARCHITECTURE STUDY
Trneetomerr
Transhrrrr
AIANN Spp
UETM
Shoat-2 Zar
Bayad-2ayyr
Episods Ketum
40
40
40
40
16
30
30
30
%
20
20
20
-10
-10
-20
-20
-20
Figure 5: A study neural architecturcs. We Transformer with 8-head attention and one Transformer encoder layer in MACTA experiments. Left two: Train attacker-only tasks using different neural architectures on two machines. Right two: Train attackers with different Transformer configurations on two machines.
Our CTA task is an example where neural architecture plays a critical role in learning a meaningful policy. We train attacker-only tasks using different network architectures different machines (de- tails in Appendix AA as shown in Figure For PPO attackers, MLP with residual connections (He et al., 2216b a) fails to achieve high episode return, while the Transformer and LSTM (Hochreiter & Schmidhuber 1997) networks succeed. For Transformers, our study shows that increasing the number of encoder layers in the Transformer can slightly improve the return but is less efficient in wall time. On the other hand, reducing the number of heads slows down learning. The above evi- dence suggests that the sequence modeling structure is critical for CTA attack policy learning. Our hypothesis is that successful attack is composed of a series of events, which may contain history- dependent relations among events, and Transformers can effectively model such relations. While the prior work (Luo et al. 2023) also shows that Transformers can be used for RI. CTA attacker, we provide more in-depth studies on different model architectures in this work.
RELATED WORK
Detectors for Cache Timing Attacks CC-Hunter (Chen & Venkataramani 2014) proposes de- tect cache-timing attacks using recurrent patterns generated during cache contention between attack and victim processes. More specifically, uses autocorrelation tween the two event trains. Replay Confusion (Yan et al.) 2016) records and deterministically replays a program's 's memory traces, changing the mapping Executing the traces in different memory addresses can expose abnormal access patterns observed between an attacker and a victim, which do not exist in benign traces. Cyclone (Harris et al.) 2019) uses cyclic interference from cache contention during attack. This detector assigns domain tags to processes, then uses performance counters to enumerate abnormal cache contention behaviors triggered by each domain tag. PerSpectron (Mirbagher-Ajone
detect periodic interleaving be-
cache addresses but retaining the cadences.
et al. 2020) trains neural network classifier using the memory and latency event logs generated from attack examples. The follow- up work EVAX MMirbagher-Ajorpaz al. 2022) improves the classifier accuracy using generative
8

---

Published as a conference paper at ICLR 2023
adversarial networks (GAN), Existing detectors based on known attacks cannot deal with evolving attackers. Our study shows that the RL attacker can learn novel strategies to bypass existing static detectors. MACTA solves this problem by enabling auto-discovery of attacker policies.
Game Theory in Security Games Game theory provides a framework for decision-making and strategy, modeling how selfish agents interact and affect system outcomes. In Stackelberg games, a defender must first commit limited resources to protect disparate locations and an attacker that subsequently targets locations, potentially having seen the configuration of defenses (e.g.. (Bier et al. (007)) ). Such games have masked systems from probes (Schlenker et al., 2018), defended systems against varied attack types (Thakoor et al., 22200), and assigned human analyst ysts to automated system alerts (Schlenker et al. (017). Whereas Stackelberg requires the defender to move first, we consider how the defender's policy should respond to the attacker's evolving policy. Game theory inspires GAN for security (Zolbayar et al. [021]] Baimukan & Zhu 2021). Unlike prior works that explore adversarial samples in the neighborhood of heuristic attack policy, our RL approach explores broader, unknown space of attack policies with a well-defined objective. RL is an instance of stochastic games, often modeled by a Markov Decision Process. Representative studies of such games for distributed systems include threat detection and resource allocation (Krishnamurthy al., 2007 Fan et al. 2019). To the best of our knowledge, we are the first to formulate a stochastic game for realistic, practical hardware timing attacks.
Population-based Multi-agent Reinforcement Learning Independent Reinforcement Learning in multi-agent environments suffers from the non-stationary opponent issue (Tan 1993). While It- erated Best Response methods alleviate the above problem by learning from stationary opponents: they tend to over-fit to other players' policies and cause cycles in policy learning (Vinyals et al.) 2019). Interacting with diverse opponent policies or heterogeneous agents is one effective way to avoid such cycles. copulation-based MARL is thus proposed to solve large-scale extensive form games by creating diverse pool of agents. Related work includes population-based reinforcement learning (Parker-Holder et al. 2020p, Neural Fictitious Self-Play (Heinrich & Silver 2016). Fic- titious Co-Play (Strouse et al., 2021), prioritized self-play (Vinyals et a 2019). Double Oracle (DO) (McMahan et al. 2003) and its gencralization Policy Space Response Oracle (PSRO) (Lancto) et al. 2017). The most closely related applications of population-based MARL to security games, such as those of Egghtsad et al. (2020) and Wang et al. (2019), use variants of Double Oracle, but they deal with different and less stochastic domains than ours.
7 CONCLUSIONS AND FUTURE WORK
Our work explores the application of multi-agent reinforcement learning in the cache timing attack and detection domain. We first introduce the environment MA-AUTOCAT that allows learning for both attackers and detectors, and their complex interactions with caches. Then we propose to combine the same-theoretie concept of Fictitious Play and Proximal Policy Optimization to train both agents (MACTA). Empirically, we found that the detector generated by MACTA can capture the general pattern of attacks and generalize to unseen attacks. The exploitability study of the detector also indicates the detectors down the attacks. On the other hand, the MACTA attacker is able to explore new policy space and mimic the benign agents to bypass the detectors. Finally, the neural architecture study demonstrates the strong representation-aablitt of Transformers.
impede the learning process of adaptive attackers and slow
Deployment in Real Systems We use cache simulator study CTA, but we believe the trained attacker and detector can be applied to real hardware with sufficient engineering efforts. For attack- Luo et al. (2023) demonstrate that an attack pattern learned in a cache simulator can be applied to multiple Intel processors. Similarly, we also show that the attack sequences from a MACTA attack can work on commercial processors in Appendix A.S] For the detectors, with hardware changes, the neural network model can be deployed inside a processor with a reasonable area and power overhead, as demonstrated by Mirbagher-AAorppaa et al. (2020).
Convergence of the Policies In MACTA, we adopt the Transformer-baser PPO algorithm as the policy learning oracle, so there is no guarantee that the algorithm will return the best response to the opponents in limited optimization steps. Meanwhile, little previous work discusses the convergence of Fictitious Play when it is used as a game- game--hherretic meta solver in the general-sum MARL setting. As training continues, we observe that the detector's 's ability to generalize slightly diminishes, indi- cating that it is forgetting some past attacks. We hypothesize it can relate to the convergence of one player's policy, which causes low policy diversity in the pool.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
This work at UT Austin has partially taken place in the Learning Agents Research Group (LARG). LARG research is supported in part by NSF (CPS-1739964, IIS-1724157, FAIN-2019844), ONR ARO (W91 INF-1922330)) DARPA, GM, Bosch, and UT Austin's 's Good Sys- tems grand challenge. Peter Stone serves as the Executive Director of Sony AI America and ceives financial compensation for this work. The terms of this arrangement have been reviewed and approved by the University of Texas at Austin in accordance with its policy on objectivity in research. Mulong Luo is partially supported by NSF grant ECCS- 1932501, and Geunbae Lee was partially supported by Commonwealth Cybersecurity Initiative. We thank Vishnu Kumar Kalidasan and John G. Harris at Virginia Tech for their technical support.
(N00014-18-2222
STATEMENTS
Ethical Statement Our work builds environment enabling automated exploration of both at- tack and defense policies of CTA. The attacker agent may used for developing new attacks maliciously. However, shown in the vast amount of attack papers in computer security venues, exploring and understanding security attacks is a necessary and important step developing future secure systems. Compared with prior works on security attacks real systems Liu et al. (2015); Oren et al. (2015); Kocher et al. (2019): Lipp et al. 20; Ravichandran et al. (2022), the MACTA agent only explores the attacks in cache without considering other system level activities. In addi- tion, our framework results a stronger detector, which can help design more secure systems.
Reproducibility Statement Our experiments are reproducible in reasonable range near the mean performance across multiple training instances reported. It is unavoidable that training with dif- ferent random seeds or on machines with different hardware will result different results, given the variance of explorations in reinforcement learning. However, our result is not a selection among multiple random seeds biased towards our benefits. We repeated the training for another three instances, giving similar result. We provide model checkpoints and publish the code at https///gthbbbboommmmcoooreeaaaa
REFERENCES
Ahmed H Anwar, George Atia, and Mina Guirguis. Toward a protected cloud against side channel attacks: A game-theoretic framework. In 2018 56th Annual Allerton Conference Communica- tion, Control, and Computing (Allerton), pp. 78-83. IEEE, 2018.
Nurpeis Baimukan and Quanyan Zhu. Concealment charm (concealgan): Automatic generation of steganographic text using generative models to bypass censorship. Game Theory and Machine Learning for Cyber Security, pp. 357-365, 2021.
Vicki Bier, Santiago Oliveros, and Larry Samuelson. Choosing what to protect: Strategic defensive allocation against an unknown attacker. Journal of Public Economic Theory, 94):5533-577, 2007.
Nathan Binkert, Bradford Beckmann, Gabriel Black, Steven Reinhardt, Ali Saidi, Arkaprava Basu, Joel Hestness, Derek R Hower. Tushar Krishna, Somayeh Sardashti, et al. The gem5 simulator. ACM SIGARCH computer architecture news, 39(2):1-7, 2011.
Samira Briongos, Pedro Malagón, José M Moya, and Thomas Eisenbarth. RELOAD+ REFRESH:
Abusing cache replacement policies to perform stealthy cache attacks. In 29th USENIX Security Symposium (USENIX Security 20), pp. 1967-1984, 2020.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint 1rXiv:1606..1555 2016.
George W Brown. Iterative solution of games by fictitious play. Act. Anal. Prod Allocation, 13(1): 374, 1951.
Noam Brown and Tuomas Sandholm. Superhuman ai for multiplayer poker. Science, 365(6456): 885-890, 2019.
10

---

Published as a conference paper at ICLR 2023
James Bucek, Alaus-Dieter Lange, and Jóakim v. Kistowski. Spec cpu2017: Next-generen com- pute benchmark. In Companion of the 2018 ACM/SPEC International Conference on Perfor- mance Engineering, pp- 41-42, 2018.
Jie Chen and Guru Venkataramani. Cc-hunter: Uncovering covert timing channels on shared proces- hardware. In 2014 47th Annual IEEE/ACM International Symposium Microarchitecture, pp. 216-228. IEFE, 2014.
Taha Eghtesad, Yevgeniy Vorobeychik, and Aron Laszka. Adversarial deep reinforcement learning based adaptive moving target defense. In International Conference on Decision and Game Theory for Security, pp. 58-79. Springer, 2020.
Richard Elderman, Leon Pater, Albert S Thie, Madalina M Drugan, and Marco Wiering. Ad-
versarial reinforcement learning in cyber security simulation. In ICAART (2), pp. 559-566,
2017.
Songchun Fan, Seyed Majid Zahedi, and Benjamin C Lec. Distributed strategies for computational sprints. Communications of the ACM, 02(2):98-1066 2019.
Yanan Guo, Xin Xin, Youtao Zhang, and Jun Yang. Leaky way: conflict-based cache covert channel bypassing set associativity. In 2022 IEEE International Symposium on Microarchitecture (MICRO), pp. 1458-1473. IEEE, 2022a.
Yanan Guo, Andrew Zigerelli, Youtao Zhang, and Jun Yang. Adversarial prefetch: New cross-core cache side channel attacks. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1458- 1473. IEEE, 2022b.
Austin Harris, Shijia Wei, Prateck Sahu, Pranav Kumar, Todd Austin, and Mohit Tiwari. Cyclone: Detecting contention-bus based cache information leaks through cyclic interference. In Proceedings of the 52nd Annual IEEE/ACM International Symposium Microarchitecture, pp. 57-72, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nieu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016, pp. 630-645, Cham, 2016a. Springer International Publishing. ISBN 978-3-319- 46493-0.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016b.
Johannes Heinrich and David Silver. Deep reinforcement learning from self-play in imperfect- information games. arXiv preprint arXiv:1603.0112/, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1785-1780, 1997.
Brian Johannesmeyer, Jakob Koschel, Kaveh Razavi, Herbert Bos, and Cristiano Giuffrida. Kasper: Scanning for generalized transient execution gadgets in the linux kernel. NDSS Symposium 2022, 2022.
Paul Kocher, Jann Horn, Anders Fogh, Daniel Genkin, Daniel Gruss, Werner Haas, Mike Hamburg, Moritz Lipp, Stefan Mangard, Thomas Prescher, et al. Spectre attacks: Exploiting speculative execution. In 2019 IEEE Symposium on Security and Privacy (SP), pp. 1-19. IEEE, 2019.
Vikram Krishnamurthy, Michael Maskery, and Minh Hanh Ngo. Game theoretic activation and transmission scheduling in unattended ground sensor networks: A correlated equilibrium ap- proach. Wireless Sensor Networks: Signal Processing and Communications Perspectives, pp- 349-388, 2007.
Mare Lanctot, Vinicius Zambaldi, Audrumas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. A unified theoreth approach to multiagent rein- forcement learning. Advances in neural information processing systems, 30, 2017.

---

Published as a conference paper at ICLR 2023
Moritz Lipp, Michael Schwarz, Daniel Gruss, Thomas Prescher, Werner Haas, Anders Fogh, Jann Horn, Stefan Mangard, Paul Kocher, Daniel Genkin, et al. Meltdown: Reading kernel memory from user space. In 27th USENIX Security Symposium (USENIX Security 18), pp. 973-990, 2018.
Fangfei Liu, Yuval Yarom, Qian Ge, Gernot Heiser, and Ruby B Lee. Last-level cache side-channel attacks are practical. In 2015 IEEE symposium on security and privacy, pp. 605-622, IEEE, 2015.
Mulong Luo, Wenjie Xiong. Geunbae Lee, Yueying Li, Xiaomeng Yang, Amy Zhang, Yuandong Tian, Hsien Hsin S Lee, and G Edward Suh. Autocat: Reinforcement learning for automated ex- ploration of cache-timing attacks. In 29th Sympisum High Performance Computer Architecture (HPCA), 2023.
H Brendan McMahan, Geoffrey J Gordon, and Avrim Blum. Planning in the presence of cost functions controlled by an adversary. In Proceedings of the 20th International Conference on Machine Learning (ICML-03), pp. 536-543, 2003.
Samira Mirbagher-Ajoraaz Gilles Pokam, Esmaeil Nohammadian-KKruyye Elba Garza, Nael Abu- Ghazaleh, and Daniel A Jiménez. Perspectron: Detecting invariant footprints of microarchitec- tural attacks with perceptron. In 2020 53rd Annual IEEE/ACM International Symposium Mi- croarchitecture (MICRO), pp. 1124-1137. IEEE, 2020.
Samira Mirbagher-Ajorpaz, Daniel Moghimi, Jeffrey Neal Collins, Gilles Pokam, Nael Abu- Ghazaleh, and Dean Tullsen. Evax: Towards practical, pro-active & adaptive architecture for high performance & security. In 2022 55th IEEE/ACM International Symposium on Microarchi- tecture (MICRO), pp. 1218-1236. IEEE, 2022.
National Academy of Engineering. 14 grand challenges for engineering of the 21st cen- tury.
2007. URL
wwwp://wwwwwc http://www..nn.eeeennnhhhaaaaaaa http:/hhhh....ooo....ccccaaaaaaa
cyberspace.asge
Olcksii Oleksenko, Bohdan Trach, Mark Silberstein, and Christof Fetzer. SpecFuzz: Bringing spectre-type vulnerabilities to the surface. In 29th USENIX Security Symposium (USENIX Se- curity 20), pp. 1481-1498, 2020.
Yossef Oren, Vasileios P Kemerlis, Simha Sethumadhavan, and Angelos D Keromytis. The spy in the sandbox: Practical cache attacks in javascript and their implications. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, pp. 1406-14188 2015.
Dag Arne Osvik, Adi Shamir, and Eran Tromer. Cache attacks and countermeasures: the case of AES. In Cryptographers' track at the RSA conference, pp. 1-20. Springer, 2006.
Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, et al. Stabilizing transformers for reinforcement learning. In International conference on machine learning, pp. 7487-7498. PMLR, 2020.
Jack Parker-Holder, Aldo Pacchiano, Krzysztof M Choromanski, and Stephen J Roberts. Effective diversity in population based reinforcement learning. Advances Neural Information Processing Systems, 33:18050-18062 2020.
Aleksei Petrenko, Zhehui Huang, Tushar Kumar, Gaurav S. Sukhatme, and Vladlen Koltun. Sam- ple factory: Egocentric 3d control from pixels at 100000 FPS with asynchronous reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Re- search, pp. 7652-7662. PMLR, 2020. URL http: httt:://procedings.mio wtt.m..p......///// petrenko20a. html
Zhenxiao Qi, Qian Feng, Yueqiang Cheng, Mengjia Yan, Peng Li, Heng Yin, and Tao Wei. Spectaint: Speculative taint analysis for discovering spectre gadgets. In NDSS, 2021.
Joseph Ravichandran, Weon Taek Na, Jay Lang, and Mengjia Yan. Pacman: attacking arm pointer authentication with speculative execution. In ISCA, pp. 685-698, 2022.
12

---

Published as a conference paper at ICLR 2023
Tim Roughgarden. Algorithmic game theory. Communications of the ACM, 53(7):78 86, 2010.
Gururaj Saileshwar, Christopher W Fletcher, and Moinuddin Qureshi. Streamline: fast, flushless cache covert-channel attack by enabling asynchronous collusion. In Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 1077-1090, 2021.
Aaron Schlenker, Haifeng Xu, Mina Guirguis, Christopher Kickintveld, Arunesh Sinha, Milind Tambe, Solomon Sonya, Darryl Balderas, and Noah Dunstatter. Don't bury your head in warn- ings: A game-theoretic approach for intelligent allocation of cyber-security alerts. 2017.
Aaron Schlenker et al. Deceiving cyber adversaries: A game theoretic approach. In International Conference on Autonomous Agents and Multi Agent Systems, 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.063477 2017.
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the Game of Go with Deep Neural Networks and Tree Search. Nature, 529(7587):4444488, January 2016. ISSN 0028-0836. doi: 10.1038/naturre16661
DJ Strouse, Kevin McKee, Matt Botvinick, Edward Hughes, and Richard Evereu. Collaborating with humans without human data. Advances in Neural Information Processing Systems, 34: 14515, 2021.
of the tenth international conference on machine learning, pp. 330-337, 1993. Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings
Omkar Thakoor, Phebe Vayanos, Milind Tambe, and Minlan Yu. Game theory for strategic ddos mitigation. 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017a.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, Fergus. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017b. URL htttps::/prooeeddnggggnooo/ neurips http://sr.rrrrro.paaeeeee/////// TEDe22334ccee1DDDsccccaaaaaaaaa
pdff
Pepe Vila, Pierre Ganty, Marco Guarnieri, and Boris Köpf. Cachequery: Learning replacement policies from hardware caches. In Proceedings of the 41st ACM SIGPLAN Conference on Pro- gramming Language Design and Implementation, pp. 119-532, 2020.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforeement learning, Nature, 575(7782)3350-33 2019.
Yufei Wang, Zheyuan Ryan Shi, Lantao Yu, Yi Wu, Rohit Singh, Lucas Joppa, and Fei Fang. Deep reinforcement learning for green security games with real-time information. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 1401-1408, 2019.
Wenjie Xiong and Jakub Szefer. Leaking information through cache LRU states. In 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA), pp. 152- IEEE, 2020.
Mengjia Yan, Yasser Shalabi, and Josep Torrellas. CoplaiConfusion: detecting cache-based covert channel attacks using record and replay. In 2016 49th Annual IEEE/ACM International Sympo- sium Microarchitecture (MICRO), pp. 1-14. IEFF, 2016.
13

---

Published as a conference paper at ICLR 2023
Xiaomeng Yang, Brandon Cui, Teng Li, and Yuandong Tian. RLMeta: A Flexible Frame- work for Distributed Reinforcement Learning, 2022. URL pttts://gitthut com/ seeebookresearcc/Elmeete
Yuval Yarom and Katrina Falkner. FLUSH+ RELOAD: A high resolution, low noise, cache side-channel attack. In 23rd USENIX security symposium (USENIX security 14), pp. 719-732, 2014.
Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo, et al. Mastering complex control in moba games with deep reinforce- ment learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 6672-66799 2020.
2020.
Bolor-Erdene Zolbayar, Ryan Sheatsley, Patrick McDaniel, and Mike Weisman. Evading machine learning based network intrusion detection systems with gans. Game Theory and Machine Learn- ing for Cyber Security, pp. 335-356, 2021.
14

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1
WHY STUDY CACHE TIMING ATTACKS
CTA are stealthy but powerful. They do not violate any access control policies enforced by the op- crating system and low-level hardware and they are shown to pose serious security concerns in prac- tice. For example, some implementations of security critical software such as encryption algorithms have secret dependent access pattern, and an attacker can use CTA to obtain secret keys (Os- vik et al. 2006 Liu al., 2015). CTA also enables covert communication channels between two domains and breaks the existing security isolation mechanism, e.g,, sandbox in javascript (Oren ct al. 2015), isolation between processes (Kocher et al., 2019), and the system privilege levels (Lipp et al. 2018). CTA can also facilitate brute forcing hash values stealthiyy without triggering excep- tions, which is shown to help break the ARM pointer protection mechanisms (Ravichandran et al.) 2022). One of the important defensive strategies is to detect unique characteristics of memory ac- cess patterns of attacker programs that are different from usual benign ones, as leveraged by the state-of-the-art tache-timing channel detectors (Yan et al., [016 Chen & Venkataramani, [2014] Harris et al. 2019 MirbagherrAAjopaaz et al. (200)). However, many new attacks (Briongos et al., 2020 Luo al. 2023) avoid the characteristies that the detector uses and it is hard adapt existing detectors to previously unseen attacks or access patterns.
A.2 ENVIRONMENT CONFIGURATIONS
Table 5: Environment hyper-yaramenn parameters.
Parameter Group
Parameter Name
Parameter Value
MA-AUTOCAT
Max Episode Length
64 steps
MA-AUTOCAT
Observation Window Size for the attacker and the detector
64 steps
MA-AUTOCAT
Probubility between Attack Scenario and Benign Scenario during Training
[505e, 50%)
MA-AUTOCAT
Benign Program Logs (Train)
48 Million Steps
MA-AUTOCAT
Benign Program Logs (Validation)
4 Million Steps
MA-AUTOCAT
Benign Program Logs (Teat)
40 Million Steps
MA-AUTOCAT
Attacker Memory Address Range
8-15
MA-AUTOCAT
Victim Memory Address Range
0-7
Cache Simulator
Cache Configuration
LI Cache, set way
Cache Simulator
Replacement Policy
Least Recently Used (LRU)
Game Mechanism. In MA-AUTOCAT, within a fixed- episode, the attacker agent can guess the secret address of the victim as many times as possible and get reward for every cor- rect guess (successful atluck). In the meantime, the detector agent can monitor the cache access history and interactions of two programs and decide whether to raise a flag/alarm to terminate the episode to prevent further information leakage.
Attacker's Reward Function. The attacker is punished by 0.01 for every time step, +10 if guess the victim's secret successfully, -10 if incorrectly. It will get one-time punishment of 20 if it reaches a timeout without any attack, a one-time punishment of -10 if identified by detector. The episode length to collect reward is affected by the detector.
Detector's Reward Function. The detector can raise a flag to terminate the episode. If the detec- tor raises a flag an attack scenario, then the detector receives reward for remaining steps [max step- current step]: if the detector raises a flag in benign scenario, then it receives a large penalty [5 max step]. If the detector lets the episode going, for benign scenario there is no punishment; while the detector gets -10 every time the attacker attacks successfully.
For each time step, the attacker can choose an action a E {a' ag aoy, dgy } where a&x represents access address X, at represents letting the victim access secret-related ad- dress, a represents letting the victim access a random address and dgy represents guessing the secret address to be Y.
Attacker's Action.
Detector's Action.
For each time step, the defender can choose ad E {atermo Looee) where aterm means terminate the episode and a means let the episode keep going.
15

---

Published as a conference paper at ICLR 2023
Attacker's Observation. The attacker's observation space includes history of attacker ac- tions and memory access latency it receives from the cache simulator. For each time step, a new step observation (sfar Suty Sact Stepp) is appended to the observation window, where fiat E {Shät, Smiss; ) represents the access latency, <be E {st, represents whether wait for the victim's 's action, Sact records the attacker's current action and Step is the current time step.
Detector's Observation. The detector can observe a history window of the memory ac- cess actions of both programs. For each time step, the new observation is composed of ad step)) where slat E {Shit, Smiss) represents the access latency (access la-
(slat) 8 "program Seer tency of all programs are visible to the detector), s *program E {8a,,5) indicates the identity of the program, seet represents the cache set being accessed at the current step, and Sstep represents the current defender time step.
Benign Programs. The benign programs share the same action space with the victim and the attackers, and their domain id is randomized per episode. The actions of benign programs sampled from pre-collected memory traces of the benign programs and we map the traces to the cache configuration (8 sets and way) in this paper. A detailed introduction the benign datasets we use and the detectors' false alarm rate comparison is in Section A.3
A.3 BENIGN DATASET
The memory traces of different benign programs are sampled from the Standard Performance Evaluation Corporation (SPEC) 2017 benchmark suite (Bucek et al., 2018). Specifically, we run each individual benchmark in the gem5 simulator (Binkert et al. (211T) and generate log files containing all memory accesses executed and their corresponding timestamps. To sample typi- cal memory access patterns, we follow the standard practice in computer architecture studies to skip the first 2 million operations which represent the warm-up phase of programs, and sam- ple the memory accesses in the middle of execution by skipping more steps. We collect mem- ory access traces from the following 10 SPEC benchmarks, of which names are in the form of (program id. progrrrammmaaae _amennaammmee www._aaaaeeeeeaaaeeeee speed/rate): 200.perlbench.r, 502.gec.r. 505.mef_r, 548. 548.exxhange2.r, 249.fotonik3d.r, 602.gec_s, w07.cccttBSSN... ss, 631. deepsjeng.s, 638. imaaicmaiskss, and 64 leellas. Since our benign scenario consists of two benign programs, we mix two memory traces based on the timestamp in the simulation using either different benchmarks or identical benchmarks with dif- ferent starting points. We prepare each benign trace to have 4 million memory accesses total. Note that there can be combinatorically many different traces of two programs, we only select representative subset of them for the training and evaluation. Finally, we project the memory access traces onto the valid action space given the current cache configuration (8-set, 1-way, cache line size of 64 bytes).
We randomly select three programs 5000.perleenchrr 502.gcc.r, 505.mcf.r) r),, and use different com- binations (with replacement and with different skip steps) of them as training set. For example, the trace file name 3550-2MM500044M means it contains two perlbench programs sharing the same cache. They start from different times (skipping the first 2 million memory accesses and skipping the first 4 million memory accesses, respectively) and continue until there 4 million memory accesses by either program. We generate the validation dataset 5449.ottonk3d..r 607. www.cccttSSSS s) and test dataset exchaexchange2.r, deeepssjnss, 638.668...... ama.inkg and 641. l.l.lecla_s in the same way as the training set.
The ML models (MACTA, IBR-PPO, Cyclone) are trained on the same training set and tuned on the validation set. In the meantime, CC-Hunter's threshold is selected based on the validation set. Since the memory traces can exhibit different behaviors, we provide our per-trajectov false positive rate estimate in Figure g All the machine learning models appear to perform better on the test set than the training set, which indicates a potential distribution shift between the training set and the test set. CC-Hunter's 's false positive rate is too high (ranging from 7.5 30%) to be plotted in the figure. In general, MACTA and Cyclone detectors have similar false positive rate on benign programs. The IBRPPO detector has lower false positive rates but it also has much lower detection rates. In addition, we closely inspect the benign traces that cause false positive. We find most of them variations of the Prime+Probe attack on subset of cache sets. This is because even though the benign programs do not have malicious intentions, but they can still generate small pieces of memory access patterns that happen to be attack pattern.
16

---

Published as a conference paper at ICLR 2023
10
CydoneeOnn- Class SVM)
Train Set
Val Set
Test Set.
IBRPPO Detector
100.0.0
18.0%
362%
16.4%
MACTA Detector
False Positive Rates (%)
500-2M, 500-4M
502-MM 902 4M
508 505-AM
500-2M, 502-2M
502-2M, 505, 205.2M
525
2M
596-aM 502-4M
362 6M, sts oM
sos-at ass-ad
sep rort 5522 10M
Soo Son-1rM 1oM, 5os, 10M
502-100g 1iM sos. 10M
S15-MM 607 24444
SKi--M se8
638-aM. 536.4M
6l1 .M
DrM.aM. 54 1-4M
548-2M, 608-23 2MM
681 2M
54th 64 6-1
631 -aM 638 2M
ed% a.M, 641 ---M
638-2M 541 541-2M
56n-2M
5in
est
s4lk
Figure 6: False positive rates on different datasets, report per-dataset mean false positive rate for models. CC-Hunte((tneesholdO---) false positive rates are too high to be included here.
100
100
80
80
60
60
Recall
Recall
40
40
MACTA
MACTA
IBR-PPO
IBR-PPO
CC-Hunter
CC-Hunter
Cyclone(SVM)
Cyclone(SVM)
Cyclone(One-Claas SVM)
Cyclone(One-Class SVM)
20
False Positive
False Positive
Figure 7: The relative positions of all detectors' performance on the ROC figure. The recall shown for the Prime+Probe attacks (Left) and the AutoCAT attacks (Right). The false positive rate is measured on the proposed test benign datasct. Here, Cyclone is trained Prine+Probe attack sequences. But we did not provide the Prime+Probe attack sequenees MACTA detector explicitly.
A.4 TRAJECTORY ANALYSIS
Figure 8 fillustrates different attackers' attack sequences given fixed secret bit. Here, we use victim secret=5 as an example. The top row shows sampled pattern of benign programs. In that case, the two programs act independently and alter the access to the cache frequently. The Prime+Probe attacker causes cache contention by accessing the cache frequently, and only invokes the victim to access the cache when needed. Once contention with the victim in one cache set is observed (i.e., cache miss after the same address is accessed by the attacker), the attacker will make guess without more memory accesses. The IBR-PPO attacker takes a similar strategy as AutoCAT's, s, but it learns to insert some extra victim invocation steps confuse the detector. The issue with the extra invocation strategy that the victim only accesses its secret bit, which can be easily captured by the detector. The MACTA attacker, however, leams advanced strategies. learns to invoke random victim
17

---

Published as a conference paper at ICLR 2023
Cache Miss Observed by the Attacker
Cache Hit Observed by the Attacker
Victim Random Access Not Observable to the Attacker
Cache Set Access Not
3
3
3
Observable to the Attacker
Time Step
14
15
18
Benign
Program
5
7
Program
2
7
S
Victim Secret
Correct Guess
Victimm Reset 3
Prime + Probe
Attacker
6
3
a
Victim
5
Victim Secret
Correct Guess
Victim Reset
AutoCAT
Attacker
0
2
3
2
5
3
4
2
6
Victim
5
5
Victim Secret
Correct Guess 5
Victim Reset
IBR-PPO
Attacker
7
3
6
4
2
2
0
Victim
Correct Guees 5
Victim Reset
Corract Guess 3
MACTA
Attacker
5
5
Random
Figure 8: Example trajectories of different attackers and benign agents in 8-set 1-way I.I cache. The number indicates the cache being accessed. Red and green boxes show the observation the attacker. The latency other programs (i.e., victim benign) cannot observed by the attacker, but they can observed by detectors. The program IDs are randomized during training, and the attacker can be any of the two programs in the system. The cache is initialized with random states.
accesses to alter its behavior to be more similar to benign programs. Note that invoking random accesses from victim cause noise in the attacker's latency observation and make the steps needed for successful attack longer. This means that to evade the MACTA detector, just inserting some victim invocation steps is far from enough. The attacker has take risk invoke random accesses from victim instead because the "easiest" policy space has been exhausted.
A.5 REAL HARDWARE ANALYSIS
Table 6: Attack evaluation on commercial processors. We report the attack correct rates of MACTA attack sequences three commercial Intel processors for 10,000 episodes, MACTA attackers achieve > 99.9% correct the simulator, and still 99% real hardware.
CPU
Cache Level
#Sets
#Ways
Attack Correct Rate f
17-6700 (SkyLake)
L3
(pprtitioned))
100.00%:
17-7700k LKabyLake)
L3
(pprtitioned)
99.97%
evaluated the effectiveness of the attack sequences produced by MACTA attackers on real hard- ware by running them two commercial processors through CacheQuery (Vila et al.) 2020). The attack sequences are generated from a MACTA attacker that is trained using a simulated environment for 8-set 1-way cache configuration. Then, the attack sequences performed real hardware to obtain the attack correct rate. Table demonstrates the attacker policy from the simulator can be transferred real hardware with negligible discrepancy.
18

---

Published as a conference paper at ICLR 2023
A.6 MODEL ARCHITECTURES
In Section 5.4 we compared three different neural network architectures: Transformers (Vaswani
al. 2)) LSTM (Hochreiter & Schmidhuber 1997). and MLP. All of the methods are controlled at the same scale of parameters and trained with PPO (Schulman et al. 2017) without dual-clip (Ye al. 2020) on two different machines. The details about the model architectures are listed below.
Transformer. In our experiments, all the policies use 8-head 1-encoder-layer Transformer with model = 128 and feedforwoer = 2048. For the model architecture study, we study the changes in the number of heads in the multi-head attention mechanism, and the number of Transformer Encoder layers. Similar Luo et al. (2023), we apply an average-pooling to reduce the step dimension.
LSTM. We employed --layer LSTM with hidden dimension of 256. The input to the LSTM is the embedding of the pre- pre-padding history of the observation. We concatenate the hidden and cell states of the last step and use it the sequence embedding.
MLP. The MLP model we used directly feeds the input embeddings into pre-activation residual blocks (He et al. 2016b a) with hidden dimension 128. Each residual block is composed of 2 ReLU- Linear layers with residual connection.
We obverse that the MLP model fails even when tested multiple different machines while Trans- former and LSTM models both work. The learning speed of Transformer and LSTM models can be different. Our hypothesis is that the CPU/GPU configurations of different machines may affect the policy lag (Petrenko et al., 2020) of Asynchronous PPO training, which may lead to different learning speed for different models.
A.7 ALGORITHM AND TRAINING HYPER- HYPER-PARANN PARAMETERS
The MACTA algorithm is explained in Algorithm I and detailed training hyperparameters can be found in Table 7 In MACTA, the policy pool is created by sampling new model to do batch actions per step. It is an infrastructure implementation to produce faster sampling speed, so not strictly per-step sampling, but is per-step sampling considering large amount of data. Additionally, we are exploring the per-trajectory policy sampling because it has nice theoretical properties. Nevertheless, it needs further infrastructure support.
Algorithm MACTA
1: Initialize Number of Fictitious Play Iterations 1, Attacker Policy Pool PA, Detector Policy Pool PD. Number of Epoch per Fictitious Play Iteration E, Add a policy to Pool per N epochs. PPO the Proximal Policy Optimization. U the uniform random sampling of policies per step.
6 0.j <- 0, k
2:
while do
3:
--0 while j E do
4:
5:
*A)=PPO(U(PD))
b Train attacker policy against the pool of the detectors
6: if mod N- 1= then
7:
PA + PA U TTj end
b Add an attacker checkpoint to the attacker pool
8:
9:
j j+ j +1 end while
10:
11:
0
12:
while j < E do
13:
TD}=PPO(U(PA)
Train detector policy against the pool of the attackers
14:
if j mod N- 1= then Pn Pp TDj
> Add detector checkpoint the detector pool
16:
end
j j+j+ +1
18: end while
end while
20:
return TA. TD
b return the last policy of attacker and detector
19

---

Published as a conference paper at ICLR 2023
Table 7: Training hyper-paramette for MACTA.
Parameter Group
Parameter Name
Parameter Value
Fictitious Play
Fictitious Iterations
18 iterations
Fictitions Play
Epochs per Iteration per Agent
50 epochs
Fictitious Play
Truining Steps per Epech
3000 steps
Fictitious Play
Frequency of Adding one Policy to Pool
10 epochs
Computing Resource
Number of Sampling Actors
72 Actors
Computing Resource
Sampling Instance per Worker
Actors / Worker
Computing Resource
Remote Model Push Frequency
10 steps
Computing Resource
GPU Information
Nvidia Tesla V100 16G / 32G
Computing Resource
CPU Information
80 Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz
Proximal Policy Optimization
Replay Buffer Size
262144
Proximal Policy Optimization
Training Batch Size
512
Proximal Policy Optimization
Learning Rute
lo-4
Proximal Policy Optimization
Entropy Coefficient
0.03
Proximal Policy Optimization
Discount Factor
0.99
Proximal Policy Optimization
Max Gradient Norm
1.0
Proximal Policy Optimization
GAE X
0.95
Proximal Policy Optimization
Policy Ratio Clipping r
0.2
Proximal Policy Optimization
Value Clipping
0.2
Proximal Policy Optimization
Value Loss Coefficient
Proximal Policy Optimization
Dual-Clip Threshold
Model Architecture
Number of Transformer Encoder Layers
Model Architecture
Trunstorner d model
128
Model Architecture
Transformer nhead
Model Architecture
Trunsforner dim feedforward
2048
Model Architecture
Transformer dropout
0.0
A.8
HEURISTIC CACHE TIMING ATTACKS AND DETECTORS
A.8.1 HEURISTIC ATTACKER ALGORIHTM
Prime+Probe (Algorithm 2 Osvik et al. 2006). First, in the prime phase, the attacker fills the cache set with its address value (lines 3) in a randomized way, then waits for the victim to access the cache set. Next, the victim accesses one of the cache sets, then replaces the loaded address value with its address (lines 5). Lastly, in the probe phase, the attacker accesses the cache sets again in a random permutation order, then measures the cache latency to load each set of the primed address value (lines 7 to 8). In a cache set accessed by the victim, the attacker observes increased latency (cache miss) and makes a guess.
Algorithm 2 Prime+Probe Attack
1: step < step +
2: if slep < len(allacker address range) then
3: action = sarrime_addrrssss (slep, cache_size)
> attacker fills cache by attacker's address
www.aaaaeeeeerrraaaaaaaaaaaeeee 4: else if step len(attacker saddress address_range) then
5:
action = www.tgeeg.r..ctillee.......... .victim(step))
> victim accesses a cache and fills its own address
6: else
7:
action
cache
> attacker access caches again
8: measure latency(action))
9: end if
10: if latency- =1 1 then
D attacker observes for any cache miss 11: action guess cache_size) >> attacker makes guess victim's 's secret address
12: end if
13: Return action
20

---

Published as a conference paper at ICLR 2023
A.8.2 DETECTOR ALGORITHMS
CC-Hunter Chen & Venkataramani 2014). Cache timing channels rely on the latency of events perform timing modulation. To send information, two processes (i.e., the trojan and the spy) generate a sufficient number of alternating conflict events (cache misses) allow the adversary to decode the transmitted bit based on the average memory access times (hit/miss). Those behaviors show periodic, oscillating patterns of conflicts between the two processes. Therefore, autocorrelation used to identify those patterns. Autocorrelation is the correlation coefficient of the signal with limelaageed version of itself, along with measuring the event train X, as variable a time instance of t. Two cases of conflict miss, i.e., either the victim eviting the attacker's cache line the attacker evicting the victim's is cache line, are considered for the event trains. For example, we can check the autocorrelation Cp time lag p, which is expressed as:
Cp = Di [[x; X) -X++=xX)
Σ (X, (X²
If there exists time lag p which p S P. where P is predefined parameter such that makes Cp larger than threshold value, then is assumed as an attack.
We tune the threshold be p 0.45 alarm rate and 38% detection rate on Prime+Probe. However, this threshold fails to generalize to the test set, giving us a 27% average false positive rate, as reported in this paper. The main issue with applying CC-Hunter to our environment is that our episode length is short, and the cache is initialized with random loads after resetting. As result, even attackers' latency histories can have low auto-correlations.
validation set, and this threshold yields 7.5% false
Cyclone Harris et al. (2019). The concept cyclic interference commonly found all known cache contention side-channe attacks and has been used for detecting those attack patterns. Interfer- ence occurs from the attacker to the victim process or vice versa, considered directional, and affects the behavior of microarchitecture in disruptive manner. The cyclic interference be noted (a ~. ~* a), where interference (a ~* b) is followed by (b ~. a). However, interference including a third party between attacker and victim, like distinguish attack and benign patterns, Cyclone uses a one-class support vector machine (One-Class SVM). our experiments, however, we found that the one-class SVM is not effective in detecting our Prime+Probe implementation and has a high false positive rate under our testing configuration. Consequently, we also experimented with an SVM trained to classify Prime+Probe attack and be- nign traces, the Cyclone paper suggests the feature can be used for other classifiers.
b ~+ c). is not considered cyclic interference. To
(c) IBR-PPO
(d) MACTA
(e) 631-2M
(1) 631--M
(9) 502-2M 505-2M
(h) 549-2M
Prime+Probe
AutoCAT
631-4M+31
631-4M
607-2M
Figure 9; Example Cyclone features for various scenarios: (a. b, c. d) represent typical features when attackers interact victim; (e. g, h) depict typical features resulting from interactions between benign programs. The feature value in the grey areas is O, and the intensity the blue color indicates the frequency cyclic inference, with darker shades representing more frequent occurrences. (c) illustrates the interaction between program 63 dewpppeengg......... starting 2 million (M) steps and the same program 4M+31 steps. (f) demonstrates trace of 631 .deepsjeng_s self-mix from the test (g) shows trace from the training set, and presents sample trace from the validation set. Typical test set features are similar to those of the train validation set.
To extract the programs' cyclic features, we use 8 buckets and observation window of 17 steps (which the same as Prime+Probe's frequency). We train both Cyclone (One-Class SVM) and Cyclone (SVM) models with Gaussian kernels using 24,000 normalized train samples until conver- gence. For Cyclone (One-Clate SVM), we choose the upper bound on the fraction of training emrors and lower bound of the fraction of support vectors be %, with all training data generated from
21

---

Published as a conference paper at ICLR 2023
the benign (DBB) scenario. For Cyclone (SVM), the regularization coefficient C' is set to 100, with 50% the train data from benign scenarios, and 50% of them from malicious (DAV) scenario.
We observed that Cyclone (One-Class SVM) fails detect Prime+Probe attacks and exhibits high false positive rates on datasets where two programs exccute identical tasks. This inability to de- tect Prime+Probe is primarily due to the low bandwidth of the implemented attack. Regarding the elevated false positive rates, several factors contribute to this issue. One reason is that the cache con- figuration and system setting we employ could potentially bring high cyclic behaviors even between two benign programs. In addition, we discovered that when two benign programs with periodic memory access patterns are mixed, the difference in the starting time of the two programs could significantly impact the cyclic behaviors between the programs. Notably, for 631-2M 631-4M in
the test set, Cyclone (One-Class SVM) exhibits 100% false alarm rate (Figure f) however, we shift the start time of the program, the false alarm rate decreases to 0% (Figure )))) Cyclone (One-Class SVM) also demonstrates fluctuations with the program start time on the 641 self-mix but is generally more stable on other program combinations. In contrast, found that the MACTA detector and Cyclone (SVM) are not sensitive to the program start-time offsets or combinations of programs, consistently showing low false alarm rates across various test sets and offsets. For fair comparisons, we report the statistics of all methods on the same test set without counterfactual dataset selection, but we encourage the readers be mindful of the potential bias.
22


Reviews:
[{"review_id": "qHL7t42iE4", "reviewer": "Reviewer_9uNr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Cache-timing Attack (CTA) endangers privacy and security of data. However, traditional detection methods are not extendable. This work tackles CTA and detection from game-theoretic perspective by proposing a deep multi-agent reinforcement learning (MARL) approach with transformer for attackers and detectors.", "summary_of_the_review": "Instead of heuristically iterating each possible CTA by human experts, this work provides a novel view from game-theoretic (to be specific, RL) perspective to address the issue. Compared to ML approach, this work does not simply learn behaviors of possible CTA, but it allows attackers and detectors to interactively develop accordingly. Although this work is lack of some theoretical explanations on MARL for CTA, the experiments themselves look good if they are not cherry-pick and it still worths to consider as an application work. I would like to hear from authors especially on W2 and W3, and I will also consider other reviews to make the final decision later.", "strength_and_weaknesses": "Strength:\n\n1. Converting detection of CTA problem into a multi-agent reinforcement learning is a fresh perspective to consider attack and detection in the area of data security and privacy.\n\n2. This work builds CTA-related gym for multi agents to benefit detect-and-terminate defense strategy.\n\nWeakness:\n\n1. Since this work is based on game theory, it is good to provide a theoretical formalization on the problem and related proofs. Otherwise, it looks less sound and people may wonder the ability of this approach.\n\n2. This work does not address explore-exploit tradeoff very well since it is usual in common RL scenario.\n\n3. Although transformer is popular and good for many sequence learning tasks, this work does not explain enough on why transformer works well in the MARL for CTA. Authors may try to consider advantages of transformer for CTA in MARL from universal-approximation perspective like “ARE TRANSFORMERS UNIVERSAL APPROXIMATORS OF SEQUENCE-TO-SEQUENCE FUNCTIONS? ([https://openreview.net/pdf?id=ByxRM0Ntvr](https://openreview.net/pdf?id=ByxRM0Ntvr))” published on ICLR 2020.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "m4FI2JRFBGa", "reviewer": "Reviewer_aHoC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper formulates cache timing channel attack and defense as Partial Observable Markov Games (POMGs). Evaluation shows that the attack/defense agents trained from POMG achieve better generalization against unseen detectors/attacks compared to other works. The authors also discover that applying Transformers as the neural encoder of policy nets performs significantly better than MLPs.", "summary_of_the_review": "The paper definitely studies an important problem. However, overall I feel this paper does not contribute much on the intellectual side compared to prior work and the current evaluation is thin.", "strength_and_weaknesses": "Strength:\n\nInteresting problem space to apply RL\n\nWeaknesses:\n\nIntellectual difference between AutoCAT [Luo et al., 2022] and MACTA is small. The paper overall is an extension of AutoCAT where detector is formulated as part of the RL problem. On this end, the formulation presented in this work is incremental to Luo et al.\n\nThe evaluation is thin. I like the neural architecture study. However, as a security focused paper, I look forward to more security analyses providing insights that why specific attack/defense succeeded or failed. Perhaps cases studies as in Luo et al. can help.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "-gg_myjy0g", "reviewer": "Reviewer_Mrpg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "(before I begin, I report that he text of the paper contains hyperlinks to external URLs (e.g., “14 grand challenges of engineering”). I invite the authors to remove these and report the links either as footnotes or actual references, as it is not possible to appreciate them by those people that read the paper on printed-paper (aside from representing a security risk, as the URL is hidden). )\n\n\nThe paper tackles the problem of cache timing attacks (CTA) detection. Specifically, the paper proposes MACTA, a method that leverages “Multi Agent Reinforcement Learning” (MARL) to simultaneously launch CTA, and detect them. Experiments on a synthetic testbed show that the attacks “learned” by the “attacking agent” developed through MACTA can bypass existing detection methods; whereas the “detection agent” developed through MACTA can not only detect the attacks of the “attacking agent”, but also (to some degree) those tailored for detection methods employing handcrafted heuristics, or traditional supervised learning techniques. \n", "summary_of_the_review": "I liked reading the paper, and it was very hard to find a “recommendation”. On the good side, the paper tackles an underinvestigated problem (at least in the context of Machine Learning for Cybersecurity), and the experimental evaluation is interesting enough to (potentially) inspire future work in this domain. On the bad side, the paper falls short in providing a significant advancement in the context of machine learning, some results are underwhelming/expected, and the testbed is entirely simulated.\n\nPut simply, I think this paper would be **perfect** as a workshop paper, but it is not of good enough quality for ICLR. I recommend a “borderline reject”. I will provide below an explanation of all the major issues that affect the paper, alongside a set of questions and potential improvements. I look forward to reading an “enhanced” version of this paper. Furthermore, I also stress that I will gladly engage in a discussion with the authors.\n\n\n**Unfair novelty claim.**\nAccording to the introduction, “our work is the first to show a real-world hardware security problem can be addressed by jointly improving both attacks and detection.” This claim is unfair: using reinforcement learning to both “attack” and “defend” a security system is not novel at all. For instance, Apruzzese et al. [A] (published in 2020) do exactly this. The fact that this paper shows this for a “hardware” security problem is not very relevant to support a novelty claim. (especially because the experimental evaluation is done on a simulation, and not on “real” hardware)\n\n**Lack of a true “realistic” use case.**\nAccording to Section 3, the paper will ultimately consider two cases: one in which the environment includes a malicious entity (in which the goal is *detecting the malicious actions*); and one in which the environment includes only benign programs (in which the goal is *avoiding the generation of false positives*). The paper clearly states that “We leave more complicated settings, such as scenarios with both victims and benign programs as future work.” While I appreciate the transparency, I do not think that such “omission” can be simply overlooked: a common issue in *real* detection settings is when the environment includes both benign and malicious activities, wherein the latter are typically a “needle in a haystack”. I acknowledge that these may be “hard” to model, but such a lack is a significant limitation of this paper – especially given that it is submitted to a premier venue such as ICLR. For this reason, the real-world impact of the proposed method is difficult to gauge.\n\n**Misleading (claimed) advantage.**\nThe proposed method, MACTA, ultimately attempts to overcome the limitations of “human-defined” heuristics (quoting from the abstract: “However, the current detection of cache timing attacks relies heavily on heuristics and expert knowledge, which can lead to brittleness and inability to adapt to new\nattacks.”). However, since the entire RL environment (i.e., action space, reward, observation) is designed also by humans, wouldn’t this simply move to problem to a different “setting”? In other words: the RL environment is *also* designed by humans by using their expert knowledge. At the same time, an “improper” generation of the environment can lead, to an agent that can (synthetically) learn devastating attacks (and to a detector that can detect such attacks) but which may be not realistically conceivable. In turn, this can also lead to blind spots, i.e., if the environment does not take into account all the factors that can be realistically modeled (this is an implicit limitation of RL approaches).\nPut simply, I think that the authors should tone down the way MACTA is presented.\n\n**Questions/issues on Abstract, Introduction and Section 2.**\n\n•\tThe abstract is filled with subjective and vague terms to present the improvement of the proposed method. For instance, “substantially outperform” should be quantified in an objective way; “more stealthily” should also be numerically quantified. Finally, “less exploitable” is also unclear.\n\n•\tIn both the Abstract and Introduction, the text reports a very vague statement: “The experiments show that learned policies trained with MACTA can generalize to unseen detector/attackers.” Specifically, what does the term “unseen” stand for? Does it mean attacks that have not been “seen” before the operations performed by the RL agent? In which case, it is “obvious” that the learned policy can also detect such attacks; conversely, if the policy can detect attacks that are not seen “even during the learning phase of the agent”, then the result would be slightly more impressive (note, however, that [A] also showed a similar finding). \n\n•\tWhy is the environment called “MA-AutoCAT”? Shouldn’t it be “MA-AutoCTA”?\n\n•\tI think Section 2.1 is redundant. It presents information that is ultimately not necessary to understand the contribution of the paper, while at the same time not providing enough details to fully allow a non-expert reader to understand the issues of CTA. I invite the authors to remove such Section (perhaps putting it in an Appendix, and maybe expanding it). The current Section 6 can replace Section 2.1, and the space obtained by such “cut” can be used to expand on the technical contribution.\n\n•\tOn this note, something I’m curious about is whether anything has been done to counter CTA not with RL, but with GANs – given the many overlaps between these two techniques.\n\n**Questions/issues with Sections 3 and 4 (method):**\n\n•\tSection 3 “In our environment, the secret is reset after the attacker’s attempt to guess the secret and the victim will access an address depending on the secret when triggered.” Why? Please provide an explanation for this choice. If it is an original idea, then a description and a motivation is necessary. If it is drawn from well-known practices, then such practices should be referenced.\n\n•\tFigure 1 (caption): “Cache timing channel attack is formed when the attacker process and the victim process map their memory to the same system cache.” Shouldn’t it be “same address of the system cache”? Clearly, the attacker and the victim “must” use the same cache for a CTA to be conceivable…\n\n•\tSection 3: “Detector (D) aims to raise the alarm as soon as possible when an attacker presents while avoiding a false alarm for benign programs.” Presents should be “is present”?\n\n•\tSection 4: “The CTA is a POMG with three fundamental characteristics:” this is unclear. Does it mean that “any CTA is a POMG”, or that “the CTA we consider is a POMG”?\n\n•\tSection 4: “In CTA, the attacker knows whom to attack but can only see its own actions and latencies”. It is unclear whether “its” refers to the attacker, or to the target (“whom”).\n\n•\tSection 4: “Especially the attacker must learn both low-level skills to perform attacks and high-level strategies to avoid adversaries” are the “adversaries” represented by the “detector”? In general, avoid using “attacker” and “adversaries” with different meanings in the same sentence.\n\n**Questions/issues on Experiments**\n\n•\tSection 5.2: the experiments are carried out by simulating an 8set-1way L1 cache. Modern CPUs have far “larger” caches. Why did the authors consider such a simplistic setting? I believe the overall feasibility of the approach has deep connections to the architecture of the cache, and choosing such a simplistic one may (unfairly) induce favorable results for MACTA over prior baselines.\n\n•\tWhy is it that, in Table 3, the “IBR-PPO Detector” has 0 detection rate against ALL attacks (including the IBR-PPO attack)? Furthermore, it is unfair to “remove” the perfect detection rate of Cyclone from Table 3. Moreover, also concerning Cyclone, I do not agree with the statement that “fails to detect RL attackers”: its detection rate is 20%, which is only 5% less than the proposed MACTA; at the same time, Cyclone exhibits a much better false positive rate than MACTA (4.2 vs 11.3): in these cases, Cyclone is much better than what the authors describe.\n\n•\tIn Section 5.2.1, the paper states that “MACTA has low detection speed”: where is this reported? Note that I do not refer to “epochs” but to actual “time”. Furthermore, what the paper does not report (afaik) is the amount of computational resources required to develop (i.e., write code and train) MACTA: indeed, I am inclined to believe that implementing MACTA requires a huge effort – potentially superior than the one used to develop, e.g, Cyclone.\n\n\nEXTERNAL REFERENCES\n\n[A]: Apruzzese, Giovanni, et al. \"Deep reinforcement adversarial learning against botnet evasion attacks.\" IEEE Transactions on Network and Service Management 17.4 (2020): 1975-1987.\n", "strength_and_weaknesses": "\nSTRENGTHS:\n+ Simple, but interesting methodology\n+ Easy to understand\n+ The experiments consider several baselines\n\n\nWEAKNESSES\n- Poor novelty\n- Lack of a “realistic” use case\n- Misleading advantage\n- Everything is simulated\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster