Fast DL-based Simulation with Microarchitecture Agnostic Traces and Instruction Embeddings

Authors: santosh  pandey
amir  yazdanbakhsh
hang  liu
Venue: iscaconf_org_ISCA_2024_Workshop_MLArchSys
Published: 2024-05-13 18:53:40

Abstract: Microarchitecture simulators are indispensable tools for microarchitecture designers to validate, estimate, optimize, and manufacture new hardware that meets specific design requirements. While the quest for a fast, accurate and detailed microarchitecture simulation has been ongoing for decades, existing simulators excel and fall short at different aspects: (i) Although execution-driven simulation is accurate and detailed, it is extremely slow and requires expert-level experience to design. (ii) Trace-driven simulation reuses the execution traces in pursuit of fast simulation but faces accuracy concerns and fails to achieve significant speedup. (iii) Emerging deep learning (DL)-based simulations are remarkably fast and have acceptable accuracy, but fail to provide adequate low-level microarchitectural performance metrics such as branch mispredictions or cache misses, which is crucial for microarchitectural bottleneck analysis.
Additionally, they introduce substantial overheads from trace regeneration and model re-training when simulating a new microarchitecture. 

Re-thinking the advantages and limitations of the aforementioned three mainstream simulation paradigms, this paper introduces TAO that redesigns the DL-based simulation with three primary contributions: First, we propose a new training dataset design such that the subsequent simulation (i.e., inference) only needs functional trace as inputs, which can be rapidly generated and reused across microarchitectures. Second, to increase the detail of the simulation, we redesign the input features and the DL model using self-attention to support predicting various performance metrics of interest. Third, we propose techniques to train a microarchitecture agnostic embedding layer that enables fast transfer learning between different microarchitectural configurations and effectively reduces the re-training overhead of conventional DL-based simulators. TAO can predict various performance metrics of interest, significantly reduce the simulation time, and maintain similar simulation accuracy as state-of-the-art DL-based endeavors. Our extensive evaluation shows TAO can reduce the overall training and simulation time by 18.06$\times$ over the state-of-the-art DL-based endeavors.

Main paper: 
 i. introduction: 
 Since its inception, microarchitecture simulators rapidly become the most commonly used tools in computer architecturerelated research (see the report [37] ). As of today, computer architecture simulation is the textbook standard and virtually used in any architecture explorations, e.g., design space exploration [18] , [22] , [23] , [40] , microarchitectural bottleneck analysis [5] , [14] , workload characterization [16] , [30] among many others [25] , [41] . As a common practice, architecture researchers often use popular software architecture simulators to incorporate their radical new ideas. The simulation yields a range of metrics that characterize the execution of benchmarks, with the level of detail in the simulation dictating the specificity of these metrics. Such output metrics provide feedback to the researcher for further explorations and/or decision makings.The quest towards a fast, accurate and detailed cyclelevel architecture simulation has never stopped. This cohort of researchers have mainly dedicated their efforts into three prominent paradigms, i.e., execution-driven simulation [1] , [4] , [7] , [19] , [29] , [31] , [34] , [41] , trace-driven simulation [2] , [3] , [13] , [20] , [21] , [33] , and recently the DL-based simulation [24] , [28] , [32] , [38] (see [6] , [8] , [11] , [39] for more types of architecture simulations). Figure 1(a)-(c ) illustrates the workflow used by each paradigm: (i) Execution-driven simulation offers the most detailed and accurate framework, although this comes at the cost of extremely slow speed and high maintenance overhead. (ii) Trace-driven simulation faces accuracy concerns in pursuit of higher throughput than execution-driven simulations with trace reuse. Reusing the same trace for different microarchitectures raises accuracy concerns as the execution order of different memory instructions can vary. (iii) Emerging DL-based simulations are remarkably fast and can provide comparable cycle-level accuracy, whereas hit three roadblocks: limited output metrics, expensive microarchitecture-specific trace generation, and restricted microarchitecture support.Departing from the designs and desired goals from the aforementioned three paradigms, this paper redesigns DLbased cycle-level microarchitecture simulator. Particularly, we take as input the functional and detailed traces train a DLbased simulator, support a set of desired performance metrics of interest and fast microarchitecture exploration, achieving the comparable accuracy as execution-driven simulation, and an order of magnitude higher throughput than the state-of-theart DL-driven simulator, i.e., SimNet. Figure 1(d) illustrates the workflow of our system, which encompasses the following three contributions:• First, we introduce a unique training dataset design so that the subsequent simulation (i.e., inference) only needs light-weighted and reusable functional trace as inputs.• Second, for predicting a variety of performance metrics of interest, we propose an DL model with separate embedding and self-attention based performance prediction layers.• Third, we introduce transfer learning techniques to rapidly explore various microarchitectures. This includes Challenges. TAO faces three grand challenges: (i) For the training dataset, we need to associate the microarchitecture impacts with each executed instruction in the functional trace. (ii) Reporting various performance metrics demands us to derive sufficiently powerful DL models that can capture the impacts of various hardware components. (iii) Training microarchitecture-agnostic program embeddings presents difficulties because the embeddings are biased towards the architecture they are trained on. These three challenges motivate the design of TAO.Overview. Section IV unveils TAO, our multi-modal DL architecture for microarchitecture simulation. Our approach adheres to design principle #1 by proposing a workflow to construct training datasets from detailed and functional traces which attributes the differences in these two traces to performance metrics, allowing the reuse of functional traces for varying microarchitectures. For design principle #2, we propose multi-metric predictions with feature engineering with a self-attention model to increase the simulation detail. Further, we propose techniques to train microarchitecture agnostic embedding layers that enable fast transfer learning which significantly reduces the re-training overhead of DL-based microarchitecture training and simulation.Execution trace. This paper extensively uses execution trace, which refers to the stream of instructions generated by functional or detailed simulation. The gem5 simulator is modified to generate execution traces capturing various static instruction properties and dynamic performance metrics. Functional trace refers to the microarchitecture agnostic trace generated with functional simulation using AtomicSimpleCPU model. We use the terms functional trace and microarchitecture agnostic trace interchangeably. It only contains static properties like opcode, registers, and other instruction flags. Detailed trace refers to the trace generated with the O3CPU model. It captures various microarchitecture specific performance metrics like data access misses, instruction cache misses, branch mispredictions, speculative instructions and latency of individual instructions.TAO uses functional trace as input to the model and the output (i.e. label) can be various performance metrics. This permits the subsequent simulation (i.e., inference) to only require functional trace as inputs, which can be rapidly generated and reused across microarchitectures. For the output, we use three major performance metrics, i.e., latency, branch misprediction, and data cache misses, to explain how we process the detail and function traces to arrive at the training dataset. However, it is important to note that TAO can potentially support other performance metrics.Functional and detailed traces output similar sequence order, which permits us to associate each instruction of a functional trace with a detailed trace. However, the challenge is that the difference in number of instructions between detailed and functional traces is nontrivial. Detailed trace generally differs from functional trace in the following two aspects. The detailed trace includes two types of additional dynamic instructions during execution that are missing in the functional trace. Specifically, the detailed trace contains incorrect speculative and stall instructions. Incorrect speculative instructions are the wrongly executed instructions squashed based on branch prediction. Stall instructions are used to stall the pipeline by inserting a no-operation (nop) instruction in the pipeline when any other instructions cannot be executed. Our key idea is that both types of additional instructions can be converted into numerical performance differences and attributed to specific instructions from the functional trace.Squashed speculative instructions. If the predicted branch path is correct, speculatively executed instructions will be correct, thus the instruction streams of detailed and functional traces will be identical. When a speculative path is wrong due to branch misprediction, speculatively executed instructions should be squashed. This case leads to a distinction betweenInstruction context Linear Linear K MatMul SoftMax Q V MatMul Linear • Fetch cycles • Execution cycles Branch misprediction Data access level Sigmod K MatMul SoftMax Softmax Multi-head self-attention Opcode Registers Access distance Branch history 5 [0,0,0,0] - [0,0,0,0] 2 [0,1,0,0] [0,0,0,0] - 9 [1,0,0,1] [4,0,0,0] - 3 [0,1,0,0] - - Linear Linear Embed Opcode Registers Access distance Branch history 5 [0,0,0,0] - [0,0,0,0] 2 [0,1,0,0] [0,0,0,0] -Performance metrics• …Fig. 2 : Our initial DL model architecture.functional and detailed traces. The total impact of branch misprediction can be accounted for in the functional trace with the fetch timing information obtained from the detailed trace. If a branch is mispredicted, it will delay the fetch of the next correct instruction. In a detailed trace, the fetch latency of the correct instruction does not include the speculation or branch resolution overhead. To include the miss prediction overhead, we remove the squashed instruction from the detailed trace, get the difference in the fetch clock as the fetch latency, and add it to the subsequent instruction.Pipeline stalls. Stall instructions can be handled similarly to squashed speculative instructions. When no instruction can be executed in the pipeline due to dependency or resource contention, nop instructions are filled. Similar to squashed speculative instructions, we remove and project the latency impact of nop instructions to the subsequent instruction. We use the fetch clock from the detailed trace to determine the additional fetch latency delay.Feature engineering. We propose new techniques to build cross-instruction features, in addition to the per-instruction features from the state-of-the-art [24] . We extract four key instruction properties from the microarchitecture agnostic execution trace: the opcode, registers, data access address and PC address. Opcode and registers derive the per-instruction features. For opcode, we employ an integer mapping for each unique opcode in the dataset. Regarding registers, since the instructions can involve multiple registers, we create a bitmap vector with a size equal to the total number of registers. If an instruction uses i th register, i th index in the vector will be set to 1 (0 otherwise). Both source and destination registers are included in the bitmap vector.Cross-instruction features are derived from the PC and memory addresses. We use the branch history as input to model the outcome of conditional branch instructions. This history, indicating the outcomes of prior branch instructions, is employed by existing branch predictors to predict whether the branch will be taken [15] . To model the data access level, we calculate the access distance, which is the difference between current memory access and the previous N m memory accesses. We use a memory context queue to track the access distance of N m memory accesses.DL model architecture. Figure 2 exemplifies our DL model design. The model first generates instruction embeddings from input features with two-level embedding layers and then uses multi-headed self-attention to perform multi-metric prediction. We use a sequence of N +1 instructions as input to the model. The embedding layers generate instruction embeddings in two steps. Initially, embeddings are created independently for each category of input. This separate generation facilitates enhanced representation learning for each category. Specifically, for opcode, a trainable lookup table based embedding layer is employed. For the remaining categories, distinct linear embedding layers are utilized. The individual instruction embedding is obtained by combining categorical embeddings through a linear layer. Note embedding layers independently generate instruction embeddings for current and N context instructions.Following the generation of instruction embeddings, the prediction layers employ multi-head self-attention to determine the performance metrics. Considering the impact of microarchitecture, this approach allows attention layers to model the interaction between current and earlier instructions. Using self-attention obviates the need for manually tracking context instructions, enhancing efficiency. Employing multiple heads enables each head to learn unique hardware-instruction interplay. The output from each head is concatenated and passed through a linear layer. Overall, the model predicts the latency for each instruction individually.We use different operators to predict different performance metrics based on the output of the last linear layer: (i) The fetch and execution cycles are directly predicted from the linear layer. (ii) An additional sigmoid layer is incorporated for branch prediction to predict whether the branch will be mispredicted. (iii) We use a softmax layer for the data access level, as the output can be multiple categories. (iv) More performance metrics like instruction cache miss and TLB miss can be predicted through a sigmoid layer. During training, a loss is computed from each performance metric and combined with a linear ratio in backpropagation. To obtain the total cycle of all instructions, we use the retire clock of instructions. Retire clock is computed as current clock + fetch latency + execution latency. The retire clock of the last instruction of a benchmark determines the total cycles.Intuitive explanation on supporting a set of performance metrics. Multi-metric prediction exploits the relatedness of performance metrics. With the attention model and microarchitecture agnostic input, our design allows us to output various performance metrics of interest. It can capture the relation-ship between each performance metric and the specific input features that impact the metric. This allows all metrics to be derived from the same hidden layers. Multi-metric prediction has two benefits. First, it increases the output details of the simulation. Second, individual loss from data access level and branch prediction helps the model relate the cycle prediction with memory and branch behavior during training. Microarchitecture agnostic embedding design. The shared embedding layers generate embedding for each individual instruction, and microarchitecture specific prediction layers predict the performance labels. The prediction layers of each microarchitecture computes the gradients for the embedding layers separately. We propose to combine them to update the shared embedding layers.G ! W ! " +G # W # " 2 G ! = ∇L ! G " = ∇L " G ! W ! # G " W " # G ! W ! # G $ W " #Such designs that combine gradients to update shared layers can face two critical issues: negative transfer and imbalance in gradient magnitude for shared layers: (i) Negative transfer [26] , [43] occurs when the shared layers receive gradients from different microarchitecture that are opposite to each other. (ii) Imbalance in gradients magnitude [12] arises when one microarchitecture is too dominant during training, inducing gradients with relatively large magnitudes. These issues impact convergence and generalization [12] , [43] .In Granite, to derive the gradients for shared embedding layers, the gradients from the prediction layers of each µArch are averaged. Just averaging the gradients may resolve neither the negative transfer nor gradient imbalance problem [42] . Using gradient imbalance as an example, if the gradient of one task is larger in magnitude than the other, the larger one will dominate the average gradients. GradNorm, addresses the imbalance in gradient magnitude for multi-task learning by using learnable combination weights (w A and w B ) to combine the losses from each task. This indirectly controls the magnitude of the gradients. The underlying rationale is to dynamically adjust the combination weights in response to the gradient magnitudes of shared layers, ensuring they neither become excessively large nor too small. While GradNorm can address gradient magnitude imbalance, it cannot adequately address negative transfer issues that arise from conflicting gradient directions. Of note, conflicting gradients may appear when the performances of two different microarchitectures are opposite for the same instruction. Modifying the magnitude of gradients may not effectively change gradient direction in joint training [42] . Hence, it may not fully mitigate the adversarial effect of gradients. Adding this linear projection layer resolves the negative transfer issue as follows: during backpropagation, to compute the gradients for the linear projection layer, we multiply the gradients from the earlier layer G A with the transpose of the weight matrix W A , i.e., G A W T A based on the chain rule. Under most of the cases, this operation rotates the gradients in the gradient space, changing the direction of gradients.Figure 3 illustrates our design that tackles negative transfer and gradient imbalance. In contrast to GradNorm which relies on reactive approaches of projecting conflicting gradients to a different plane [42] or finding common direction [35] to mitigate negative transfer, we adopt a proactive solution. We add an individual embedding adaptation layer, i.e., W A for µArch A, similarly W B for µArch B, between the embedding and performance network. The linear layer W A projects the shared embedding (i.e., Green layers) into microarchitecture specific spaces (i.e., µArch prediction layers) during forward propagation. To tackle the gradient imbalance concern, we normalize the gradients for the embedding layers based on the magnitude of the gradients G A W A and G B W B to reduce any existing gradient magnitude imbalance.Training dataset. TAO only uses two microarchitectures based on performance variations to train the model efficiently with the desired accuracy. This is significantly more efficient than training general embedding layers with random microarchitectures. To achieve the accuracy and efficiency goal, we define metrics to measure the architectural variations and select the two architectural variations with the most difference.To measure the microarchitecture variations, we select four performance metrics, i.e., CPI, L1 cache miss, L2 cache miss, and branch misprediction rate. We measure the performance metrics difference of different microarchitectures with Mahalanobis distance [27] instead of Euclidean or Cosine distance for two reasons: (i) Euclidean distance is sensitive to a larger value of one metric, and Cosine distance ignores the value difference. (ii) The other two distances do not consider the correlation among the performance metrics or their scales during distance computation. We use the widely adopted SPEC CPU2017 [10] benchmark suite to evaluate TAO. To construct the training dataset, we first generate program traces of 100 million instructions from each training benchmark with default test workloads using the gem5 O3CPU and AtomicSimpleCPU model, respectively. We study the simulation error and throughput in this section. Particularly, simulation error represents the absolute CPI prediction error for each benchmark and is defined as|CP I pred -CP I truth | CP I truth × 100%.Simulation throughput is measured in million instructions per second (MIPS).Our evaluation of accuracy shows that for most microarchitectures and benchmarks, TAO closely matches the simulation error of SimNet. On average, SimNet and TAO exhibit simulation errors of 5.11% and 5.23%, respectively. The slightly higher simulation error of TAo can be attributed to prediction error for branch misprediction and cache misses.Table I compares the overall time for training and simulation of SimNet vs TAO. Both SimNet and TAO are trained until the error during training is under 6%. It takes 54.2 hours to train a CNN SimNet model. Meanwhile, with microarchitecture agnostic embeddings and transfer learning, TAO can train a model with similar accuracy in merely 1.9 hours. It improves the training time by 28.52×. For simulation, SimNet requires 13.22 hours to generate an input trace with 10 billion instructions. In contrast, utilizing the microarchitecture independent trace, the trace generation time is significantly reduced to 0.53 hours for TAO. For SimNet, it takes 1.93 hours to simulate 10 billion instructions with a simulation throughput of 1.46 MIPS. On the other hand, TAO completes the simulation in 1.41 hours with a throughput of 1.98 MIPS. This speedup against SimNet results from the use of inexpensive functional trace and avoiding history context simulation involving CPU-GPU data movements. Overall, TAO demonstrates a remarkable speed advantage for simulating a new microarchitecture, being 18.06× faster than SimNet. Compared to gem5, TAO provides a speedup of 3.66×.This section compares the prediction error for L1 cache miss and branch misprediction. For evaluation, we vary the L1 Dcache size (16KB, 32KB, 64KB, 128KB) and the branch predictors (Local, Tournament, BiMode, and TAGE_SC_L).Figures 4(a) compares the average cache MPKI across four test benchmarks obtained while varying L1 DCache size for gem5 simulation and TAO. The simulated cache MPKI decreases as the cache size increases from 16KB to 128KB. Cache MPKI predicted by TAO aligns with the simulated result from gem5, demonstrating that a cache size of 128KB results in the least MPKI.In Figures 4(b) , we compare the average branch MPKI across four test benchmarks using different branch predictors for gem5 and TAO. The simulated result from gem5 indicates the highest branch MPKI for the Local and the lowest for the Tage_SC_L. Branch MPKI predicted by TAO also aligns with the simulated result from gem5. The prediction error is lower for simpler branch predictors like Local, experiencing only a marginal increase for relatively complex branch predictors like Tage_SC_L. Nonetheless, TAO maintains the relative accuracy across the spectrum of branch predictors.Unseen benchmarks. TAO can be generalized across a wide variety of unseen benchmarks. The generality of TAO comes from the fact that the deep learning model is trained at the instruction level. We use multiple diverse training benchmarks to train over a variety of instructions. That allows TAO to predict performance metrics for each instruction across different benchmarks accurately. Our evaluation confirms that TAO maintains a good accuracy over unseen benchmarks.Unseen architectures. TAO is designed to simulate singlecore out-of-order superscalar processors. To simulate an unseen microarchitecture, we gather a training dataset through gem5 simulation and train the DL model with transfer learning (see Figure 1(d) ). TAO can accommodate changes in ISAs similarly to microarchitecture changes with some additional feature engineering for ISA-specific opcodes and registers. TAO cannot be directly used to simulate multi-core CPU and GPU architectures. However, the techniques proposed in this paper, i.e., microarchitecture agnostic trace, embeddings, and multi-metric prediction, establish a framework for a rapid DLbased simulation and is transferable to other architectures.Earlier ML-based performance models [17] , [36] opt to build a performance model that can extrapolate the performance to unseen microarchitecture designs by simulating a few designs. Recent DL-based prediction models overcome the limitations of ML-based performance models by increasing the level of abstraction at the instruction level. Ithemal [28] and Granite [38] are two recent works performing basic block prediction. These models first gather training datasets by collecting the basic blocks with tools like Dynamorio [9] . The models predict the latency of each block separately. The input features are constructed based on each instruction and their structure in the basic blocks. Ithemal uses LSTM to construct the embeddings for each basic block hierarchically. Meanwhile, Granite leverages the structure and dependency graph of instructions within the basic block and GNN models for throughput prediction. Basic block throughput prediction models are limited to static basic block prediction, ignoring the impact of caches and branch prediction.This paper introduces a DL-based simulator TAO that supports detailed, accurate and fast microarchitecture explorations. Notably, it achieves 18.06× higher throughput compared to the state-of-the-art simulator, i.e., SimNet.While TAO notably enhances simulation throughput, further research is required to enhance the practicality of DL-based simulation. DL-based simulators currently rely on traditional detailed simulations to gather training datasets for unseen microarchitectures, incurring overhead in simulator design. Futhermore, current DL-based simulation is confined to singlecore architectures, highlighting the need for research into modeling inter-core interaction and synchronization for multicore architectures. Addressing these shortcomings could make DL-based simulation more appealing for architects. 
 
 Appendix:  


Decision: Congratulations! We are pleased to inform you that your paper has been accepted for presentation at MLArchSys 2024. We look forward to your participation at the workshop. Further details regarding the schedule and format will be provided soon. See you at the workshop!

Reviews: review: title: title: Review— TAO: Re-Thinking DL-based Microarchitecture Simulation
strength_weakness: strength_weakness: None.
questions: questions: Overall, the paper was well-written and strong. There were many interesting and unique ideas for thinking about how representation learning should be done to elicit improved (ML-based) simulation results.   

I had a couple questions and suggestions for potential improvement:

(1) Could you provide more details on your dataset in the paper? It is not clear to me as a reader at the moment exactly which benchmarks you generated SPEC traces from for training. It would be nice to know the size of the dataset to understand how much data is really needed to train these models. 

(2) As mentioned, “As a common practice, architecture researchers often use popular software architecture simulators to incorporate their radical new ideas.” One of the benefits of a more traditional uArch simulator is the ability to modify the software and explore uArch design benefits. It is not clear to me right now from reading how an architect can easily do this in your current proposed set up. I understand there is the ability to easily perform transform learning on other uArchs but how would I evaluate a new uArch idea of mine? Would I need to first implement it in gem5, for example, collect a bunch of trace data, and then train a DL model? This seems like I would have already learned if my idea was good or not during the trace data collection but perhaps I am misunderstanding the work.

(3) PerfVec: “Learning Independent Program and Architecture Representations for Generalizable Performance Modeling” by Li et al. is another related work to consider. 

(4) Looking at Figure 1, your evaluation compares (d) [your proposed TAO] to (a) [execution-driven simulator] and (c) [DL-based simulator] but not to (b) [trace-driven simulator]. Adding this to your evaluation would be helpful as a reader to put your results into context of the mechanisms you laid out.
title: title: An improved approach to leveraging DL for microarchitectural simulation
strength_weakness: strength_weakness: While the paper makes claims on being able to predict performance metrics, a table that reports the same would be helpful. There is a brief qualitative discussion, please consider adding some quantitative metrics. Alternatively, it was not clear to me if the simulation errors only capture CPI error, or also errors in specific performance metrics.
questions: questions: While the paper makes claims on being able to predict performance metrics, a table that reports the same would be helpful. There is a brief qualitative discussion, please consider adding some quantitative metrics. Alternatively, it was not clear to me if the simulation errors only capture CPI error, or also errors in specific performance metrics.
title: title: TAO: Re-Thinking DL-based Microarchitecture Simulation
strength_weakness: strength_weakness: NA
questions: questions: # Summary 
* This paper presents TAO a neural network based microarchitecture simulator.
* The authors present the network architecture of TAO, which comprise of a single layer of multi-head
attention which is capable of predicting multiple microarchitecture metric.
* The authors claim that the weights trained on a specific microarchitecture is easily transferrable
to other microarchitectures via fine tuning.

# Comments
Thank you for submitting your work to MlArchSys. I liked the paper and found the idea of using a single
network to predict multiple microarchitecture features very interesting. I have some clarification and
suggested improvements listed below. I hope you find this helpful for further improving your submission.

1. The network proposed in TAO is depicted as the cost model it is unclear from the text of the paper
on how a state of the system would be simulated solely using TAO.
2. It is unclear what the input and the output space of the proposed model is. From the text the reader
can estimate the the entire sequence of instructions is fed to the model as a vector, however it is not
clear whether the model estimates the performance metrics for each instruction or once for the entire
sequence of instructions.
3. A suggested use case highlighting the deployment of the tool would also be useful for the reader to
appreciate the contribution. For example it is not clear if one needs to write a new simulator to train
TAO if the users are exploring a new architecture idea, or would TAO help them perform the exploration by itself.
title: title: Needs more technical details and clarification
strength_weakness: strength_weakness: - Many technical details need to be clarified
- The methodology requires to analyze detailed traces (e.g., "squashing" for speculative and stalls), which requires to run time-consuming detailed simulations.
- The scope is limited to single-core CPUs.
questions: questions: This paper presents an ML-based simulation methodology targeting CPUs. Although the approach is interesting, many technical details need more clarifications.

- Some definitions could be clarified. For example, more precise definitions of "functional trace" and "detailed trace" could be provided.

- The bitmap-based register representation doesn't seem to be distinguishing between source and destination registers. Why this works could be further discussed.

- The ISA used in the evaluation could be clarified. Also, the coverage of the proposed model (RISC only or flexible to cover CISC as well) could be clarified.

- The "squashed speculation instructions" pipeline stall modeling methodology needs more clarification. Because the information is provided only from a high level (basically, utilizing detailed traces), many details are unclear. For example, it is unclear how TAO handles variants of forwarding logics for data hazard resolution.
title: title: Good approach, lacks clarity
strength_weakness: strength_weakness: 1. The evaluation is shallow
2. The paper lacks clarity in many places
questions: questions: Overall, the approach is novel, however, the lack of clarity in many places makes it very hard to follow the paper. I have the following set of questions for the authors:

1. When you compare detailed and functional traces, you say that detailed traces include various performance metrics. What kind of trace is it? Is it not an instruction trace? Can you provide an example of this trace?
2. In figure 2, why is instruction context a subset of execution trace? Probably  it is related to the  statement "we use a sequence of N+1 instructions as input to the model". However, nowhere this is clarified and nowhere is N defined. Kindly clarify
3. Are N_m memory accesses unique?
4. How is the negative transfer solved using the embedding adaptation layer? The explanation on page 4 right column top paragraph only pertains to solving the gradient imbalance problem. 
5.  What does the simulation error in section IV correspond to? Is it the combined error (what kind of mean?) in all the predicted metrics or something else?
6. What is meant by history context simulation involving CPU-GPU data movements? Is it used in SimNet?
title: title: Paper about using multi-headed attention to improve DL based uarchitecture simulations
strength_weakness: strength_weakness: 1) Solution can not expand straightforwardly for analyzing multi-core architectures, NUMA memory and more complicated coherency and consistency models which are practical state of the art hardware. 
2) Not clear how cache replacement or related dynamic features can be modeled by their formula for memory access distance. 
3) Despite the fact that, they have comments on controlling magnitude of values of different features and performance numbers for different uArch, it is not clear what and how they particularly fine tune features for transfer learning. Is it about only negative and absolute value manipulation that they mention, or is there any other clever stuff?
questions: questions: This paper presents work on improving Deep Learning based micro-architecture simulators via multi-headed self-attention to model a correlation between the uarch features and to design a uarch agnostic simulator in general. Paper provides clear motivations and explanations, except couple of questions on scalability and practicality that were addressed at  "top reasons to reject the paper". In general, I think that TAO is an interesting paper that can spur some discussions about further usage of transformer models for uarch simulation between the participants of the workshops and wider community, hence I support acceptance of the paper.