Using On-Premises FPGAs andDistributed Metasimulationh2ps://ﬁres.im@ﬁresimproject
ISCA 2023 Tutorial
Speaker: Sagar Karandikar

Tutorial Roadmap
Custom SoC
Configuration
FireMarshal
Bare-metal &Linux

RTL Generators
RISC-V
Cores

CustomWorkload

VCS

Verilator

Peripherals

CustomVerilog

RTL Build Process
FIRRTL IRQEMU & Spike

Software RTL Simulation

Multi-level
Caches

Accelerators

FIRRTL
Transforms

FireSim FPGA-Accelerated Simulation
Simulation

Debugging

Local/Meta

Verilog

Automated VLSI Flow
HammerTechplugins

Toolplugins

Agenda• Using On-Premises FPGAs• Case Study: How to build/run simula:ons locally with your own FPGA?

• Build Farms, Run Farms, Bit Builders, Deploy Managers Deep Dive• What are they and how do they conﬁgure the manager?

• Distributed Metasimula?on• Scale-out soEware simula:on3

Some of our most requested ques-ons…

4

Some of our most requested questions…

“AWS EC2 F1 FPGAs are great but how do I use
the on-premises FPGAs that I have?”5

Support for On-Premises FPGAs• Previously: Added support for Xilinx
Alveo U250 FPGAs• (was experimentally released in 1.14.0)• Integrates seamlessly with exis?ng
FireSim collateral + tooling• Few line change to target onpremises FPGA vs AWS EC2 F1 FPGAs

6

Case Study: How to build and run simula5ons locally?

7

Building a U250 bitstream• Creating a new build recipe• Use the bit_builder_recipe field to build a Vitis U250 bitstream• Everything else is shared from AWS EC2 F1 to Vitis!

Single-core Rocket conﬁgura?onwith single DRAM channel

firesim_rocket_singlecore_no_nic:
DESIGN: FireSim
TARGET_CONFIG: FireSimRocketConfig
PLATFORM_CONFIG: BaseVitisConfig
deploy_triplet: null
platform_config_args:
fpga_frequency: 60
build_strategy: TIMING
post_build_hook: nullmetasim_customruntime_config: null
bit_builder_recipe: bit-builder-recipes/vitis.yaml

8Building a U250 bitstream• Running the bitstream build• Use the externally provisioned build farm to use a local machine• Everything else is the same!

• Run firesim buildbitstreambuild_farm:
base_recipe: build-farm-recipes/externally_provisioned.yaml
recipe_arg_overrides:
default_build_dir: <PATH TO USER BUILD DIRECTORY>
build_hosts_to_use:- localhost
builds_to_run:- firesim_rocket_singlecore_no_nic

9Building a U250 bitstream• Expect to see a HWDB entry in deploy/built-hwdb-entries/*• Similar format to AWS EC2 case, only has an xclbin instead of agfi
firesim_rocket_singlecore_no_nic:
xclbin: <PATH TO XCLBIN FILE>
deploy_triplet_override: FireSim-FireSimRocketConfig-BaseVitisConfig
custom_runtime_config: null

• Support for sharing xclbins through URI• Store on-premises bitstreams in publicly accessible location (e.g. AWS S3)• Share bitstreams amongst multiple users
10Running a U250 bitstream• Uses externally provisioned run farm totarget local FPGAs• In this case, a local machine with 4 U250s• Use VitisInstanceDeployManager
to setup run farm hosts for U250s• Use the HWDB entry created in theprior section• Same process as before!
• launchrunfarm, infrasetup,runworkload, terminaterunfarm• Attach to running screen session to interact• Have results automatically copied backrun_farm:
base_recipe: run-farmrecipes/externally_provisioned.yaml
recipe_arg_overrides:
default_platform: VitisInstanceDeployManager
default_simulation_dir: <PATH TO SIM DIR>
run_farm_hosts_to_use:- localhost: four_fpga_spec
target_config:
topology: no_net_config
no_net_num_nodes: 1
link_latency: 6405
switching_latency: 10
net_bandwidth: 200
profile_interval: -1
default_hw_config:
firesim_rocket_singlecore_no_nic
workload:
workload_name: linux-uniform.json

11Behind the Scenes: Build + Run Farms• Manager rearchitected for
maximum configurability• Target different clouds/clusters• Convenient defaults for AWS EC2and set of unmanaged machines(typical pre-setup cluster)

Default Distributed AWS EC2 Setup

12

Behind the Scenes: Build + Run Farms• Manager rearchitected for
maximum configurability• Target different clouds/clusters• Convenient defaults for AWS EC2and set of unmanaged machines(typical pre-setup cluster)Completely Local Setup
13

Behind the Scenes: Build + Run Farms• Manager rearchitected for
maximum configurability• Target different clouds/clusters• Convenient defaults for AWS EC2and set of unmanaged machines(typical pre-setup cluster)

Mixed Setup:
Local Builds + Distributed Simula?ons

14

Behind the Scenes: Build + Run Farms• In config_<build/runtime>.ini• base_recipe sets type of build/run farm
• You can modify its defaults by• Modifying the recipe file directly• Overriding using recipe_arg_overrides

config_runtime.yaml example

15

Behind the Scenes: Build + Run Farms• Two types of default build/run farm types• AWS EC2 (aws_ec2.yaml)• Default build/run farms used on AWS EC2• Fully distributed builds and simulations• Equivalent functionality to pre-1.14.0• Externally Provisioned (externally_provisioned.yaml)• Use a pre-setup cluster of machines (including running locally)• Just needs FPGA platform (i.e. Vitis), number of FPGAs, and IP/hostname

16

Behind the Scenes: Bit Builders + Deploy Managers• Notice how nothing was mentioned about type of FPGA used!
• Target different FPGA platforms as well!
• AWS EC2 F1 or Vitis Alveo U250 FPGAs• This is done by• Bit Builders – abstract bitstream build process• Deploy Managers - abstract setup of run farm hosts for FPGA platform• You can see this in config_build_recipes.yaml and a specific
run farm recipe (i.e. aws_ec2.yaml)

17

Behind the Scenes: Maximum Conﬁgurability• Manager rearchitected formaximum configurability• Target different clouds/clusters• Target different bitstreams• And any combinations of them!

AWS EC2 Build
Farm

Cluster Build
Farm

Custom Build
Farm

AWS EC2 Run
FarmCluster Run
Farm

Custom Run
Farm

AWS EC2 F1Bitstreams

Vitis U250
Bitstreams

18

“Gah! My FireSim simula2on breaks,
how do I do FireSim SW-level metasimula2on again?”

19

Metasimulation Recap“FAME-1” Transformed RTL Design
Target-Level
SW Simula?on<- Resp QueueMetasimula?on

RTL Design
Req Queue ->

DRAM
Model
100
cycle
latencyMem
ChannelAbstract
Model
PhysicalDRAM
100ns
latency

FPGA Fabric
20

Metasimulation Recap• Software RTL Simulation• Target design transformed by Golden Gate
• Host-FPGA interfaces/shell emulated using abstract models• Uses existing FireSim models (i.e. DRAM, UART)

But how do I run it?
21Running Metasimulations• Original make API• In $FDIR/sim$ make
EMUL=<verilator|vcs>
DESIGN=FireSimNoNIC
run-asm-test-debug• Issues• What are the make variables/targets I need to pass in?
• How do I run mul:ple tests in parallel? Bash script it myself?
• How do I run my exis:ng FireMarshal workload with this?
22Running Metasimulations• Original make API• In $FDIR/sim$ makeBe`er
yet! Ju
EMUL=<verilator|vcs>
manag st have theDESIGN=FireSimNoNIC
er do eFireSimverythrun-asm-test-debug• Issuesing!

• What are the make variables/targets I need to pass in?
• How do I run multiple tests in parallel? Bash script?
• How do I run my existing FireMarshal workload with this?
23

Running In Metasimulation Mode
• In config_runtime.yaml use the metasimulation mapping• enabled: FPGA simula:on → SW RTL metasimula:on• host_simulator: Choose to run Verilator/VCS w/ and w/o waveforms• *plusargs: Extra non-FireSim speciﬁc arguments to pass to simulator• Has same features as FPGA simula?ons!
• Use arbitrary Run Farms• Automa:c copying of results• Use FireMarshal workloads• Quickly do FPGASim ↔ MetaSim• Same performance results

metasimulation:
metasimulation_enabled: true
metasimulation_host_simulator: verilator
metasimulation_only_plusargs: …
metasimulation_only_vcs_plusargs: …

24

Example Workflow
1. Write default RTL in Chipyard
2. Debug in Chipyard w/ target-level simulation
3. Port to FireSim (change config. files, use FireMarshal workload)
4. DSE and debugging w/ single/multi-node metasimulations
5. Testing w/ single/multi-node FPGA simulations using Vitis U250s
6. Scale-out to datacenter scale with AWS EC2 F125

Example Workﬂow
1. Write default RTL in Chipyard
2. Debug in Chipyard w/ target-level simulation
3. Port to FireSim (change config. files, use FireMarshal workload)
Uniﬁed workﬂow
for agile
4. DSE and debugging w/ single/multi-nodemetasimulations
research of RISC-V systems!
5. Testing w/ single/multi-node FPGA simulations using Vitis U250s
6. Scale-out to datacenter scale with AWS EC2 F126

Demo Time!

27

FireStation v1 Machine Specs• Intel Core i7 13700K• Liquid cooler (w/RGB)• 32 GB DDR4 (w/RGB)• Xilinx Alveo U250 (ac2ve)• Motherboard spec’d for:• 2 U250 + GPU• OR• 3 U250• 1500W PSU to support mulL-FPGA/GPU• Thermaltake Core P3 Red Case
• Ubuntu 18.04$1500 without FPGAs or GPUs
28

Porting to new Xilinx FPGAs should be easy!

Credit: David
Christoph Metz &Magnus Själander(NTNU), First
FireSim/Chipyard
Workshop, colocated withASPLOS 2023.

29

Support for two new FPGAs added to FireSim
in 2 days: (#1) Xilinx VCU118• Similar to U250 / F1, but “dev board” format• Highly requested by users• Usual PCIe + DRAM setup available to FPGA
• Less than 24 hours from “decide to do it” to “booting Linux”

30Support for two new FPGAs added to FireSimin 2 days: (#2) RHS Research Nitefury II• M.2 Form-factor board
originally developed forBitcoin mining• Xilinx XC7A200T-2FBG484E• 1GB DDR3

3. 1”

• A few reports on theinternet of successful
usage in an M.2 toThunderbolt enclosure w/
regular XDMA IP/soDwarestack• So we bought a few!

31

RHS Research Nitefury II vs. Prior FireSim HostPla[orms• Previous host platforms
supported by FireSim are cloud
platforms or require $8-10K perboard• Also, require desktop/server withPCIe• Instead:• $150 FPGA
• $150 M2 <-> Thunderbolt
enclosure (cheaper ones may work,untested)• Linux Laptop32

On-premises FPGA support now available!
• High-level of automaIon/reproducibility enabled by FireSim on AWS F1cloud now extended to local/on-prem FPGAs:• Went from new machine with no FPGA aXached to working FPGA-accelerated
simulaLon in 1 hour and 40 mins

• New in latest release: Xilinx VCU118, Xilinx Alveo U250 / U280 XDMA-shell,
RHS Research Nitefury II• Use exisIng FireSim features at-scale and locally!
• Cycle-accurate simulaLon• Debugging• Integrated logic analyzers, trace dumps, synth. assert/prints, co-simulaIon• So[ware support• FireMarshal workload management• … and more!

33

Summary• Customize how/what/where you build/run things• Local Builds → Fully Distributed AWS EC2 Builds• Local Simula:ons → Fully Distributed AWS EC2 Simula:on• And everything in between• Target both local and AWS EC2 FPGAs• Distributed SW RTL metasimula?onsCheck out hfor m ttps://docores.
usag• Debug RTL using uniﬁed infrastructure
e de fires.im/tails• Use FireSim modeling + features in SW RTL simula:on34