Package: verlpy
VeRLPy**Under Construction**VeRLPyis an open-source python library developed to improve the digital hardware verification process by using Reinforcement Learning (RL). It provides a genericGymenvironment implementation for buildingcocotb-based testbenches for verifying any hardware design.InstallationThe easiest way to start using VeRLPy is to install it usingpip install verlpyVeRLPy is currently dependent on OpenAIGym,cocotbandStable Baselines3. These packages should get installed alongside VeRLPy when installing usingpip. For running the verification, a simulator compatible with cocotb is additionally required. Please refer to the officialcocotbdocumentation to set this up.Usage GuideHaving familiarity withcocotb, OpenAIGymandthis whitepaper on VeRLPywill be very beneficial to get started with the VeRLPy library.The hardware design provided in Verilog or VHDL is simulated by cocotb using the chosen simulator. VeRLPy aims to offer a clean interface for bringing RL logic into the conventional cocotb testbench while adhering to the OpenAI Gym environment structure allowing users to leverage the standard RL tools. The DUT and the verification testbench are the environment for the RL agent to act on. The agent chooses an action which is executed on the environment and the consequences of that action are informed back to the agent in terms of the state and the reward.Identifying verification goals and defining the MDPTo build a testbench using VeRLPy, we need to start by defining the verification goals in terms of functional coverage. This involves identifying the events occurring in the DUT that correspond to the features that are part of the design specifications. The reward given to the RL agent will be a function of how often these events occur during the simulation.In addition, since there is the additional RL component involved, the MDP has to be defined. This means identifying what each timestep and each episode corresponds to in terms of verification logic and input stimulus to the DUT, and what the state and the action space of the MDP represent.Each RL episode starts with a call to to thereset()function of the Gym environment. Followed by this, there are one or more calls to thestep(action)function untildoneis returned with a value ofTruefrom the RL environment.  Refer toGymfor a more detailed understanding of how the control flow occurs in a Gym environment.Inheriting CocotbEnvThe library provides a Gym environment objectCocotbEnvwhich can be inherited to build the verification testbench. This class has predefined functions/coroutines which interact with the RL agent to facilitate the learning process. These functions are triggered by thereset(),step(action),donevariable, etc. Please referherefor more detailed explanations of these functions and their implementation details. We start with implementing this class that inheritsCocotbEnv.# test_my_example_design.pyimportcocotbfromverlpyimportCocotbEnvclassMyExampleDesignCocotbEnv(CocotbEnv):def__init__(self,dut,observation_space):super().__init__()self.dut=dut# DUT object used for cocotb-based verificationself.observation_space=observation_space# state space of the RL agent# add here any "self." variables that need to be accessed in# other functions below@cocotb.coroutinedefsetup_rl_episode(self):# add here the logic to be# executed on each call to reset() by the RL agent@cocotb.coroutinedefrl_step(self):# add here the verification logic to be# executed on each call to step() by the RL agent@cocotb.coroutinedefterminate_rl_episode(self):# add here the logic to be executed at the end# of each RL episode when done == 1 for the Gym envdeffinish_experiment(self):# add here the logic to be executed after all# the episodes are completedNote that all the coroutines with the deoratorcocotb.coroutinerequire ayieldstatement in the body like how it is in standard cocotb testbenches.Instantiating the verification environment objectThe functions/coroutine implementations defined inMyExampleDesignCocotbEnvshould contain the appropriate logic that must run from the cocotb side. Once this class is defined, the cocotb test can be added to invoke the verification logic from this class. While defining this, the state space of the MDP should also be passed as an argument as shown below.# test_my_example_design.pyimportcocotbfromverlpyimportCocotbEnv,utilsimportgymclassMyExampleDesignCocotbEnv(CocotbEnv):def__init__(self,dut,observation_space):super().__init__()self.dut=dutself.observation_space=observation_space# add here any "self." variables that need to be accessed in# other functions below@cocotb.coroutinedefsetup_rl_episode(self):# add here the logic to be# executed on each call to reset() by the RL agent@cocotb.coroutinedefrl_step(self):# add here the verification logic to be# executed on each call to step() by the RL agent@cocotb.coroutinedefterminate_rl_episode(self):# add here the logic to be executed at the end# of each RL episode when done == 1 for the Gym envdeffinish_experiment(self):# add here the logic to be executed after all# the episodes are completed# entry point for the cocotb verification test@cocotb.test()defrun_test(dut):cocotb_env=MyExampleDesignCocotbEnv(dut,gym.spaces.Discrete(1))# gym.spaces.Discrete(1) => Just 1 state in the state spaceyieldcocotb_env.run()# plot the results of the verification experimentutils.visualize(cocotb_env.log_file_name)VeRLPy also provides some plotting capabilities which can be accessed fromutilsas shown above.Adding coroutines to track eventsThe identified functional coverage events can be tracked by using cocotb coroutines like in conventional cocotb-based verification.# test_my_example_design.pyimportcocotbfromverlpyimportCocotbEnv,utilsimportgymclassMyExampleDesignCocotbEnv(CocotbEnv):def__init__(self,dut,observation_space):super().__init__()self.dut=dutself.observation_space=observation_space# add here any "self." variables that need to be accessed in# other functions below@cocotb.coroutinedefsetup_rl_episode(self):# add here the logic to be# executed on each call to reset() by the RL agentself.cocotb_coverage.clear()# clear last episode's coverageself.coverage_coroutine=cocotb.fork(monitor_signals(self.dut,self.cocotb_coverage))@cocotb.coroutinedefrl_step(self):# add here the verification logic to be# executed on each call to step() by the RL agent@cocotb.coroutinedefterminate_rl_episode(self):# add here the logic to be executed at the end# of each RL episode when done == 1 for the Gym envself.coverage_coroutine.kill()deffinish_experiment(self):# add here the logic to be executed after all# the episodes are completed@cocotb.coroutinedefmonitor_signals(dut,cocotb_coverage):whileTrue:yieldRisingEdge(dut.CLK)s=[(int)(dut.reg_1.value==1),(int)((dut.reg_2.value)%4!=0),(int)(dut.reg_3.value==32)]# Here reg_1, reg_2 and reg_3 are some# key registers of interest in the DUTs=''.join(map(str,s))cocotb_coverage.append(s)# entry point for the cocotb verification test@cocotb.test()defrun_test(dut):cocotb_env=MyExampleDesignCocotbEnv(dut,gym.spaces.Discrete(1))# gym.spaces.Discrete(1) => Just 1 state in the state spaceyieldcocotb_env.run()# plot the results of the verification experimentutils.visualize(cocotb_env.log_file_name)Themonitor_signalscoroutine added above monitors the DUT for events of interest that count towards the functional coverage. The  boolean logical expressions in the listsabove correspond to the logical expressions for identifying each event. The number of times these events occur affect the reward signal given to the RL agent.monitor_signalsshould track these events and add them to thecocotb_coverageattribute of theMyExampleDesignCocotbEnvclass that we wrote.monitor_signalsis invoked in thesetup_rl_episodecoroutine along with the clock and reset coroutines. It is passed thecocotb_coverageattribute as an argument. Note thatmonitor_signalsis killed in theterminate_rl_episodecoroutine at the end of each RL episode. This is important for all coroutines since it might otherwise lead to performance issues with multiple "alive" coroutines still ongoing from previous episodes.Configuration FileA configuration fileconfig.inineeds to be provided to specify the parameter related to the simulation and the RL agent. A sample coniguration file is provided below with comments for what each section and key corresponds to.; This section is to provide the; main parameters for the verification runs[main]; number of RL steps for which the experiment is runnum_steps=1000; number of functional events trackednum_events=3; weightage of each functional event for reward computationreward_function=[0, 0, 1]; set log_step as 0 for logging just aggregated results and 1 for logging details in each steplog_step=0; set mode as 0 to generate the random baseline without RL and 1 for using RLmode=1; specify the stable_baselines3 algorithm to be used from SAC, DDPG and TD3algorithm=SAC; fsm_states contains the regex patterns for; state-based binary sequence generation; (leave as [] unless utils.get_next_state_of_FSM() is needed in the code)fsm_states=[]; Provide the discrete action component names here.; The valid dscrete value set for the specified keys; should be given in the [discrete] sectiondiscrete_params=['count_width', 'fmap_len']; This section is to provide the bounds; of the continuous dimensions of the action space.; If multiple dimensions are there, provide the list of bounds for each dimension; eg: lower_bounds = [0, 0, 1] and upper_bounds = [1, 1, 3] corresponds to; [0, 1] x [0, 1] x [1, 3] as the continuous action space[continuous]lower_bounds=[0, 5]upper_bounds=[1, 7]; This section is to provide the list of valid; discrete values for each discrete action; component named in discrete_params[discrete]count_width=[1, 2, 3, 4, 5, 6, 7, 8]fmap_len=[100, 200, 300, 400, 500, 600, 700, 800, 900, 1000]; This section is to provide the required; hyperparameters of the chosen stable_baselines3 algorithm[RL]policy='MlpPolicy'learning_starts=100learning_rate=0.0003train_freq=(1, 'episode')verbose=1Thereward_functionkey specifies how the functional events tracked in theself.cocotb_coverageattribute need to be rewarded for improving coverage.reward_functionset as[0, 0, 1]like above implies that if the third functional event occurs during a step, a reward is given to the RL agent. Refer to thepaperfor the actual computation details.The[continuous]and[discrete]sections together specify the total action space of the RL agent. The continuous dimensions of the action space based on the above configuration file is the cross product[0, 1] x [5, 7]. The discrete dimensions of the the action space is the cross product{1, 2, ..., 8} x {100, 200, ..., 1000}. Therefore the complete action space is the cross product[0, 1] x [5, 7] x {1, 2, ..., 8} x {100, 200, ..., 1000}.Filling in the verification logicFinally, the body of each of the coroutines overriden inMyExampleDesignCocotbEnvneed to be completed. The action suggested by the RL agent based on theconfig.inican be accessed through theself.continuous_actionsandself.discrete_actionsattributes of the class. In the above example,self.continuous_actionswill sample from[0, 1] x [5, 7]andself.discrete_actionswill sample from{1, 2, ..., 8} x {100, 200, ..., 1000}.The listself.cocotb_coverageneeds to be updated with the strings corresponding to the covered events from the previous timestep of the RL episode for proper reward computation based on the reward function defined in theconfig.inifile. This update will happen on its own ifself.cocotb_coverageis passed as the argumentcocotb_coverageto themonitor_signalscoroutine defined above  Refer to the examples folder for more concrete examples on how this is done in various designs.Multi-step RLVeRLPy by default assumes a single step single state MDP. If a multi-step MDP is required, it can be implemented by overridingcompute_rl_observationfunction in theMyExampleDesignCocotbEnvclass. The internal elements of the DUT that need to be tracked for computing the observation/state after each step can be done so by utilizing a separate coroutine likemonitor_signalsis used for computing the reward.Make fileThe make file can be written like how it is done usually in cocotb testbenches. Once it is done andmakeis run, the verification simulation runs and the logs, models and plots are generated. Use themodeconfiguration parameter inconfig.inifor running the verification with/without the RL feedback.CitationIf you find this work useful, consider citing it with this BibTex:Change Log08-08-2021 - Usage guide added