My Objective:
We aim to optimize the '7' objective performance metric values of 'area', 'power dissipation', 'DC gain', 'Slew rate', '3db frequency', 'Unity gain bandwidth', and 'Phase margin' (multi-objectives) of a pre-determined two stage operational amplifier circuit topology. We need to optimally configure/tune the design device parameters (device dimensions and values, as variables) such as transistor dimensions ['L1', 'L3', 'L5', 'L6', 'L7', 'W1', 'W3', 'W5', 'W6', 'W7'], Bias currents ['Ib'], Bias voltages ['Vb'], and values of the passive devices ['Cc', 'Cl', 'Rp'] associated in the circuit design, in a such a way that the resulting design satisfies the predetermined performance measures (design specifications). In this overall construction of optimization process, the pre-determined performance measures are considered as constraints as well as target optimization metrics to configure the optimized design device parameter values. 

For my objectives, I need to implement the reinforcement learning (RL) code with multiple output heads Graph neural networks on multigraphs, GNN is embedded by the technique of hybrid MPNN-RGAT architecture that contains the message passing attention mechanism. Here GNN is with in the agent the action and rewards are performed from the following requirement, for this requirement suggest me the best optimization techniques that take actions and generate rewards and to update the parameters.

Environment: My environment is the Cadence virtuoso simulator, where I am simulating the pre-designed fixed circuit topology. I am accessing the simulator available in the local server through the OCEAN skill command controlled by python programming. The environment gets the input variables for the design device parameters ['L1', 'L3', 'L5', 'L6', 'L7', 'W1', 'W3', 'W5', 'W6', 'W7', 'Ib', 'Cc'] from the agent and gives the output performance metrics ['area', 'power dissipation', 'DC gain', 'Slew rate', '3db frequency', 'Unity gain bandwidth', 'Phase margin'] corresponding to the given input variables. Circuit topology is fixed, only the design device parameters present in the schematic are variables to be optimally configure to met the desired output performance metrics.

State: The RL agent processes the circuit graph component by component. For a circuit with n components in topology graph G, the state sk for the kth component is defined as sk = (k; t; h),
where k is the one-hot representation of the transistor index, t is the one-hot representation of component type and h is the selected model feature vector for the component which further distinguishes different component types. For the NMOS and PMOS, the model parameters. For the Bias current, capacitor and resistor, we set the model parameters to zeros. For instance, for a circuit with ten components of five different kinds (NMOS, PMOS, R, C, Ib) and a five-dimensional model feature vector.

Action Space. The action vector varies for different types of components because the parameters needed to search are not the same. We use a continuous action space to determine the transistor sizes even though we will round them to discrete values. The reason why we do not use a discrete action space is because that will lose the relative order information, also because the discrete action space is too large.

Action Representation: Inspired by human designers who iterate with fine-grained tuning steps to find optimal device parameters, we use discrete action space to tune device parameters. For each tunable parameter 𝑥 of a device (e.g., width and finger number of transistors), there are three possible actions at each step: increasing (𝑥+del 𝑥), keeping (𝑥 +0), or decreasing (𝑥−del 𝑥) the parameter, where “del 𝑥" is the smallest unit to update the parameter within its bound [𝑥min, 𝑥max]. Assuming total 𝑀 device parameters, the output of the policy network is an 𝑀 × 3 probability distribution matrix with each row corresponding to a parameter. The action is taken based on the probability distribution.

Reward Function:The reward is directly related to the design goal. We define the reward 𝑟𝑖 at each time step 𝑖 as 𝑟𝑖 = 𝑟, if 𝑟 < 0 or 𝑟𝑖 = 𝑅, if 𝑟 = 0, where 𝑟 = summation of  min{(𝑔𝑗𝑖 − 𝑔𝑗∗ )/(𝑔𝑗𝑖 + 𝑔𝑗∗), 0}. where, it is a normalized difference between the obtained specifications 𝑔𝑖 and the given specifications 𝑔∗. The upper bound of 𝑟 is set to be 0 to avoid overoptimizing the parameters once the given specifications are reached. All 𝑁 specifications are equally important. We also give a large reward (i.e., 𝑅 = 10) to encourage the agent if the design goals are reached at some step. The episode return 𝑅𝑠0,𝑔∗ of searching optimal device parameters for the given goals 𝑔∗ starting from an initial state 𝑠0, is the accumulated reward of all steps: 𝑅𝑠0,𝑔∗ = summation of 𝑟𝑖 . Our goal is to train a good policy to maximize 𝑅𝑠0,𝑔∗ .