## 3 AR/VR Hardware and Software Requirements
### AR/VR Hardware: Designing and Prototyping  
AR/VR utilizes 3D lifelike virtual avatars known as codec avatars [74] to mimic human behavior in the virtual world. They are implemented using two main building blocks, i.e., encoding and decoding [75; 76]. These two blocks work simultaneously to virtually represent faces in a realistic fashion. Firstly, the encoding captures the facial expression data of one user for continuous transmission to the other user, and then, the decoding is performed by rendering a lifelike avatar on the other user's headset. However, the need to achieve high frame rates with frequent updating of the status in the encoding and decoding blocks to have a low latency for the codec avatars for realistic interactions arises. As a result, these mobile AR/VR devices should possess the capability to provide high visual computing requirements to reach a target frames per second (fps). To overcome these limitations of the workload of the codec avatars on mobile AR/VR devices, a silicon solution has been designed to achieve the requirements of a low-latency system [77]. A small-scale system-on-a-chip (SOC) prototype was presented in [78] for low-latency energy-efficient performance and to enable eye gaze extraction of the codec avatar model, which used a test chip, fabricated in 7 nm technology, featuring a neural network (NN) accelerator consisting of a 1024 multiply-accumulate (MAC) array, a 2 MB on-chip SRAM, and a 32-bit RISC-V CPU. This VR-based prototype system used a low-power deep neural network (DNN) accelerator test chip to offload some of the workload. To meet the challenging mobile AR/VR SoC specifications for a codec avatar demonstration, a convolutional-NN (CNN)-based eye gaze extraction model was utilized to mitigate the system-level energy and latency costs for the entire model to fit on the chip. This approach enabled the prototype SoC to achieve 30 fps, satisfying the requirements for a system with low latency and low power consumption. To estimate the 3D pose, i.e., translation and orientation in a 6-DoF pose, of the observer, visual-inertial odometry (VIO) was utilized to measure the motions from the images of the camera(s) and linear acceleration and angular velocity from the IMUs. A lower pose update latency was required for the estimation of the 6-DoF pose of the HMD to achieve a latency of less than 20 ms for a motion-sickness-free immersive experience. The noisy data obtained from the camera were analyzed by a maximum likelihood estimator to estimate the system dynamics or the state in a typical VIO processing pipeline [79]. There are some VIO systems whose functioning is similar to an extended Kalman filter (EKF) [80] to update their states and decide the best pose estimate available. The best pose is decided immediately upon receiving the sensory inputs, including the image frame or IMU sample, without waiting for an optimal solution via local or global optimization based on past estimates. The acceleration of EKF-based monocular VIO [81] was modeled in [79], by a flexible, scalable, and power-efficient micro-architecture. This enabled direct processing of IMU samples without pre-integration to obtain an ultra-low latency and accurate instantaneous pose updates. This micro-architecture was designed to be adaptive and easy to integrate with a host CPU or micro-controller, enabling software-based configuration control while retaining programmability. This hardware architecture, denoted as VIO HWA [79], was integrated in a typical embedded vision SoC in which the identified acceleration candidates were partitioned into the VIO HWA as specialized modules interfacing an on-chip Shared L2 SRAM (SL2) sub-system. They utilized some intelligent pre-fetching techniques to make all the modules of the micro-architecture flexible and tolerant to the data fetch latency by leveraging decoupling first-in, first-out (FIFO) in order to hide the latency of data access to SL2.