## 3 AR/VR Hardware and Software Requirements
### Low-Latency Displays and Headsets for AR/VR  
The AR/VR industry has witnessed an evolution with the release of several consumer-grade VR headsets (e.g., Oculus Rift CV11, HTCVive2, and PlayStation VR3) in recent years [60]. These devices are responsible for linking the users with an environment created by the coexistence of the real and virtual world in real-time. They require a registration [61] that is both spatial and temporal in nature with very few errors so as to maintain this coexistence of the real world with the virtual one. However, some errors [62] occur in the registration due to system latency, which results in the distraction of the users due to the lag in the virtual imagery arriving at the intended position. The system latency is caused by the change in the imagery in response to the user's movement while using the VR headset. When the rendering of the imagery occurs frame by frame with the transmission of the data to the display in the conventional manner, the registration errors cause latency. To minimize the system latency caused by waiting for the vertical synchronization, Nvidia introduced  
Figure 4: This figure details the requirements for various types of latencies associated with AR/VR technologies from the hardware, software, and design perspectives in order to enable a low-latency connection in an immersive environment. a head-worn display system known as G-SyncTM, where the delay can be reduced with the maximization of the input responses by allowing the display to accept the frames immediately after being rendered by a graphics processing unit (GPU) [63]. Motion-to-photon (MTP) latency is the difference between the time of the motion captured by the headset sensor and the time of the appearance of the image on the headset's display. Long MTP latency can cause various side-effects for users, such as nausea or motion sickness. Inertial measurement unit (IMU) sensors play an active role in sampling the motion data consisting of the user's pose information, which will be later transmitted to the rendering engine to generate a framebuffer [64]. A time lag is observed when the framebuffer is sent to the headset to display the rendered imagery. Therefore, a post-render warping approach has been introduced [65] for this type of headset to optimize the latency. This technique uses homography transformation to transform an image captured at a specific camera pose into an image at a new camera pose. The addition of a post-rendering stage with the usage of a light-weight, low-cost, optical-see-through (OST) HMD has also been shown to be highly beneficial to overcome the temporal inconsistency of registration [61]. This leads to an updated rendered imagery with the most-recent data based on head tracking. This display provides an average end-to-end latency of 80 \(\upmu\)s and can present binary frames to the user up to 16 kHz. The processing of post-render warping is performed by a field programmable gate array (FPGA) for just-in-time tracking of the updates by utilizing a novel pseudo-random pulse density modulation (PDM) [61] technique for the conversion of binary pixels into a perceived grayscale format. A tracking system to report the updates frequently can be utilized to reduce the perceived latency to the sum of that tracking latency for about 80 \(\upmu\)s to 124 \(\upmu\)s of display latency. In addition to the OST display technology for post-render warping, an open-source mixed reality headset, supporting both virtual and augmented reality applications [66], provides a better MTP latency of 13.4 ms on average and delivers an immersive experience to the user by using the post-render warping technique along with a 240 Hz digital light projector (DLP). The Xilinx Ultrascale+ ZCU102 System-on-Chip [67] has been used to prototype the hardware architecture, and the post-render warping accelerators are processed by the FPGA of the ZCU102 to design the system of the headset.