## 1 Introduction
### Background and Related Work  
Here, we provide an overview of surgical video recording and 3D view synthesis approaches and the challenges in their application to non-feature-rich frames of surgeries.  
**Surgical Video Recording.** Recording surgery is needed for various purposes, such as preserving case studies, educating trainees, and passing on skills [1; 2; 3; 4]. Contrary to laparoscopic surgery [5; 6], recording in open surgeries, in which physicians directly view the affected areas, is challenging because of possible interference with the surgery, spatial limitations, and occlusions by surgeons.  
There are two major ways to install cameras to record surgeries: cameras on the surgeon's head [2; 7; 8; 9] and arms [10], and in the operating room [11]. Knowing possible cable management, image noises (e.g., motion blurs), and occlusions, McSL is a reasonable solution for multi-view observations [1].  
Displaying non-aligned multi-view images makes the analysis difficult as it requires mental image warping. Mathematically aligning such images requires calibrating the cameras. However, objects in surgical videos are often featureless, which makes calibration challenging. Calibration before video recording is useless as cameras in McSL move when surgeons manipulate it. To address these issues, we implement a feature-matching scheme that utilizes multiple frames over time to collect sufficient feature points for calibration.  
**3D View Synthesis.** To seek system requirements, we reviewed comments from two surgeons who have experience using a McSL recording system at Keio University School of Medicine. One is a Prof. Dr. of the Department of Plastic and Reconstructive Surgery and the other is a Dr. of the Department of Anatomy. The surgeons requested to see and compare several frames in 3D each at the beginning and end of the surgery, especially for plastic surgery. Therefore, we implement a video player that allows medical doctors to seek and select frames so that the system can instantly reconstruct the view in 3D at the selected frames only. From this background, we did not use view synthesis approaches that require per-frame extensive training [12; 13].  
Another approach is to convert all frames into a 3D video, such as neural videos [14; 15; 16; 17] and multi-layer mesh and texture atlas videos [18; 19]. However, neural videos are typically in low resolution and are computationally demanding. A multi-layer mesh provides faster rendering at higher resolutions, although conversion is nonetheless computationally expensive [18]. We instead provide on-demand 3D views only at selected frames.  
**Disocclusion Rendering.** Real-time disocclusion rendering is often referred to as diminished reality [20]. Multi-view approaches bring observed background pixels from cameras to the current view to disocclude foreground objects [21; 22; 23]. These approaches either rely on plane geometry proxy [21] or range sensors [22; 23] for millisecond-order rendering, which prevents us from applying them to our application. Another approach relies on image inpainting, in which the recovered area is completely synthetic (i.e., pixels are hallucinated by a collection of pixels within the field of view) [24; 25; 26]. Therefore, inpainting approaches are not suitable for our application.  
Rather than relying on the above approaches, we utilize a recently developed offline multi-view approach [27] and soft 3D representation [28; 29] (i.e., multi-layer scene representation) that is known to be more robust than explicit 3D representations. Although 3D rendering allows users to peep at occluded backgrounds, we further extend the soft 3D representation for disocclusion rendering.