The Evolution of the Virtual Production Studio as a Game Changer in Filmmaking

This chapter analyses the key methodological and conceptual shifts in filmmaking brought about by the integrated use of VR technology, game engines, and LED walls in Virtual Production Studios (VPS). VPS combine the real time processing capabilities of game engines with LED walls and camera tracking to produce live in-camera visual effects. In this chapter, we focus not only on the elements that make an integrated VPS system, but also on what makes it qualitatively different from traditional and digital filmmaking. The methodology of the VPS transforms the relationships among display technology (screen), live action (actors), image production (cameras), and postproduction (visual effects or VFX). An understanding of how these technologies work together will form the premise for an analysis of the qualitative and performative shifts that filmmaking is undergoing. There are a number of crucial elements of an integrated VPS technology: a game engine, LED walls, a camera tracking system, a motion tracking system, Simulcam, and a Render. VP has its roots in traditional filmmaking techniques associated with combining different images in one shot, such as back projection and green screen. The back projection technique immerses performers in the environment of the action with an analogue (filmed) or a digital backlot (this technique was used, for example, in the film Oblivion in 2013), but brings the problem of matching the brightness of foreground and background, and, more crucially, it requires the camera on set to be still in order to avoid problems with parallax. The traditional green screen technique allows actors to perform with placeholder props and extras in an empty room lined with an even colour background. Neither of them is ideal. More recently, LED walls have replaced green and blue screen techniques, creating an active stage for actors’ performance in a VPS and a safe passage from digital to virtual backlot for filmmakers. While with a green screen stage and digital backlot the final image is composed on screen, with LED walls and a virtual backlot the final image is displayed on LED walls and updated in real time using a game engine. VFX are created in camera, making the distinction between screens and cameras superfluous, to some extent. VPS systems based on game engines have been tested since 2001, when Peter Jackson used a virtual camera in Lord of the Rings and Steven Spielberg used an iPad attached to a Vive controller to visualise potential shots in a CG environment updated in real time for the film AI. The technological components of a VPS have become progressively more sophisticated, with studios testing and adopting in-house solutions to issues of synchronisation, data management, and lighting. Everything changed when the game company Epic Games partnered with Lux Machina (LED walls), Magnopus (VR/AR), Profile Studios (mocap and VP), Quixel (3D graphics), and ARRI (cameras) to launch an out-of-the-box VPS solution at SIGGRAPH 2019 in Los Angeles with the director of photography Matt Workman. Since these developments are so recent, existing literature generally focuses on technical details and individual components of the VPS system, rather than on its impact on filmmaking. The use of LED walls in a VPS makes a radical methodological and conceptual difference to live performance and to filmmaking itself. The game changer here is the link between on-stage camera and virtual in-game-engine camera that makes changes in real time on the LED wall, as virtual camera movements are updated in sync with the movements of the on-stage camera. The analysis in this paper is supported by a selected review of publications relevant to the subject to address the domains that require further investigation. Publications that offer overviews and integrated analyses are beginning to emerge, particularly as educational institutions start to develop VFX courses that encompass VP, but an in-depth conceptual analysis of the changes in progress to capture this unique historical moment is of critical importance. We present a review of how VP and game engine technologies have evolved and converged, as well as an overview of the impact of VPS on filmmaking.