Adopting metaverse‐related mixed reality technologies to tackle urban development challenges: An empirical study of an Australian municipal government

IET Smart CitiesVolume 5, Issue 1 p. 64-72 CASE STUDYOpen Access Adopting metaverse-related mixed reality technologies to tackle urban development challenges: An empirical study of an Australian municipal government Aaron An, Corresponding Author Aaron An [email protected] orcid.org/0000-0002-0330-2341 School of Electrical Engineering and Telecommunications, UNSW, Sydney, New South Wales, Australia Correspondence Aaron An. Email: [email protected]Search for more papers by this author Aaron An, Corresponding Author Aaron An [email protected] orcid.org/0000-0002-0330-2341 School of Electrical Engineering and Telecommunications, UNSW, Sydney, New South Wales, Australia Correspondence Aaron An. Email: [email protected]Search for more papers by this author First published: 14 February 2023 https://doi.org/10.1049/smc2.12051AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Fast-growing urban areas bring challenges to government authorities. The urban planning system in Australia is still very analogue with paper-based systems as standard practice. Two-dimensional document-based blueprints and Geographic Information System (GIS) maps do not always give stakeholders and investors the necessary insights. By exploring Mixed Reality (MR) application at the municipal government level, this research explains how metaverse-related immersive technologies can assist the government in better engaging with stakeholders and potential partners to improve their understanding of the urban environment and showcase future city developments to foster new opportunities and contribute to the development of the real living world. 1 INTRODUCTION Wyndham City is a municipal government located in the western suburbs of the Melbourne metropolitan area. It is one of the fastest-growing municipalities in Australia and has been experiencing challenges, including infrastructure gaps across the city for many years [1]. Until today, planning systems throughout Australia are still quite old-school, relying exclusively on document-based systems. It can be challenging for stakeholders to understand a development's real impact only based on printed planning application documents or PDF files and online maps, particularly in fast-growing urban areas. With high-rise buildings in the construction pipeline, it is challenging for urban planners, council members, and residents to understand the impact of those buildings in an established area that previously only had two to three storeys buildings. Therefore, it is becoming critically important for stakeholders to access powerful technology-based data-driven solutions to provide integrated analysis, design and visualisation for better communication and planning participation. To tackle these issues and work and engage more smartly, Wyndham City started exploring innovative ways to work with stakeholders such as residents, businesses, universities, and other city councils. Metaverse-related technologies, such as Augmented Reality (AR), have a lot of potential applications. Recent research has shown the benefits of AR systems as the tool can improve results and user satisfaction in a wide range of industry fields, such as marketing, retail, training [2] and education [3], gaming [4], health [5], tourism [6, 7]. However, limited research has been done in a government context. This research explores a case study of the first municipal government in Australia [8] to adopt such technology to improve stakeholder engagement and decision-making [9]. The CityLens project To help tackle the above challenges brought on by rapid population growth, the City of Wyndham created a groundbreaking stakeholder engagement and decision support platform using Mixed Reality (MR) technology. The project was first called 'WynLens – Visualisation, Analysis and Modelling of Holographic City' [10], a tool for developers and city residents to visualise development projects and their impacts in the city based on Microsoft's HoloLens. After winning national awards, the city council renamed the project 'CityLens' [11]. Mixed reality: Augmented Reality and Virtual Reality Compared with Virtual Reality (VR), which uses a computer-generated world to completely replace the real-world environment [12], AR uses digital technologies to enhance the real physical world. It combines the real and virtual worlds with additional digital visual elements, sound, other perceptual information, and real-time interaction [13]. As shown in Figure 1, MR is a fusion of virtual and real worlds that enables digital and physical objects to interact in real-time. MR does not take place exclusively in either realm but is a synthesis of reality and virtuality. From a virtual display perspective, [14] defined the Reality-Virtuality (RV) Continuum as shown in Figure 2. FIGURE 1Open in figure viewerPowerPoint Mixed reality is a combination of elements of human, environment, and computer [15]. FIGURE 2Open in figure viewerPowerPoint Reality-virtuality (RV) continuum. FIGURE 3Open in figure viewerPowerPoint Microsoft HoloLens [16]. Source: Cropped under CC BY 2.0 license. Microsoft HoloLens Microsoft HoloLens is a mixed-reality headset that allows users to see and interact with holograms in the real world. It was developed by Microsoft and was first released in 2016. The HoloLens uses advanced technology, such as cameras, sensors, and holographic processors, to create a mixed-reality experience. It allows users to see virtual objects and information overlaid in the real world and to interact with these holograms using natural gestures and voice commands. One of the key features of the HoloLens is its ability to enable collaboration between users in different locations. Using the HoloLens, users can share a mixed-reality experience and work together on virtual projects, even if they are physically separated by long distances. Before deciding on which device to use, Wyndham City tested several different devices, and the HoloLens became a clear choice due to important factors, such as the compatibility of existing datasets and the existing relationship with the city council. 1st generation: As shown in Figure 3, the Microsoft HoloLens features see-through holographic lenses with two high-definition 16:9 light engines which can produce 2.3M light points holographic resolution, a 32bit Intel processor, 2GB RAM, 64GB Flash, and more than ten sensors, including four environment understanding cameras, one depth camera, one 2MP still and video camera, a light sensor, four microphones and an IMU. It has a 3.5 mm audio jack, built-in speakers, Micro USB 2.0 connection port, and wireless connection via Wi-Fi 802.11ac and Bluetooth 4.1. Its battery lasts about two to three hours when used actively and lasts up to two weeks when on standby. The device uses gaze tracking, gesture input and voice to understand user actions and uses spatial sound to understand the environment. Its software is based on Windows 10, with pre-installed software including Holograms, Calibration, Learn Gestures, Windows Store, Microsoft Edge, Windows Feedback, Photos, and Settings [17]. 2nd generation: The upgrade version of the first generation, built on the Qualcomm Snapdragon 850 Compute Platform. It features 2k 3:2 light engine holographic resolution, 4GB LPDDR4x DRAM Memory, 64GB UFS 2.1 Storage, USB Type-C connection port, and wireless connectivity via Wi-Fi 802.11ac 2 × 2 and Bluetooth 5.0. It has an upgraded camera that supports 8MP photo and 1080p30 video. The device is capable of human hand tracking, eye tracking, and voice control. And uses technologies such as spatial real-time environment mesh mapping, Six Degrees of Freedom world-scale positional tracking, MR photos and videos capturing to understand the environment [18]. Its operating system is called Windows Holographic, with several specifically designed features for Hololens 2 [19]. It also added more pre-installed software, such as 3D Viewer, Cortana, File Explorer, Mail and Calendar, Movies & TV, OneDrive, Dynamics 365 Guides, and Dynamics 365 Remote Assist [18]. 2 PREREQUISITES AND DEVELOPMENTS The CityLens project combines existing 3D models with proposed capital works development layers to create a 3D city view of the future using AR holographic experiences. As shown in Figure 4, it integrates AR views with a wide range of information generated from various IoT sensors with GIS (OpenStreetMap) and the Amazon Cloud. Internet of Things (IoT) data sets, including wind speed, air quality, temperature, smart bin fill level, car park availability and traffic jams, are integrated via sensors' API endpoints and displayed as different layers on the client-side devices. FIGURE 4Open in figure viewerPowerPoint CityLens technical architecture. The underlying geospatial data is sourced from OpenStreetMap, with cached map tiles streamed from the Amazon cloud. A model of Wyndham City was created by blending a ground model based on building footprints. Traditional file formats such as OBJ and KML have been converted to Cesium 3D tiles to make it easier to display MR views on client-side devices. This process involved breaking down large models into more manageable blocks using Feature Manipulation Engine (FME) Workflows. Then code was developed to render the Cesium 3D tiles on the MS HoloLens. This allowed the heavy computational process to be handled on the server side, making it possible to display large 3D models anywhere in the world on a mobile AR headset [20]. The developed software suite also features a web-based software management tool with seamless integration of 3D-built environments and sensor-derived data. 3 RESULTS Using the mixed reality platform made possible by this project, users can visualise and interact with future building models to better understand proposed development and infrastructure proposals in better detail, enabling better decision-making. As shown in Figures 5 and 6, the user can interact with the holographic view to switch between different data sets and 3D city models by pointing fingers towards the holographic view and selecting different layers via the interactive menu interface. FIGURE 5Open in figure viewerPowerPoint CityLens integrates different data layers generated by IoT sensors. FIGURE 6Open in figure viewerPowerPoint CityLens integrates live temperature data. Figure 7 shows the smart bin data discussed in recent research on Wyndham's smart bin waste management system [21]. It shows the statistics of one set of smart bin fullness data between 1 July 2018 and 30 August 2018. In Figure 8, the CityLens project integrates the Smart Bin fullness data associated with their locations (green poles) with the 3D view of the suburb of Werribee and the already approved building developments in the suburb. FIGURE 7Open in figure viewerPowerPoint Wyndham smart bin - Park Hotel G Werribee (Serial Number 1511205) fullness stats from 1 July to 30 August 2018. FIGURE 8Open in figure viewerPowerPoint CityLens integrates Smart Bin data with 3D view of buildings in the town centre of the suburb of Werribee. Figures 9 and 10 show holographic AR views of part of the Williams Landing major activity centre with 3D models for future buildings with permits granted, together with 3D buildings already existing and live temperature. Users can click on each 3D model and view their planning permit-related information. As shown in Figure 11, MS HoloLens is used to engage with council decision-makers in indoor and outdoor environments in real-world settings. FIGURE 9Open in figure viewerPowerPoint CityLens shows the live temperature at Williams Landing with 3D models of existing buildings and 3D models of future buildings that have been approved. FIGURE 10Open in figure viewerPowerPoint CityLens integrates planning permit information with 3D buildings and real-time traffic information. FIGURE 11Open in figure viewerPowerPoint CityLens project using Microsoft HoloLens for stakeholder engagement in real-world settings: indoor (left) and outdoor (right). Mixed reality technology helps bring an abstract or technical idea into the real world so stakeholders can see it, discuss it and examine it. They could also view the plans and designs in real environments at their future locations, making it possible for them to understand the plan better than traditional artist impressions [22]. Such technology is also useful for documenting, storytelling, and speculating about the past, present and future, attracting potential investors, influencing decision-makers, and better consulting with residents and communities. The project formed a fundamental building block for Wyndham's first-ever Smart City Strategy, adopted in 2018. One of the main objectives stated in the strategy is to enable the city council to make data-driven decisions that are transparent and with confidence [23]. The CityLens project helps the council achieve this objective by better engaging with the community, businesses, other levels of government, universities, and innovators to foster new opportunities for collaboration, which also aligns with one of the strategy's main themes: "Partnership and collaboration" [23]. 4 DISCUSSIONS The implementation of pioneering innovation and exploration projects within the realm of municipal government is a challenging endeavour. This is particularly true in areas characterised by rapid population growth, diverse communities, and diverse challenges. The successful execution of such projects necessitates cultivating a collaborative environment characterised by mutual trust and support, spanning from the decision-making level of elected officials to executive and management levels within the organisation. Importance of having a supportive city council for entrepreneurial spirit Projects that utilise new and greenfield technologies will always be challenging. The courage to embark on a journey of adventure and an intrapreneurial spirit made the project possible. Pioneering the mixed reality project in local government required entrepreneurial spirit within the city council. A level of risk-taking is necessary for local governments to explore what the future may look like through exploratory projects. Wyndham city council staff expressed that working for a council willing to change the way local government works is a fortune, especially compared to other city councils that may need to be more open to adapting new technology [24]. Technical knowledge and agile project management methodology The incorporation of greenfield technologies requires a rigorous research process to establish a comprehensive understanding of interoperable display formats. This understanding enabled Wyndham to optimise the development process by reducing time and cost. Furthermore, adopting Agile project management methodology facilitated the inclusion of new features and support for additional platforms, ultimately improving the final product's overall quality. Legal aspects for collaboration and partnership A Memorandum of Understanding (MoU) is a non-binding agreement that outlines the agreed-upon terms and objectives of a partnership or collaboration between two or more parties. It does not create legal obligations but serves as a framework for the parties to build upon and formalise their relationship. Establishing contracts and executing MoUs with partners, such as other city councils and universities, can be a time-consuming endeavour. It has been observed that the process of developing legal documents and engaging partners took longer than initially anticipated. Therefore, it is recommended that partners be identified and engaged at an earlier stage in order to allocate sufficient time for the development of legal documents and ensure the smooth execution of the partnership or collaboration. Limitations, competitions, and possible future While some reports estimate the market for AR/MR is increasing dramatically in the coming years [25-28], as of 2022, not many people own an AR/MR device yet, and the devices from major companies are still quite expensive. In 2022, prices for mixed reality glasses in Australia range from approximately AUD$800 to over AUD$5,500 [29-35]. Consumer ownership of mixed reality helmets or glasses is still not as good as PCs, laptops, and smartphones. It is ok if we only need to engage with a limited number of stakeholders, but it is becoming an issue if we engage with hundreds or thousands of stakeholders. With advances in hardware technologies, more companies are entering the AR/MR field, and more manufacturers are starting to produce more affordable devices. By that time, this limitation could be resolved. Additionally, various mixed reality devices compete among themselves. For example, AR technology is also available on mobile devices like Apple's iPhone. Apple's ARKit requires an A9 or later processor with iOS 11.0 or later version of iOS [36], which means iPhones from the 6s and onwards all have AR capability [37]. Further research is needed to determine which devices are most effective and preferred by users in different circumstances. It is also worth noting that different devices may be used simultaneously to complement each other. 5 CONCLUSION In an Australian first, Wyndham City transformed how it responds to residents' requests for more innovation and efficiency through its CityLens project. The project harnessed cutting-edge mixed reality technology to develop an immersive holographic, 3D decision support system using Microsoft's HoloLens and was recognised as a game-changer by winning a national award [38]. It provides the capability to simulate future scenarios and map live data from sensors across the city. Through open data platforms such as data. gov.au, a wide range of datasets can be integrated into AR devices. By providing dynamic visualisation, contextualisation, and improved collaboration, decision-making and community engagement, this platform enhances the existing ability to transform data into information and intelligence and optimises collaboration and engagement with community stakeholders and potential investors to view future projects creatively. Government employees and relevant stakeholders can use interactive AR 3D and immersive landscapes to visually explore local surroundings and improve planning outcomes. An individual can simulate walking through and around a building at their proposed location in real environment settings to understand the proposed development's impact better. The immersive experience gives people an intuitive opportunity to interact with the urban environment, making it easier for them to understand it more quickly. The project serves as an integral component of Wyndham's pioneering and award-winning Smart City Strategy [23] and is outlined in the Project Implementation Schedule [39]. Local citizens' engagement and participation have been facilitated through the Smart City portfolio committee and a series of programs, including the Wynnovation program, the Wyndham Learning Festival and the Victorian government's Digital Innovation Festival. In order to realise more potential, it is necessary to look at extending to other mixed reality platforms and establish additional partnerships in the future. Despite current limitations, the ongoing advancement and maturation of metaverse-related mixed reality technologies are poised to provide a valuable tool for government engagement with stakeholders and hold significant promise for developing smart, liveable cities. ACKNOWLEDGEMENTS No Funding. CONFLICT OF INTEREST STATEMENT No potential conflict of interest was reported by the author. Open Research DATA AVAILABILITY STATEMENT The smart bin data used in this study are available at: https://data.gov.au/dataset/ds-dga-660a87c3-480e-498b-bfc3-ec84dc504b1c/details. REFERENCES 1 Infrastructure Victoria: Social Infrastructure in Melbourne's Growth Areas. Infrastructure Victoria (2021) 2Kaplan, A.D., et al.: The effects of virtual reality, augmented reality, and mixed reality as training enhancement methods: a meta-analysis. Hum. Factors 63(4), 706– 726 (2021). https://doi.org/10.1177/0018720820904229 3Yuviler-Gavish, N., et al.: Augmented virtuality systems as a tool for improving numeracy decision-making among children. Simulat. Gaming 53(4), 317– 334 (2022). https://doi.org/10.1177/10468781221099358 4Nevelsteen, K.J.L.: Virtual world, defined from a technological perspective and applied to video games, mixed reality, and the Metaverse. Comput. Animat. Virtual Worlds 29(1), e1752 (2018). https://doi.org/10.1002/cav.1752 5Tepper, O.M., et al.: Mixed reality with HoloLens: where virtual reality meets augmented reality in the operating room. Plast. Reconstr. Surg. 140(5), 1066– 1070 (2017). https://doi.org/10.1097/prs.0000000000003802 6Bec, A., et al.: Virtual reality and mixed reality for second chance tourism. Tourism Manag. 83, 104256 (2021). https://doi.org/10.1016/j.tourman.2020.104256 7Trunfio, M., Campana, S.: A visitors' experience model for mixed reality in the museum. Curr. Issues Tourism 23(9), 1053– 1058 (2019). https://doi.org/10.1080/13683500.2019.1586847 8 Municipal Association of Victoria: CityLens Project – Wyndham City Council. Municipal Association of Victoria (2019). https://www.viccouncils.asn.au/newsroom/resources/case-studies/citylens-project-wyndham-city-council. Accessed December 2021 9 Monkey Media: IoT Sensor Networks Creating Smart Cities. Monkey Media (2018). https://councilmagazine.com.au/iot-sensor-networks-creating-smart-cities. Accessed November 2021 10 The Department of Infrastructure, Transport, Regional Development and Communications, 2017 Winners. The Department of Infrastructure, Transport, Regional Development and Communications (2017). https://www.infrastructure.gov.au/territories-regions-cities/local-government/national-awards-local-government/past-winners/2017-winners. [Accessed November 2021] 11 Wyndham City Council: CityLens. Wyndham City Council (2018). https://www.wyndham.vic.gov.au/project/citylens. Accessed November 2021 12Steuer, J.: Defining virtual reality: dimensions determining telepresence. J. Commun. 44(2), 73– 93 (1992). https://doi.org/10.1111/j.1460-2466.1992.tb00812.x 13Wu, H.-K., et al.: Current status, opportunities and challenges of augmented reality in education. Comput. Educ. 62(2013), 41– 49 (2013). https://doi.org/10.1016/j.compedu.2012.10.024 14Milgram, P., et al.: Augmented Reality: a class of displays on the reality-virtuality continuum. In: Proceedings of SPIE – the International Society for Optical Engineering (1995) 15Wen, Q., et al.: What is mixed reality? (2021). https://docs.microsoft.com/en-us/windows/mixed-reality/discover/mixed-reality. Accessed December 2021 16 Microsoft Sweden: File: HoloLens 2.Jpeg. Microsoft Sweden (2015). https://en.wikipedia.org/wiki/File:HoloLens_2.jpeg. Accessed December 2021 17Zeller, M., et al.: HoloLens (1st Gen) Hardware. Microsoft (2021). https://docs.microsoft.com/en-us/hololens/hololens1-hardware. Accessed December 2021 18Cooley, S., et al.: About HoloLens 2. Microsoft (2021). https://docs.microsoft.com/en-us/hololens/hololens2-hardware. Accessed December 2021 19Cooley, S., et al.: HoloLens 2 Release Notes. Microsoft (2021). https://docs.microsoft.com/en-us/hololens/hololens-release-notes. Accessed December 2021 20An, A.: CityLens Project Hitting Key Milestones (2018). https://www.smartcity.org.au/article/citylens-project-hitting-key-milestones. Accessed December 2021 21An, Y., Qiu, J., Dong, Z.: Smart Waste Management System for Decision Makers by Using Smart Bins - a Case Study for an Australian Municipality. Australian Journal of Civil Engineering (2022) 22An, A.: Wyndham Fast Becoming A Smart City (2017). https://www.smartcity.org.au/article/wyndham-fast-becoming-smart-city. Accessed December 2021 23 Wyndham City Council: Smart City Strategy 2019–2024. Wyndham City Council(2019) 24Gutierrez, P.: Wyndham Council Uses AR to Visualise the Future. nextmedia Pty Ltd. (2017). https://www.iothub.com.au/news/wyndham-council-uses-ar-to-visualise-the-future-461382. Accessed November 2021 25 GlobeNewswire, Inc.: Global Augmented Reality Market Report 2021: Surging Demand for AR Devices and Applications in Healthcare - Forecast to 2026. GlobeNewswire, Inc. (2021). https://www.globenewswire.com/en/news-release/2021/10/06/2309582/28124/en/Global-Augmented-Reality-Market-Report-2021-Surging-Demand-for-AR-Devices-and-Applications-in-Healthcare-Forecast-to-2026.html. Accessed December 2021 26 Statista: Augmented (AR), Virtual Reality (VR), and Mixed Reality (MR) Market Size Worldwide from 2021 to 2024. Statista (2021). https://www.statista.com/statistics/591181/global-augmented-virtual-reality-market-size. Accessed December 2021 27 Grand View Research, Inc.: Augmented Reality Market Size, Share & Trends Analysis Report by Component, by Display (HMD & Smart Glass, HUD, Handheld Devices), by Application, by Region, and Segment Forecasts, 2021 – 2028. Grand View Research, Inc. (2021). https://www.grandviewresearch.com/industry-analysis/augmented-reality-market. Accessed December 2021 28 Yahoo! Finance: The Augmented Reality Market Was Valued at USD 14.7 Billion in 2020 and Is Projected to Reach USD 88.4 Billion by 2026; it Is Expected to Grow at a CAGR of 31.5% from 2021 to 2026. Yahoo! Finance (2021). https://finance.yahoo.com/news/augmented-reality-market-valued-usd-111700555.html. Accessed December 2021 29 Microsoft,: HoloLens 2 – Pricing and Options. Microsoft (2022). https://www.microsoft.com/en-au/hololens/buy. Accessed December 2022 30 Magic Leap, Inc., Magic Leap 2. Magic Leap, Inc. https://www.magicleap.com/magic-leap-2. [Accessed December 2022] 31 Amazon.com, Inc.: Google Glass Explorer Edition XE V2 (BLACK/GRAY), Amazon.Com, Inc., (2022). https://www.amazon.com/Google-Glass-Explorer-BLACK-GRAY/dp/B00I4CLB6I. Accessed December 2022 32 Lenovo: ThinkReality A3 Smart Glasses. Lenovo (2022). https://www.lenovo.com/us/en/p/smart-devices/virtual-reality/thinkreality-a3/wmd00000500. Accessed December 2022 33 Iristick: Order Now - Iristick Smart Glasses Webshop - Iristick Smart Glasses. Iristick (2022). https://iristick.com/store. Accessed December 2022 34 Vuzix: Vuzix Blade 2 Smart Glasses. Vuzix (2022). https://www.vuzix.com/products/vuzix-blade-2-smart-glasses. Accessed December 2022 35 Amazon.com, Inc.,: ThirdEye X2 MR Glasses (2021). https://www.amazon.com.au/dp/B08GCYNCBM/ref=cbw_us_au_dp_ags?smid=A4XRJ8S0WXSO0 36 Apple: Verifying Device Support and User Permission. Apple (2021). https://developer.apple.com/documentation/arkit/verifying_device_support_and_user_permission 37 Apple: iPhone 6s - Technical Specifications. Apple (2021). https://support.apple.com/kb/sp726?locale=en_US 38 Australian Local Government Association: Wyndham City Council Wins 2017 National Award for Excellence in Local Government. Australian Local Government Association. https://alga.asn.au/15124-2/. Accessed November 2021 39 Wyndham City Council: Smart City Implementation Plan. Wyndham City Council(2019) Volume5, Issue1March 2023Pages 64-72 FiguresReferencesRelatedInformation