Telecommunications
Smart cities, Urban / Architecture / Civil engineering, DXMR w/ AI: Assessing future landscapes using enhanced
mixed reality with segmentation by deep learning
Division of Sustainable Energy and Environmental Engineering, Graduate School of Engineering
Associate Professor Tomohiro Fukuda
Researchmap https://researchmap.jp/tomohirofukuda?lang=en

Telecommunications

Abstract
Architecture, engineering, and construction projects need to bepromoted in harmony with the natural environment and with theaim of preserving people’s living environment. At the planning anddesign stage, decision-makers and stakeholders share and assesslandscape images during and after construction in order to avoid
as much uncertainty as possible when performing environmental
impact assessment. Given the lack of a standard visualization
method for future landscapes that do not yet exist, mixed reality(MR), which overlays virtual content onto a real scene, has attracted attention in the field of landscape design. One challenge in MR
is occlusion, which occurs when virtual objects obscure physicalobjects that should be rendered in the foreground. In MR-basedlandscape visualization, the distance between the MR camera andreal objects located in front of the virtual objects might vary andmight be large, causing difficulty for existing occlusion handling
methods. In the process of landscape design, an evidence-based
approach has also become important. Landscape index estimation
using segmentation by deep learning, which can recognize thesurrounding environment, has been actively studied for landscapeassessment. In this study, segmentation by deep learning was integrated into an MR system to enable dynamic occlusion handlingand landscape index estimation for both existing and designed
landscape assessment. This system can be operated on a mobile
device with video communication over the internet by connecting
to real-time segmentation on a high-performance personal computer. The applicability of the developed system is demonstrated
through accuracy verification and case studies.

parameter setting, unextracted tree branches and trunks, and incorrect extraction of artificial green objects. On the other hand, theuse of segmentation solves these problems, but a landscape evaluation method integrated with MR for both current and post-designlandscapes has not yet been realized. Therefore, the novelty of thisresearch is the development of a method for achieving dynamic
occlusion and quantitative landscape assessment for both current
and post-design landscapes by integrating MR and segmentation.

Fig. 1 Proposed MR system: Integration of semantic segmentation into MR▍ Significance of the research and Future perspective
We are using deep learning to improve the performance of mobile mixed reality, which is expected to be applied in the industrial
and urban planning fields, especially in construction DX toward Society 5.0. In the future, we will automate the definition of occlusion
targets and improve the level of detail from the class (type) to theobject (individual object of the same type) level.

▍ Background & Results
We propose an MR future landscape assessment system that
includes dynamic occlusion handling for more accurate geometricconsistency and landscape index estimation for evidence-based
design. For occlusion, model-based methods have been proposed
for realizing large-scale MR in outdoor areas, but they are not applicable in cases where occlusion targets move or change shape.
In contrast, methods using depth information, which enables dynamic occlusion, are difficult to achieve for large-scale MR thatexamines outdoor landscapes because the depth information thatcan be obtained is limited. For landscape evaluations such as GVI,
image processing methods can be integrated with MR to evaluate
the current and post-design states, but there are issues such as

Fig. 2. MR-based visualization with dynamic occlusion handling in field validation.

Fig. 3. Planting simulation with MR and greenery estimation.

P a t e n tKido, Daiki; Fukuda, Tomohiro; Yabuki, Nobuyoshi. Assessing future landscapes using enhanced mixed reality with semantic segmentation by deep learning. Advanced Engineering Informatics. 2021; 48: 101281. doi: 10.1016/j.aei.2021.101281
T r e a t i s e Nakabayashi, Mizuki; Fukuda, Tomohiro; Yabuki, Nobuyoshi. Mixed Reality Landscape Visualization Method with Automatic Discrimination Process forDynamic Occlusion Handling Using Instance Segmentation. Proceedings of the 39th eCAADe Conference, Volume 2: 539-546.
http://papers.cumincad.org/data/works/att/ecaade2021_038.pdf
https://resou.osaka-u.ac.jp/ja/research/2021/20210324_1
URL
http://www.archifuture-web.jp/magazine/622.html
K e y w o r d mixed reality, deep learning, landscape metrics and indices, environmental design, systematization