# MpcNet: A Vision Transformer Network for Estimating Room Layouts from a Single RGB Panorama
## 1 Introduction  
When humans receive information from the outside world, more than 70% of the information is obtained through vision [1]. Computer vision is a technology that combines the eye (camera) with the brain (algorithm). This allows the computer to recognize, understand and analyze its environment, and to independently control behavior and solve problems. Computer vision developed significantly in the 1960s, covering a wide range of fields, such as image signals, texture and color modeling, geometric processing and reasoning and object modeling. The problem of estimating the 3D room layout from a single panoramic image is a subproblem of the problem of object modeling. It focuses on reconstructing three-dimensional shapes from two-dimensional images, such as spheres and cuboids. In the process of the high-quality reconstruction of a 3D room layout using panoramic images, understanding the room scene plays an extremely important role in the effect of the 3D reconstruction. For example, the floor is used as a support surface for tables and chairs, and objects such as paintings are typically aligned with the walls. Due to the rise of machine learning, scholars have tried to predict the corners and boundaries of rooms in panoramic images based on machine learning. [PERSON] et al. [2] used the Bayesian method to combine visual cues with some prior knowledge of the scene to reconstruct 3D structures. Specifically, this method generated layout assumptions based on line segments extracted from a single indoor image and selected the most appropriate layout from them. [PERSON] et al. [3] converted the indoor layout estimation problem into the conditional random field inference problem. They chose to use a voting mechanism to identify and classify the connection points and then introduced the characteristics of the connection points into the conditional random field inference problem, to estimate the layout of indoor scenes. Some subsequent studies also adopted similar methods to improve the extraction method of feature clues and the mechanism of generating the layout [4]. In general, however, the above methods of 3D room reconstruction using traditional machine learning methods have some drawbacks, such as poor model expression ability, a low amount of data processing and a tendency toward overfitting. Thanks to the rapid development of deep learning, the 3D reconstruction of a room layout has gained new possibilities, and scholars have begun to use deep convolutional neural networks to explore 3D reconstruction problems. HorizonNet [5] proposed the combination of ResNet [6] and a recurrent neural network (RNN) for 3D room layout reconstruction. [PERSON] et al. [7] proposed a method based on a convolution neural network for the task of predicting the surface normal from a single image. Intuitively, the problem of 3D room layout reconstruction should focus on the shape bias of the indoor layout instead of the texture bias. For example, the distance between two objects can be considered, as shown in Figure 1.  
However, CNNs are limited by the problem of receptive fields; for example, a common convolution kernel is \(3\times 3\). Typical hidden layer feature sizes range from 256,512 to 1024, and the information felt by the 3 \(\times\) 3 convolution kernel under the feature of these dimensions is very local. Increasing the size of the convolution kernel increases the receptive field, but it also brings about a large increase in the number of parameters and a delay in inference speed. For instance, suppose the size of the input feature is 64,384,64,64; we input this into a 24-layer depthwise convolution with a 13 \(\times\) 13 kernel size, implemented with Pytorch, and with a latency of 600 ms [8]. Recently, Vision Transformer (ViT) [9], with its global receptive field and dynamic interaction capabilities, has been used to fully represent and model the context relationships in the image, achieving better results in many downstream tasks such as semantic segmentation and image classification.