###### Abstract  
Vision Transformers (ViTs) have shown their superiority in various visual tasks for the capability of self-attention mechanisms to model long-range dependencies. Some recent works try to reduce the high cost of vision transformers by limiting the self-attention module in a local window. As a price, the adopted window-based self-attention also reduces the ability to capture the long-range dependencies compared with the original self-attention in transformers. In this paper, we propose a Local and Global Vision Transformer (LGViT) that incorporates overlapping windows and multi-scale dilated pooling to robust the self-attention locally and globally. Our proposed self-attention mechanism is composed of a local self-attention module (LSA) and a global self-attention module (GSA), which are performed on overlapping windows partitioned from the input image. In LSA, the key and value sets are expanded by the surroundings of windows to increase the receptive field. For GSA, the key and value sets are expanded by multi-scale dilated pooling to promote global interactions. Moreover, a dynamic contextual positional encoding module is exploited to add positional information more efficiently and flexibly. We conduct extensive experiments on various visual tasks and the experimental results strongly demonstrate the outperformance of our proposed LGViT to state-of-the-art approaches. vision transformer; visual backbone; overlapping windows Article  
LGViT: A Local and Global Vision Transformer with Dynamic Contextual Position Bias Using Overlapping Windows April. 202313  
methods partition the input images into non-overlapping windows, as can be seen in Figure 1, the computational area of self-attention is limited to a fixed-size window. The number of patches in each window is much smaller than the size of the whole feature map, thus leading to linear complexity and locality. In such a situation, the receptive field of self-attention is restricted in each local window or group, which cripples the modeling power of the original self-attention mechanism. To build connections across windows, halo [22] and shift [23] operations are adopted to exchange information through nearby windows. However, the receptive field is still limited in several neighboring windows at one stage and stacked blocks are needed to achieve more global self-attention like CNNs. For some dense prediction tasks, the global semantic information is of vital importance, therefore, the larger receptive field may lead to better performance. To overcome the locality limitation of CNNs and reduce the computation cost of transformers, in this paper, we present a novel Local and Global Vision Transformer named LGViT. As illustrated in Figure 2, LGViT is based on our well-designed window-based self-attention mechanism. Such a self-attention mechanism contains a local self-attention module (LSA) to enlarge the receptive field of each window and a global self-attention module (GSA) to obtain contextual information globally. For the naive window-based self-attention mechanism, the receptive field and computational cost are both proportional to the square of window size. In LSA, we adopt the overlapping window design. Specifically, instead of directly enlarging the local window size, we use the overlapping window to expand the key and value sets while keeping the query set in a non-overlapping window. In such a design, the receptive field can be enlarged with a much less cost compared to the naive window-based self-attention mechanism. For the GSA, similar to dilated convolution we utilize the multi-scale dilated pooling with different dilation rates to extend the key set and value set. As a result, global interactions on the whole input feature map/image can be captured. Like previous Vision Transformers, LGViT also requires position embedding to retain the spatial position information among input patches.