Modeling Long-Range Interactions Without Attention
By: Nabil Madali
Tags: Machine Learning, Computer Vision, Transformers, Object Detection, Self Attention

Transformer technology, which has become an important force in the field of natural language processing (NLP), has recently begun to show its strength in the field of computer vision.However, the quadratic memory footprint of self-attention has hindered its applicability to long sequences or multidimensional inputs such as images which typically contain tens of thousands of pixels.

Comparison between attention and lambda layers. (Left) An example of 3 queries and their local contexts within a global context. (Middle) The attention operation associates each query with an attention distribution over its context. (Right) The lambda layer transforms each context into a linear function lambda that is applied to the corresponding query.Source[1]

Based on the above limitations, [1] proposes termed lambda layers, or Lambda Layer, which provides a general framework for capturing the long-term interaction between the structured collection of model input and context information. It captures such interactions by converting the available context into linear functions (called lambdas) and applying these linear functions to each input separately.

Let

R^v

denote structured collections of vectors, respectively referred to as the the queries and the context.

We consider the general problem of mapping a query (q_n, n) to an output vector y_n ∈ R^{|v|} given the context C with a function F :

Such a function may act as a layer in a neural network when processing structured inputs.

A lambda layer takes the inputs

and the context

as input and generates linear function lambdas that are then applied to the queries, yielding outputs

The lambda layer first computes keys and values by linearly projecting the context, and keys are normalized across context positions via a softmax operation yielding normalized keys K¯ as following:

Where,

Its implementation can be viewed as a form of functional message passing.

With each context element contributing a content function

and a position function

Where,

is a positional embedding for the relation (n, m).

The e λ_n function is obtained by summing the contributions from the context as

The content lambda λ^c is invariant to permutation of the context elements, shared across all query positions n and encodes how to transform the q_n solely based on the context content. In contrast, the position lambda λ^{p}_n encodes how to transform the query content qn based on the content c_m and positions (n, m), enabling modeling structured inputs such images.

The output of the lambda layer is obtained as

Where,

This process captures dense content and position-based long-range interactions without producing attention maps.

[1] propose to decouple the time and space complexities of our lambda layer from the output dimension d. Rather than imposing |v|=d, they create |h| queries {q^{h}_ {n}}, apply the same lambda function λ_n to each query q^{h}_ {n}, and concatenate the outputs as:

[1] refer to this operation as a multiquery lambda layer as each lambda is applied to |h| queries.

The researchers conducted control experiments to compare LambdaNetworks with baseline ResNet50, channel attention and previous use of self-attention to supplement or replace Research method of 3x3 convolution in ResNet50. The results show that when the parameter cost is only a small part of other methods, the lambda layer is significantly better than these methods, and compared with Squeeze-and-Excitation (channel attention) to achieve +0.8% improvement.

Comparison of the lambda layer and attention mechanisms on ImageNet classification with a ResNet50 architecture. The lambda layer strongly outperforms alternatives at a fraction of the parameter cost. We include the reported improvements compared to the ResNet50 baseline in subscript to account for training setups that are not directly comparable.Source[1]

The researchers compared the lambda layer and the self-attention mechanism, and gave a comparison of their throughput, memory complexity, and ImageNet image recognition accuracy. This result shows the insufficient attention mechanism. In contrast, the lambda layer can capture global interactions on high-resolution images, and can achieve a 1.0% improvement over the local self-attention mechanism, while running almost three times faster than the latter.

The lambda layer reaches higher accuracies while being faster and more memoryefficient than self-attention alternatives. Inference throughput is measured on 8 TPUv3 cores for a ResNet50 architecture with input resolution 224x224.Source[1]

In addition, location embedding can also be shared between lambda layers, further reducing memory usage requirements with minimal downgrade costs. Finally, lambda convolution has linear memory complexity, which is very useful when encountering very large pictures in image detection and segmentation tasks.

The researchers found that it is possible to construct LambdaResNets to improve the parameters and flops efficiency of large-scale EfficientNets.

(Left) LambdaResNets improve upon the parameter-efficiency of large EfficientNets.(Right) LambdaResNets improve upon the flops-efficiency of large EfficientNets.Source[1]

Such results indicate that the lambda layer may be very suitable for use in resource-limited scenarios, such as embedded vision applications.

Finally, the researchers evaluated the effectiveness of LambdaResNets using the Mask-RCNN architecture for target detection and power segmentation tasks on the COCO dataset.

COCO object detection and instance segmentation with Mask-RCNN architecture on 1024x1024 inputs. Mean Average Precision (AP) is reported at three IoU thresholds and for small, medium, large objects (s/m/l).Source[1]

Using the lambda layer will produce consistent gains on all IoU thresholds and all object proportions (especially small objects that are difficult to locate), which indicates that the lambda layer is easy to achieve good results in more complex vision tasks that require positioning information.

Conclusion

Experiments on ImageNet classification and COCO object detection and instance segmentation demonstrate that LambdaNetworks significantly outperform their convolutional and attentional counterparts while being more computationally efficient. Finally, LambdaResNets, a family of architectures that considerably improve the speed-accuracy tradeoff of image classification models. LambdaResNets reach state-of-the-art accuracies on ImageNet while being ∼4.5x faster than the popular EfficientNets on modern machine learning accelerators.

References

1.Anonymous.LambdaNetworks: Modeling long-range Interactions without Attention,ICLR 2021.