Convolutional Set Transformer
Authors: name: Federico Chinello; name: Giacomo Boracchi
Published: 2025-09-26 20:13:00

Abstract: We introduce the Convolutional Set Transformer (CST), a novel neural
architecture designed to process image sets of arbitrary cardinality that are
visually heterogeneous yet share high-level semantics - such as a common
category, scene, or concept. Existing set-input networks, e.g., Deep Sets and
Set Transformer, are limited to vector inputs and cannot directly handle 3D
image tensors. As a result, they must be cascaded with a feature extractor,
typically a CNN, which encodes images into embeddings before the set-input
network can model inter-image relationships. In contrast, CST operates directly
on 3D image tensors, performing feature extraction and contextual modeling
simultaneously, thereby enabling synergies between the two processes. This
design yields superior performance in tasks such as Set Classification and Set
Anomaly Detection and further provides native compatibility with CNN
explainability methods such as Grad-CAM, unlike competing approaches that
remain opaque. Finally, we show that CSTs can be pre-trained on large-scale
datasets and subsequently adapted to new domains and tasks through standard
Transfer Learning schemes. To support further research, we release CST-15, a
CST backbone pre-trained on ImageNet
(https://github.com/chinefed/convolutional-set-transformer).