Title: TAN: Topological Attention Networks for Hierarchical Document Understanding
Primary area: learning on graphs and other geometries & topologies
Keywords: Transformers, Self-attention, Topological Attention Networks (TAN), Topological Data Analysis (TDA), Persistent Homology, Locality-Sensitive Hashing (LSH), Hierarchical Document Understanding, Long-Context Processing

Abstract: Transformers have achieved strong performance in natural language processing through self-attention mechanisms, yet their quadratic complexity presents challenges for scalability to long documents and hierarchical structures. We introduce Topological Attention Networks (TAN), an architecture that integrates topological data analysis into the attention mechanism to capture multi-scale structural patterns in text. TAN employs a locality-sensitive hashing (LSH) scheme guided by topological features to reduce attention complexity while preserving structural relationships in documents. Our approach computes k-nearest neighbor graphs in embedding space and aggregates features through learnable topological encoders, enabling efficient processing of documents with complex structures. We provide theoretical analysis showing improved complexity bounds and evaluate TAN on diverse benchmarks, including GoEmotions (emotion classification) and LEDGAR (legal document categorization). Experiments demonstrate that TAN achieves 64.79% F1-macro score on GoEmotions, significantly outperforming XLNet (51.92%), DistilBERT (49.29%), and DeBERTa-v3 (48.98%) with statistical significance (p < 0.001), while maintaining O(n log n) computational complexity compared to O(n²) for standard attention mechanisms.