Title: Performance vs interpretability trade-off of hand-crafted and language model features: The case of protein superfamily classification
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: Feature engineering, interpretability, proteins, CATH superfamily, hand-crafted features, attention matrix, protein language models, class imbalance

Abstract: The newfound rise of protein language models (PLMs) that leverage data and compute has introduced an interesting conflict in computational biology: a trade-off between the high predictive performance of non-interpretable features and the scientific insight that can be gained from interpretable, hand-crafted ones. In this work, we highlight and study this conflict via the task of classifying protein domains into their CATH superfamilies. We train one-vs-all linear SVM classifiers for 45 CATH superfamilies, each characterised by significant class imbalance. We address the class imbalance by using a class-balanced loss function and the arithmetic mean (AM) of specificity and sensitivity for evaluation. Our analysis compares nine feature vector types, which are either non-interpretable embeddings from PLMs or interpretable hand-crafted features. The latter includes amino acid composition (AAC), di- and tri-peptide composition (DPC, TPC), and novel sequence-order (2OAAC, 3OAAC) and structure-based features (OCPC, CSIC). Our results demonstrate that PLM-based features achieve superior test AM scores of 90-99\% with low variability, outperforming hand-crafted features by 20-30\%. While PLM features yield high classification accuracy, their lack of interpretability obscures the underlying biological determinants. Conversely, the interpretability of hand-crafted features, despite their relatively low performance, can be leveraged to infer sequence and structural characteristics of CATH superfamilies. The proposed hand-crafted CSIC feature stikes a balance between predictive performance and interpretability, because it overfits to a lesser extent. This can be valuable for downstream applications like investigating protein-related diseases and guiding rational protein design.