## 1 Introduction  
Enzymes derived from thermophilic organisms often have great practical utility as biocatalysts. While higher temperatures tend to increase catalytic efficiency, conventional enzymes denature and lose effectiveness in high-temperature environments [1]. However, thermophilic proteins to do not suffer from this limitation, making them attractive for use in biotechnology. Historically, various predictive methods using machine learning have been proposed to identify thermophilic proteins based on their amino acid sequence. However, these algorithms cannot directly use the protein sequence information as training input, necessitating various feature extraction methods to construct a feature matrix, typically requiring extensive human interventions [2]. The features extracted in this process, including amino acid composition [2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12], composition transition and distribution [2; 7; 11; 12; 13], and dipeptide deviation from the expected means [10; 14; 15], all have clear physicochemical relevance. Several machine learning classification schemes were proposed based on one or more of these physical and chemical features. Attempts to improve the accuracy of prediction relied on refining the aspects of feature extraction [16; 17], feature selection [18; 19; 20; 21], and better classifier selection [22; 23]. [PERSON] et al. [9] re-encoded amino acid sequences with RAAC [24], extracted physicochemical characteristics [25; 26], utilized auto-cross covariance [27], and reduced the number of dipeptides, resulting in an algorithm with an accuracy of 0.982 in 10-fold cross-validation testing. In terms of feature selection, [PERSON] et al. [2] introduced MRMD2.0 as a feature selection algorithm to reduce the feature matrix to 119 dimensions, achieving a final accuracy of 0.9602. To improve classifier use, [PERSON] et al. [10] adopted multi-layer perceptron as a machine learning algorithm, obtaining an accuracy of 0.9626 in independent tests. Of these approaches, re-encoding the amino acid sequence with RAAC was the most promising, achieving an accuracy of 0.982. Furthermore, as in these tests only 500 of the 915 randomly selected thermophilic proteins and 500 of the 793 non-thermophilic proteins were used for training purposes from a pre-existing dataset, these results could be theoretically improved further. In 2023, a model named DeepTP [12] was constructed using six artificial design features as inputs, combined with attention mechanism network and multilayer perceptron as classifiers. DeepTP achieved an accuracy of 87.1% for 10-fold cross-validation with the new benchmark dataset; DeepTP was also with better independently tested results than those of reported methods for testing in various scenario, including a balanced dataset, a non-balanced dataset and homologous data. Recent rapid developments in the field of natural language processing (NLP) made it possible to directly extract information from proteins using self-supervised learning algorithms [28; 29; 30; 31]. These models treat protein sequences as natural language, interpreting amino acids as words in a sentence [32] and extracting meaningful information directly from the sequence [33]. Several deep representation learning methods have been used for such protein sequence analysis, including BERT [32; 34], UniRep [35], Word2 Vec (W2V) [36], and SSA [37]. These algorithms were utilized to predict protein interaction sites [38], and to identify peptides with bitter [39] or umami taste [40]. The work presented here extends this use of deep representation learning to extract features of amino acid sequences suitable to predict thermophilic characteristics in proteins [41]. This work is the first description of using sequence-based bidirectional representations from transformer (BERT) embedding features for this purpose. We named the method as BertThermo, which embedded amino acid via pre-trained BERT model and then used logistic regression as classifiers.