## 2 Literature  
The development of sentence vector representations has been a prolific area of study. Initially, methods centered on averaging pretrained word embeddings, as demonstrated by [1; 2; 3]. Building upon this foundation, Ref. [4] broadened the representation from word level to varying lengths of text, encompassing sentences, paragraphs, and documents. This advancement empirically outperformed the traditional bag-of-words model in various text classification and sentiment analysis tasks, showcasing the ability to surmount the limitations inherent in the conventional averaging approach. Further developments included techniques such as subtracting principal components [5] and adopting the skip-gram methodology [6], which [7] later refined for expedited training.  
Concurrently, breakthroughs in neural architectures have revolutionized sentence processing [8]. A multitude of models have emerged to strike a balance between accuracy and computational efficiency, delivering proficient performance across diverse language tasks [9; 10]. The advent of sophisticated models such as BERT has not only broadened the horizons of transfer learning but also catalyzed progress in the field of knowledge distillation within language understanding tasks [11; 12; 13]. These advancements have spurred researchers to explore diverse strategies for sentence embedding, with a notable direction being the adoption of siamese network architectures that harness BERT's robust capabilities. This innovative approach (sBert) not only leverages BERT's advanced deep learning features to produce embeddings that effectively capture complex semantic relationships within sentences but also accelerates the embedding generation process, offering a more efficient solution [14].  
Emerging from this context, knowledge distillation as a concept has been rigorously explored as a strategic means to transfer the essence of complex models into more simplified counterparts [15]. Typically, these approaches involve models with identical architectures, utilizing logits (pre-softmax outputs) from a teacher model to guide the student model towards producing similar outputs. [PERSON]. [16; 17] have integrated these approaches for language-agnostic sentence embeddings. Notably, these methods do not attempt to transfer knowledge between different architectures.  
As the utilization of large scale pre-trained models in NLP expands, the issue of managing these models within limited computational resources becomes increasingly evident. In response, [PERSON]. [18] introduced DistilBERT, a smaller, versatile language representation model that applies knowledge distillation during pre-training. This technique reduces the model's size while maintaining robust language comprehension, thus offering a practical solution for on-device calculations and an array of tasks.  
Despite the increasing complexity of neural networks for NLP, as exemplified by [PERSON] and GPT, some argue that simple neural networks can still compete if they are enhanced through knowledge distillation from these advanced models. [PERSON]. [19] have demonstrated that with distillation, a single-layer BiLSTM and its siamese variant can perform on par with their more intricate counterparts, albeit with far fewer parameters and diminished inference time.  
In a recent study, [PERSON]. [20] presented a model that transfers knowledge from a transformer model to a convolutional model in the realm of image classification. Their approach aligns the student and teacher features in two projected feature spaces.  
Our study introduces a novel approach in the field of sentence embeddings and knowledge transfer by employing a unique distillation method. Unlike previous works, we harness the distinct architectures of pre-trained _FastText_ word embeddings with _sBert_ sentence vector representations. This integration addresses the gap between robust sentence representation models and the need for computationally feasible solutions, charting a course for future research in the area.