## 1 Introduction  
Knowledge distillation is a favored method for transferring information from one domain to another. This technique involves a teacher model transmitting knowledge to a simpler student model. The differences between these models typically pertain to network parameters such as the number of layers and the size of weight matrices. The student model aims to replicate the representations taught by the teacher as closely as possible. However, training a student model through knowledge distillation can be time-consuming and resource-intensive. Furthermore, the resulting student model may still lack predictive capabilities, leading to inefficient energy usage.  
Contrary to existing literature on knowledge distillation, this paper explores scenarios where applying distillation methods can be challenging. These situations may arise when only partial access to model parameters is available or when different models need to be redeveloped using the same programming language and deep learning framework (source code for all experiments and used data are shared publicly on [[https://github.com/kadir-gunel/BoostingSentenceEmbeds--accessed](https://github.com/kadir-gunel/BoostingSentenceEmbeds--accessed)]([https://github.com/kadir-gunel/BoostingSentenceEmbeds--accessed](https://github.com/kadir-gunel/BoostingSentenceEmbeds--accessed)) on 23 October 2023).  
This paper primarily investigates knowledge transfer between two distinct model architectures: _FastText_ and _sBert_. While the training language is English, these models differ completely in architecture, training data, and output vector dimensions. Notably, the end user doesn't have access to the model parameters of _FastText_ pre-trained word embeddings. Moreover, both models are implemented using different programming languages.  
From a model perspective, FastText excels in speed by leveraging its trained word embeddings to calculate sentence vectors through a _bag-of-words_ approach, averaging the word vectors within a sentence. This methodology enables _FastText_ to generate sentence embeddings with remarkable rapidity, facilitating highly efficient vector computationsthat do not require the use of a GPU. However, this speed comes at the cost of accuracy, marking a significant trade-off. In contrast, sBert employs its transformer architecture to craft sentence embeddings. Although this slows down the generation process due to its computational intensity which requires use of a GPU, it results in better performance across various NLP tasks. This trade-off underscores sBert's enhanced sentence representation capabilities, where it sacrifices speed for a higher degree of sentence representation, as compared to FastText.  
Our aim is to design and evaluate models that simultaneously address word-level and sentence-level representations. We propose models that seek to bridge the gap between FastText word embeddings and sBert sentence vectors by producing more closely aligned sentence representations. To facilitate the information transfer, the WMT EN-ES dataset is used. Subsequently, the validation of the proposed methods is conducted through classification and semantic task similarity (STS) assessments from the SentEval datasets. The paper is organized as follows:  
* An overview of the literature (Section 2);
* Proposed methodologies for constructing knowledge tranfer between FastText word embeddings and sBert: distance minimization and distance and perplexity minimization (Section 3);
* Used datasets for training and testing proposed models on various tasks (Section 4);
* The techniques and model configuration for training (Section 5);
* Represent the results of proposed models for knowledge transfer by evaluating obtained results on word and sentence level tasks (Section 6);
* Evaluate the obtained results, how the proposed models affect the word embedding space, and what the possible solutions are (Section 7).