Semantic Word and Sentence Embeddings Compression using Discrete Wavelet Transform (2024)

Abstract: Wavelet transforms, a powerful mathematical tool, have been widely used in different domains, including Signal and Image processing, to unravel intricate patterns, enhance data representation, and extract meaningful features from data. Tangible results from their application suggest that Wavelet transforms can be applied to NLP capturing a variety of linguistic and semantic properties. In this paper, we empirically leverage the application of Discrete Wavelet Transforms (DWT) to word and sentence embeddings. We aim to showcase the capabilities of DWT in analyzing embedding representations at different levels of resolution and compressing them while maintaining their overall quality. We assess the effectiveness of DWT embeddings on semantic similarity tasks to show how DWT can be used to consolidate important semantic information in an embedding vector. We show the efficacy of the proposed paradigm using different embedding models, including large language models, on downstream tasks. Our results show that DWT can reduce the dimensionality of embeddings by 50-93% with almost no change in performance for semantic similarity tasks, while achieving superior accuracy in most downstream tasks. Our findings pave the way for applying DWT to improve NLP applications.

heading: 1 Introduction
text: Embedding models have evolved as a crucial part of any NLP application. Typically, they transform text, using different numerical analysis methods, into high-dimensional dense vectors that capture semantic and contextual aspects of the text for subsequent use by various tasks. As these models evolved, their complexity, dimensionality, and quality have all increased simultaneously, with a particular emphasis on quality and minimal attention to dimensionality. Typically generated embeddings are fixed in size for all tasks and are proportional to the model size, rendering their use in low resource
settings a challenge. Accordingly, in this paper, we investigate the adaptation of DWT from Signal and Image processing to the field of NLP for the analysis and compression of word and sentence embeddings. DWT analyzes and reduces the size of the data by identifying redundant and important information in data. We posit that DWT has the potential to generate compressed embeddings capable of retaining contextual information and semantic relationships between words as well as among sentences. Our key contributions: 1) Introduce DWT as an effective compression method for compressing embeddings with respect to an underlying task; 2) Propose a novel approach leveraging DWT for analyzing semantics in word and sentence embeddings; 3) Study the efficacy of DWT embeddings in capturing and retaining semantics by applying them to similarity and downstream tasks using various embeddings.

heading: 2 Motivation
text: The use of spectral analysis methods in Image and Signal processing is driven by their ability to analyze data in the frequency domain, providing insights and revealing hidden patterns and dynamics not detectable in the spatial domain. Methods such as the Discrete Cosine Transform (DCT) and Fourier Transform (FT) have been commonly used. However, these methods offer global frequency representations that ignore the location of frequencies in the original domain and lack the abil-
15963
ity to perform multi-resolution analysis. DWT effectively addresses these limitations by examining data change over time or position, i.e. frequency and time localization, and consequently captures varied information across different scales (multiresolution analysis). As a result, DWT achieved superior results in other fields. DWT analyzes data based on their variation, filtering low-varying or highly correlated (low-frequency) and highvarying (high-frequency) components while preserving their spatial domain information. Figure1 visualizes the result of applying DWT to an image resulting in low-frequency coefficients (approximations) that capture the overall structure of the data, hence used for compression (Lewis and Knowles, 1992; Grgic et al., 2001), while high-frequency coefficients (details) capture abrupt changes and motifs, which provide information about edges and contours, making them useful for edge detection (Xizhi, 2008; Zhang et al., 2009) and noise filtering (Dautov and Özerdem, 2018). DWT can be further applied, recursively, to any coefficients of the first level to achieve a second level of the transform that contains more approximation and details coefficients. In the realm of NLP, visualizing a word embedding vector, as shown in Figure2, we observe a signal-like structure with a few spikes indicating high variation in feature values, alongside features with minimal variation. This pattern suggests that applying spectral analysis to these vectors and numerical representations can reveal additional patterns or insights that may not be immediately apparent. Few attempts for applying spectral methods to NLP including DCT (Almarwani et al., 2019) and Higher-order Dynamic Mode Decomposition (HODMD) (Kayal and Tsatsaronis, 2019) were found successful. However, we are unaware of any application of applying DWT to word and sentence embedding other than our preliminary application of DWT to enhance DCT sentence embedding in (Salama et al., 2024). In this work, we posit that DWT can effectively compress embed-
ding representations, and analyze them at different scales to understand and capture different linguistic patterns encoded in these embeddings. This analysis allows DWT to concentrate embedding energy with high compression ratios without significant loss of information. DWT allows for more efficient representation, less storage requirements and reduced computational complexity. Additionally, DWTs are scalable and can be applied to any embedding model and for any task. Accordingly, we believe that DWT, as a spectral transform, holds promise for NLP tasks.

heading: 3 Related Work

heading: 3.1 Embedding Compression
text: Several studies have tackled embedding compression methods including code-book (Shu and Nakayama, 2017)(Kim et al., 2020), quantization (Ling et al., 2016)(Shi and Yu, 2018)(Tao et al., 2022) and factorization(Acharya et al., 2018). Other methods applied compression methods based on knowledge distillation (Gao et al., 2023). Some other models consider compressing model parameters (Mao et al., 2020), token embedding matrix (Bałazy et al., 2021), or prune model weights (Li et al., 2017). Yet all these techniques compress only word representations, regardless of the semantics they convey. Additionally, in the era of large language models (LLM), recent research (Wang et al., 2023) argues that the dimensionality of the embedding representations, specifically in sentence embeddings, are sub-optimal and the same encoded information may be represented in smaller dimensions while achieving comparable performance.

heading: 3.2 Spectral Methods
text: The success of spectral methods in studying and analyzing embeddings has recently become evident in NLP. The application of DCT in sentence embedding has shown promising results(Almarwani et al., 2019), yet it has only been applied to noncontextualized embeddings to generate sentence embeddings, not for compression. These DCT embeddings resulted in long sentence representations, corresponding to the number of coefficients considered from the transformation. We proposed using DWT to address this issue in (Salama et al., 2024), and results were promising. Similarly with HODMD (Kayal and Tsatsaronis, 2019), a sentence is represented as a signal with transitional properties captured in the frequency domain using uncorrelated coefficients to encode a sentence. Such models capture structural varia-
tion without losing on efficiency (comparable to averaging) (Zhu and de Melo, 2020) and yet outperform more complex sentence embedding models (Mikolov et al., 2018). However, these models tend to analyze frequencies along similar word embedding dimensions, on a vertical level (inter-word embedding, aka across all words), accumulating a limited number of base frequency coefficients and dropping the rest, in addition to ignoring their spatial domain position, i.e., ignoring intra-word frequencies within individual word embeddings.

heading: 4 Discrete Wavelet Transforms
text: DWTs are mathematical functions that analyze data, f(t), using a window (a function) known as the Mother Wavelet (MW) ψ(t), also called wavelets. This window moves over the data sequentially to examine the variations in the data with respect to this window (localized in time). Within a given analysis window if surrounding data features exhibit minimal variation, they can be grouped together and represented using Approximation coefficients (cA). Conversely, if certain features vary significantly, they are represented using Detail coefficient (cD). To shift over the data, a MW translates and dilates ψa,b(t) (as in equation (1)) using a shifting factor, b, and scales with a scaling factor, a, to capture different frequency variations at different data segments in time t.
ψa,b(t) = 1√ a ψ( t− b a ) (1)
The resulting coefficients are equivalent to a pair of sub-band linear (convolutional) filters (Vetterli and Kovacevic, 1996): one low-pass and one high-pass. The output of a filter pair is usually down-sampled by 2 so the combined output is of the same size (dimensionality) as the original input. This filtering + downsampling can be cascaded on multiple levels recursively to analyze data at multiple levels of resolution. Note that DWT can be compared to Convolutional Neural Networks (CNNs) (Gu et al., 2015) in that they both use sliding-window filters and downsampling. The difference is that in CNNs, the filters are learned from the training data, while in DWT the filters are designed, not learned. There are many families of MWs: Haar, Symmlets, Coiflets, and Daubechies, to name a few (Madhavan, 2003). As a proof of concept in this paper, and in the interest of space, we will only be using and reporting on a subset of the MWs that yield the best results in our experiments. 1
1For a more detailed explanation of Wavelet Transform theory, refer to (Daubechies, 1992; Madhavan, 2003; Brunton

heading: 5 Method
text: DWT filters features in embeddings by capturing how consecutive features in an embedding vector change, where small numerical difference or variation are converted into approximation coefficients. While features with large variations are converted into detail coefficients, and hence reflecting the structure and behavior of the underlying information encoded in the vector. As shown in Figure3, given a word or a sentence embedding Ed of a d-dimensional vector, we transform Ed by applying DWT, DWT (E), which decomposes the embedding vector into two vectors of low-varying coefficients, cA, and high varying coefficients, cD, for one level of transformation. The dimension for each set of coefficients, d′, is reduced by 2, d′ = d/2. The generated coefficients, cA and cD, can be further transformed for a second level, generating new vectors of approximation and detail coefficients, and downsampled by 2 again. In fact, this recursive process can be repeated many times, say L times. L represents the number of DWT levels, and is determined based on the required compression ratio and performance. For the selection of the MW used in the transformation, we primarily use Symlets, Daubechies, and Coiflets wavelets across all experiments and the best performance per task is recorded. 2 Coefficients Selection: In image processing, different sets of coefficients capture different aspects of an image. Similarly, we use different sets of coefficients as compact representations of the base embedding. We analyze the amount of information each set of coefficient encodes by studying their effectiveness in similarity and downstream tasks. We consider the following possible selection mechanisms: (a) Level-1 Coefficients: Employ Level-1 coefficients, either the approximation (cA) or detail (cD) coefficients as the compressed representation with a size downsampled by 2. (b) Higher-Level Coefficients: Employ coefficients from higher levels of the DWT (i.e., Levels 2, 3 or later). Specifically, we examine Level-2 approximation coefficients derived from Level-1 approximation and detail coefficients, namely cAA (approximation of approximation) and cDA (approximation of details) which are a quarter the size of the original
and Kutz, 2019). 2In the interest of simplicity and clarity, we will omit the details of the specific MW used in each experiment, as this paper is intended as a proof of concept,
and not for comparing different MWs.
embedding vector. Additionally, for a higher compression ratio, we explore approximation coefficients from subsequent Level-3 (cAAA) and Level4 (cAAAA), Which are 1/8 and 1/16 the size of the original embedding vector, respectively. Our emphasis lies on approximation for deeper compression, as they encapsulate an abstracted representation of the original embedding. (c) Combined Coefficients: In experiments where Level-1 coefficients do not achieve comparable performance, we incorporate coefficients from higher levels. Specifically, if Level-1 approximation doesn’t sufficiently encode all relevant semantics, we further combine it with Level-2 approximation (derived from Level-1 detail), referred to as cDA. Similarly, for detail, we combine it with Level-2 detail (derived from Level1 approximation), referred to as cAD. As illustrated in Figure 3, we enrich the red-colored coefficients from Level-1 (approximation) with the red-colored coefficients from Level-2 (obtained from Level-1 green coefficients). Thus, the combined embeddings are cA+cDA and similarly cD+cAD. Note that combining Level-2 red-colored coefficients, cAA, with cA (Level-1 red-colored) is irrelevant since they contain redundant information.

heading: 6 Evaluation
text: To evaluate the proposed approach, we first investigate its efficacy in capturing and compressing semantics in semantic similarity tasks. We further evaluate their effectiveness in a number of downstream tasks to evaluate their efficacy extrinsically. We consider different embeddings, baselines, experimental setup and tasks as follows. Embeddings: We experiment with different embeddings; Pre-trained Language Models as BERT (Devlin et al., 2018), GPT (Radford and Narasimhan, 2018), SBERT (Wang and Kuo, 2020) and RoBERTa (Liu et al., 2019) sentence embed-
dings. We also include non-contextualized embeddings from our previous work (Salama et al., 2024) for completeness. We use GloVe (Pennington et al., 2014) and FastText (Mikolov et al., 2018) with various dimensions (50, 100, 200, 300). Baselines: For every experiment, we use different baselines, in addition to the base embeddings, to explore the capabilities of DWT. Our baselines include: (1) Base embeddings (in all experiments); (2) Other dimensionality reduction methods to assess their effectiveness in comparison to DWT: PCA for dimensionality reduction(Shlens, 2014) and DCT for compression(Gupta and Garg, 2012) in some of our experiments. Experimental Setup: For all sentence embeddings experiments, we use the SentEval toolkit (Conneau and Kiela, 2018) for evaluation. For all down-
stream tasks, we leverage multi-layer perceptron (MLP) classifiers based on the default setup outlined in SentEval.3 6.1 Semantic Similarity Evaluation Word and sentence similarity tasks have become the de-facto method for semantic evaluation (Wang et al., 2022). Semantic Similarity involves measuring the degree of relatedness and similarity between pairs of words or sentences compared against human judgments or similarity scores assigned by human annotators.

heading: 6.1.1 Word Similarity Evaluation
text: We start by evaluating DWT embeddings for word similarity, using the following datasets: SimLex999 (Hill et al., 2014), MEN (Bruni et al., 2014) and WS353 (Finkelstein et al., 2001). In our initial experiment, we utilize DWT Level-1 approximation (cA) and detail (cD) coefficients as DWT embeddings for the word semantic similarity task. In this experiment we assess: (1) GloVe embeddings with dimensions 100 and 200. Baselines are: the base GloVe embeddings of the same size, alongside the base GloVe embeddings originally reduced in dimensions by 50%. (2) FastText embeddings with dimension 300. Baselines are: the base FastText embeddings, and PCA reduced embedding with size 150 dimensions. (3) BERT and GPT models, where for each model we consider two variants, base and large models, in order to conduct a thorough evaluation with original embeddings as baselines. We found DWT embeddings to consistently outperform the baselines for GloVe embeddings, with dimensions 100 and 200, as depicted in Table 1. DWT embeddings efficiently compress the semantics encoded
3The source code is publicly available on GitHub at https://github.com/engranas/DWT-Semantic-Compression
in the original embeddings, surpassing both baselines with a 50% reduction in dimensionality and in some cases surpassing the performance of embeddings that are four times larger as in SimLex and WS353, where DWT embeddings of size 50 outperforms GloVe embeddings of size 200. This empirically shows that DWT embeddings not only serve as a dimensionality reduction technique but also adeptly capture the semantics encoded in an embedding vector and effectively compress them in DWT coefficients. Additionally, DWT embeddings significantly outperform PCA-reduced embeddings, demonstrating superior performance with >20% improvement. Additionally, in the case of contextualized embeddings, DWT embeddings exhibit comparable performance to all baselines for all models. This suggests that the large dimensionality used for these models may not be significant for encoding words. It can also be noted that the DWT BERT embeddings are more comparable to the baseline than DWT GPT embeddings; this suggests that GPT models are packing more information and semantics than the BERT model for these datasets. This empirically shows that DWT reveals new aspects of embeddings that were not evident in the original domain. Nevertheless, for FastText, while DWT yields similar performance to the original embeddings for the SimLex dataset, yet for WS353 and MEN datasets Level-1 DWT embeddings failed to fully capture the encoded semantics present in the original embeddings. This discrepancy suggests that FastText embeddings contain richer semantics compared to GloVe embeddings, a conclusion supported by the similarity results achieved with the original embeddings. We further use the combined coefficients; cA+cDA and cA+cDA. As shown in Table 1, the combined DWT embeddings at a compression of 25% dimensionality augmenting the embeddings with additional semantics, resulting in a performance comparable to the baseline with a slight reduction approx. 1% in performance for WS353 and MEN datasets. Conversely, for SimLex datasets, the addition of coefficients from subsequent layers (cD+cAD) did not improve the performance and cD coefficients seems to have effectively encoded the semantics for all word embeddings. Nevertheless, it is crucial to emphasize that the selection of DWT coefficients depends not only on the type of embeddings but also on the particular task in consideration. As a result, we utilize DWT embeddings generated from
FastText for another semantic task, Concept Categorization, to further elaborate on the adaptability of DWT for different tasks. Concept Categorization groups words in different categories based on semantic clusters (Baroni et al., 2014). For this evaluation we use the datasets: AP (Almuhareb, 2006), BM (Murphy et al., 2012) and BLESS datasets (Baroni and Lenci, 2011). We use the base embeddings as the baseline. As illustrated in Table 4, cA and cD embeddings demonstrate comparable or even superior results compared to the baseline, despite 50% dimensionality reduction. This illustrates the effectiveness of DWT embeddings in encapsulating essential semantics from the original embedding for the given task. To conduct a qualitative analysis of DWT embeddings, Table 3 displays the five nearest neighbors (determined by cosine similarity) for randomly chosen words utilizing BERT word embeddings. As shown, approximation coefficients capture more relevant words like ’excited’ and ’glad’ for the word ’happy’. Also, ’island’ for ’sea’, and ’writing’ and ’coloring’ for the word ’playing’. Detail coefficients capture more appropriate words such as ’joy’ for ’happy’, ’beach’ and ’fish’ for ’sea’, and ’fighting’ for ’playing’. Additionally, it’s apparent that certain relevant words, which share similar meanings or contexts, have closer similarity, such as "creating" for "playing" and "smiling" for "happy".

heading: 6.1.2 Sentence Semantic Similarity
text: The Semantic Textual Similarity (STS) Task is a common benchmark used for evaluating the performance of semantic models. In this setting, we examine the application of DWT to contextualized sentence embeddings on STS tasks 2012 and 2016 (Agirre et al., 2012, 2016), STS benchmark (STSB) (Cer et al., 2017) and SICKRelatedness (Marelli et al., 2014). (See Appendix for more STS tasks results.) We evaluate DWT embeddings on contextualized models as they are
becoming dominant for sentence embedding (Wang et al., 2024). We consider SBERT base and large models (Wang and Kuo, 2020), RoBERTa base and large models (Liu et al., 2019).4 In this experiment, we will further explore the effectiveness of DWT in capturing relevant features and semantics within pretrained language models, illustrating that the extended size of dimensionality of these embeddings is not optimal for semantic representation, which we found consistent with the recent results and findings in (Wang et al., 2023) without the need for any further training or fine-tuning of the model. We consider Level-1 coefficients, cA and cD, with 50% reduction in dimensions. We also consider Level-2 coefficients, cAA and cDA, with 75% reduction in dimension. For the baselines we consider: (1) The base embeddings, (2) DCT (Gupta and Garg, 2012) as another spectral model used for lossy compression. We use DCT to compress the original SBERT and RoBERTa embeddings by applying the DCT transform to these vectors and selecting the first n coefficients, where n is equivalent to 50% or 25% of the original embedding size.5 As shown in Table 5, DWT consistently surpasses DCT across all tasks, particularly in STSB and SICKR, where the efficacy of DCT embeddings reflects significantly lower performance. Although DCT was capable of compressing embeddings and maintaining comparable performance in STS12 and STS16, DCT compresses by discarding residual frequency coefficients leading to the loss of important details and a subsequent decline in performance in STSB and SICKR. DWT performs similarly to the base embeddings, with at most a 0.5% decrease in performance on a few tasks for all models using Level-1 coefficients in SBERT and RoBERTa. This indicates that Level-1 coefficients effectively capture relevant semantics, condensing them into fewer dimensions. Notably, for SBERT, the approximation coefficients yield better performance, while for RoBERTa, the detail coefficients perform
4Models available in https://huggingface.co/sentencetransformers; SBERT-base-nli-v2, SBERT-Large-nli-v2, nliroberta-base and nli-roberta-large.
5In this context, DCT is not applied in the same way as proposed by (Almarwani et al., 2019), where the transform is applied across all words in a sentence to encode the sentence. In our context we apply it within a word vector.
better. This empirically demonstrates that SBERT and RoBERTa capture different types of features: SBERT captures more relational semantic features, whereas RoBERTa focuses on distinct semantic features. Additionally, DWT reveals some characteristics of these embeddings: large models tend to contain more redundant features. As a result, DWT can effectively compress them to 75% less dimensions while maintaining comparable performance across all tasks. More interestingly, for the STSB task using SBERT and RoBERTa, DWT outperforms the baseline in a manner proportional to the model size. Specifically, in the large models, cAA coefficients improve performance by 4% in SBERT and by 2.5% in RoBERTa. This shows that the cAA coefficients contain more dense semantic information suggesting that the original embeddings had highly correlated dimensions with redundant features that proved to be irrelevant when filtered out using DWT. Generally, DWT embeddings prove to have comparable performance or better in most tasks, at 50% reduction in embedding size. At a 75% reduction in size, the performance is still comparable and never drops more than 2%, and for some tasks the performance is significantly better. In most cases cA represents a good approximation for the original representation.

heading: 6.2 Downstream Tasks
text: To further evaluate our model extrinsically, we explore applying DWT embeddings to the following downstream tasks: sentiment classification on Movie Reviews (MR), Stanford Sentiment Treebank (SST2, SST5) (Pang and Lee, 2004a), product review (CR) (Hu and Liu, 2004), subjectivity classification (SUBJ) (Pang and Lee, 2004b), opinion polarity classification (MPQA), question type classification (TREC) (Voorhees and Tice, 2000), paraphrase identification (MRPC) (Dolan et al., 2004),
and entailment classification on the SICK dataset (SICK-E) (Bouden and Nibouche, 2012). Due space limitation, we will only consider RoBERTa base and large embeddings for these experiments as a proof of concept, to show the efficacy of DWT. Table 6 shows the results with base embeddings as the baseline. As shown, DWT effectively compresses the base embeddings, outperforming the baselines in all tasks. Level-1 coefficients, cD, achieve better performance by 4.5% in SST5 task, 2% in TREC task, while cA outperforms in MR and SICK-E. Using Level-2 coefficients, cAA, further improves over the baseline in MPQA and MRPC, while cDA outperforms in SST2. Generally, DWT embedding surpassed the baselines with all coefficients for all tasks except for CR and TREC where the Level-2 coefficients is comparable. This demonstrates the efficacy of DWT embeddings for downstream tasks. In conclusion, we find that DWT presents an effective balance between efficiency (compactness) and accuracy, serving as an efficient data-size reduction method that condenses embeddings with relevant features, leading to enhanced performance. Multiple Levels of DWT We further investigate the effectiveness of applying multiple levels of DWT transformation to RoBERTa large embeddings. We extend our analysis up to 4 levels of transformation, thereby achieving compression of up to 93%. We apply DWT recursively to the approximation coefficients from each level, resulting in coefficients of sizes 512, 256, 128, and 64 respectively. The results6in Figure 4 show that for dimensions of size 128 and 64, corresponding to Level-3 and Level-4 coefficients, with a reduction in dimensionality of 87%-94%, respectively. DWT embeddings outperform the baseline for all tasks by
6Detailed results for this experiment can be found in the Appendix.
0.09-1.5% except for SICK-E, SUBJ, and TREC, which experience degradation by 1.3%, 1%, and 7.4%, respectively, although still outperforming the baseline in higher levels. This decline in performance tends to correlate with an increased compression ratio indicating that the minimized dimension size is not sufficient to fully encode the necessary semantic information, resulting in a tolerable performance decline. On the other hand, other tasks like MRPC outperform the baseline by 1.5% with just 64 dimensions, indicating that DWT effectively consolidates relevant features into much fewer dimensions, while the extended dimension size contains redundant features. These findings align with those of (Wang et al., 2023), where their model, despite being trained using a two-step method, consistently degrades performance for all tasks by 2%- 9% in dimension sizes of 64. In contrast, our model exhibits comparability to the baseline in most tasks and experiences degradation by 2%-3.5% in MR, SICK-E, and SUBJ, and by 7.4% in TREC.

heading: 7 Conclusion and Discussion
text: In this paper, we explored the effectiveness of applying DWT to word and sentence embeddings to selectively reduce embeddings. Our experiments
illustrate the potential of DWT to enhance space and computational efficiency without decline in performance, reducing embedding size by 50-75%. DWT reveals new insights about embeddings, and exposes hidden patterns and semantic information that were not apparent in the base embedding space. Additionally, the generated DWT embeddings postulate that different sets of coefficients capture different semantic aspects of an embedding. We conclude our study outcomes in the following points: Correlation and Redundancy: While it has not been proven that features within the same embedding are correlated, the comparable performance achieved by DWT suggests a correlation between the dimensions of an embedding. This challenges the previous assumption that these dimensions are uncorrelated. If no correlation existed, DWT would hardly achieve comparable results to the performance of base embeddings. Our results also prove that embeddings from complex models, such as Pre-trained Language Models, contain redundant features and hence compressing them with 75% reduction in dimensionality achieves a comparable performance to the base embeddings with no more than 2% degradation. Dimensionality Reduction Techniques: Although the general idea of DWT appears to be similar to other dimensionality reduction methods in the context of decomposition and compression, its overall properties and methodology are different (Raunak et al., 2019). DWT outperforms other compression techniques such as PCA and DCT. DWT compresses the data using localized sub-band frequency components capturing both low and high features-variation. DWT stands out as a method that can be applied universally to diverse datasets and models. PCA reduces data to a low dimension space, and de-correlates the data, but is much slower because it needs to compute the eigenvec-
tors of the original data (Priya, 2014). As for DCT, despite being a frequency analysis method, it tend to capture important frequencies indiscriminately, lacking localization and discarding other frequencies and accordingly degrading performance. Mother Wavelets: MW is a component of DWT that controls feature filtering, with each MW corresponding to a unique pair of filters. We did not explore specific MW applications for each task due to space constraints, but limited ourselves to a predetermined number of MW families such that we maintain experimental consistency while empirically investigating the effectiveness of DWT. Our experiments indicate that Coiflets wavelets generally perform well across tasks. It is important to note that MW filtering is controlled by a scaling factor, which adjusts the level of variation considered in an embedding. For example, a Coiflet MW with scale 4 focuses on nearby features and their fine variations, whereas scale 17 emphasizes broader approximations, omitting fine details. Since the choice of the best MW varies from base embedding to base embedding, the selection and optimization of MW requires further study beyond this paper’s scope. As a result, we leave further exploration of MW details and selection for future research. Coefficients Selection: Much as in image processing, different coefficients tend to capture distinct aspects of the data. Therefore, it is essential to select the set of coefficients that are most suitable for the base embedding being used in a given task. Some tasks may benefit more from approximations over details, while others may see improved performance with nuanced information, and yet others may use both equally. However, it can be concluded that if minimizing the embedding size is a significant consideration, utilizing only Level1 coefficients (cA or cD) with a 50% reduction in size results in comparable performance to the baselines across nearly all tasks. Moreover, approximation coefficients consistently maintain comparable performance across multiple levels of DWT, as demonstrated in the 4-level analysis, with the exception of TREC. This consistency suggests that these coefficients retain relevant information about the original embeddings. Nevertheless, a detailed study and analysis of coefficient selection fall beyond the scope of this paper and are planned for future work. DWT for Compression: DWT offers a powerful method for compressing embeddings by exploit-
ing correlations and reducing redundancy. The performance of DWT in compressing embeddings strongly suggests that approximation coefficients can be effectively utilized for compression, akin to their role in image compression, where they typically capture essential features of the data. In the compression process, approximation coefficients not only reduce the embedding dimensionality but also reduce noise while retaining relevant features. By reducing noise, the performance of the model improves as it casts down on the error rate thus improving the accuracy of these results. Selecting the appropriate compression level for a given task is a hyperparameter that depends on various factors, including space complexity, resources allocated for a particular task and the trade-off between effectiveness and compression ratio. Computational Complexity: The DWT transformation is implemented through convolution and down-sampling (filtering) operations, which are typically linear with respect to the size of the input data. The overall computational complexity is often expressed in terms of the dimension of the embedding, d, so for a single level of transformation, the complexity is O(d). However, the recursive nature of multi-level transformations can lead to increased computations, depending on the number of levels, L, resulting in an overall complexity of O(Ld). Yet, DWT facilitates efficient dimensionality reduction, significantly reducing memory usage and increasing throughput by shrinking the size of embeddings while preserving essential features. DWT Efficacy: Based on our evaluations, the application of DWT in the context of NLP exhibits significant potential for efficiently modeling word and sentence embeddings. Our findings demonstrate that DWT coefficients have the ability to capture various aspects of the data, with the approximation coefficients serving as a general approximation of an embedding, akin to their behavior in image analysis. Furthermore, the details coefficients excel at capturing semantic nuances within the embeddings. Notably, DWT embeddings reveal new aspects and characteristics that were previously unexplored, such as establishing correlations between dimensions of a word embedding. Additionally, DWT can adapt to any embedding: words, sentences or documents of any length in the same manner as mentioned in this paper. Overall, the application of DWT holds promise, and we anticipate its effectiveness in other NLP applications.

heading: 8 Limitations
text: In this paper, our focus is to thoroughly and empirically investigate the effectiveness of applying Discrete Wavelet Transform (DWT) to word and sentence embeddings, with a primary emphasis on analysis and compression. However, DWT transformations are mainly based on the selection of Mother Wavelets (MWs), which we did not specify for every experiment due to space constraints. We employed Coiflets, Symlets, Haar, and Daubechies as MWs, with the recorded best overall results. Choosing the optimal MW for DWT is a common research question in Image and Signal processing, and we defer the investigation into the selection of the best MW to future work. The results presented in this paper serve as a proof of concept, indicating the potential for other MWs to further enhance performance. Additionally, our results were generated using publicly available models that undergo regular updates; thus, discrepancies may exist between our results and those published by the model owners.

heading: 9 Ethical Considerations
text: As we propose a novel method for applying DWT to NLP embedding, this section is divided into the following two parts.

heading: 9.1 Dataset
text: Intellectual Properties and Privacy Rights We make use of publicly-available data for all experiments with no modification or update for fair comparison.

heading: 9.2 NLP Application
text: Code Availability When used as intended, applying the models described in this paper can save people much time. Our code will be publically available to ensure reproduciblity of results
Code Reusability To run our experiments, we sometimes used public Github repositories as intended by their authors, without any modification. All such code was appropriately referenced.
Environmental Cost The experiments described in the paper makes no use of GPUs. We used CPU for our experiments. The experiments run in several hours. Several dozen experiments were run due to parameter search, and future work should experiment with distilled models for more light-weight training. We note that while our work required
extensive experiments to draw sound conclusions, future work will be able to draw on these insights and need not run as many large-scale comparisons. Models in production may be trained once for use using the most promising settings.

heading: A Appendix
text: A.1 Sentence Semantic Similarity
Visualizing Embeddings of Words in a Sentence By sketching out all the words in a sentence like "it’s a hot and sunny day," as shown in Figure 5, we observe that the energy of all word embeddings overlaps within an average sub-band. This provides a compelling explanation for why word averaging effectively represents a sentence, offering empirical support for averaging as a method for sentence embedding. This suggests that further spectral analysis of these embedding representations is promising and likely to achieve effective results, similar to those in image and signal processing.
Qualitative Analysis To further explore the efficacy of DWT through Sentence Similarity Tasks, we initially demonstrates the word similarity matrix between two randomly selected sentences from the STSB dataset using BERT(Devlin et al., 2018) embeddings with a dimension size of 768 as opposed to their Level-1 DWT cA and cD coefficients, with a dimension size down-sampled by 2, i.e. dimension size is 384. By comparing the overall distribution of similarity shown in Figure 6, we observe that it remains largely coherent between the original embeddings and the DWT embeddings with 50% reduction in dimensions.
Extended Intrinsic Evaluation:
1- InferSent Embeddings We additionally consider InferSent sentence embedding model(Conneau et al., 2017) for this experiment as an example for a parameterized sentence embedding model. We set the original embeddings as the baseline. As shown in Table7, Level-1 cD
coefficients outperform the baselines for all tasks by 1.5-3% better performance except for STSB ad SICKR which have comparable results. Level-2 cDA coefficients exceed the performance in STS12,STS15 and STS16 at a dimension reduction by 75% showing (1) that DWT condensed more relevant semantics for these task in subsequent levels of transformation and (2) that the nuance in the features represented in the original embeddings were more relevant for these tasks in the InferSent representation context, having cD outperforms cA coefficients. Still, cA coefficients also beat the baseline.
2- Semantic Similarity Tasks (2013-2015) Subsequently, we present the results for applying DWT on STS tasks 2013-2015 (Agirre et al., 2013, 2014, 2015) using Pre-trained Language Model embeddings, SBERT and RoBERTa, for a detailed study for the performance of DWT on more semantic similarity tasks. Table 8 shows the result of applying DWT embeddings using SBERT and RoBERTa models. As shown, DWT outperforms the baselines for STS13 and is very comparable to STS14 and STS15.
A.2 4-Level DWT Embedding Results In this section we represent the detailed results for Figure4 for 4-Levels of DWT tasks applied to downstream tasks included in Section6.2 using RoBERTa Large embeddings. Table9 shows the results for Level-1 approximation coefficients, cA, Level-2 approximation coefficients cAA, Level-3 approximation coefficients, cAAA and Level-4 coefficients, cAAAA. The results show the efficacy of DWT approximation coefficients to maintain relevant information despite decreasing the size of the embedding by more than 90%.

heading: Future Work
text: Several dozen experiments were run due to parameter search, and future work should experiment with distilled models for more light-weight training