HyperEmbed:  Tradeoffs Between Resources and Performance in NLP Tasks with Hyperdimensional Computing enabled embedding of n-gram statistics 

1 INTRODUCTION . Recent work ( Strubell et al. , 2019 ) has brought significant attention by demonstrating potential cost and environmental impact of developing and training state-of-the-art models for Natural Language Processing ( NLP ) tasks . The work suggested several countermeasures for changing the situation . One of them recommends a concerted effort by industry and academia to promote research of more computationally efficient algorithms . The main focus of this paper falls precisely in this domain . In particular , we consider NLP systems using a well-known technique called n-gram statistics . The key idea is that hyperdimensional computing ( Kanerva , 2009 ) allows forming distributed representations of the conventional n-gram statistics ( Joshi et al. , 2016 ) . The use of these distributed representations , in turn , allows trading-off the performance of an NLP system ( e.g. , F1 score ) and its computational resources ( i.e. , time and memory ) . The main contribution of this paper is the systematic study of these tradeoffs on nine machine learning algorithms using several benchmark classification datasets . We demonstrate the usefulness of hyperdimensional computing-based embedding , which is highly time and memory efficient . Our experiments on a well-known dataset ( Braun et al. , 2017 ) for intent classification show that it is possible to reduce memory usage by ∼ 10x and speed-up training by ∼ 5x without compromising the F1 score . Several important use-cases are motivating the efforts towards trading-off the performance of a system against computational resources required to achieve that performance : high-throughput systems with an extremely large number of requests/transactions ( the power of one per cent ) ; resource-constrained systems where computational resources and energy are scarce ( edge computing ) ; green computing systems taking into account the aspects of environmental sustainability when considering the efficiency of algorithms ( AI HLEG , 2019 ) . The paper is structured as follows . Section 2 covers the related work . Section 3 outlines the evaluation and describes the datasets . The methods being used are presented in Section 4 . Section 5 evaluates of the experimental results . Discussion and concluding remarks are presented in Section 6 . 2 RELATED WORK . Commonly , data for NLP tasks are represented in the form of vectors , which are then used as an input to machine learning algorithms . These representations range from dense learnable vectors to extremely sparse non-learnable vectors . Well-known examples of such representations include onehot encodings , count-based vectors , and Term Frequency Inverse Document Frequency ( TF-IDF ) among others . Despite being very useful , non-learnable representations have their disadvantages such as resource inefficiency due to their sparsity and absence of contextual information ( except for TF-IDF ) . Learnable vector representations such as word embeddings ( e.g. , Word2Vec ( Mikolov et al. , 2013 ) or GloVe ( Pennington et al. , 2014 ) ) partially address these issues by obtaining dense vectors in an unsupervised learning fashion . These representations are based on the distributional hypothesis : words located nearby in a vector space should have similar contextual meaning . The idea has been further improved in Joulin et al . ( 2016 ) by representing words with character n-grams . Another efficient way of representing a word is the concept of Byte Pair Encoding , which has been introduced in Gage ( 1994 ) . The disadvantage of the learnable representations , however , is that they require pretraining involving large train corpus as well as have a large memory footprint ( in order of GB ) . As an alternative to word/character embedding , Shridhar et al . ( 2019 ) introduced the idea of Subword Semantic Hashing that uses a hashing method to represent subword tokens , thus , reducing the memory footprint ( in order of MB ) and removing the necessity of pretraining over a large corpus . The approach has demonstrated the state-of-the-art results on three datasets for intent classification . The Subword Semantic Hashing , however , relies on n-gram statistics for extracting the representation vector used as an input to classification algorithms . It is worth noting that the conventional n-gram statistics uses a positional representation where each position in the vector can be attributed to a particular n-gram . The disadvantage of the conventional n-gram statistics is that the size of the vector grows exponentially with n. Nevertheless , it is possible to untie the size of representation from n by using distributed representations ( Hinton et al. , 1986 ) , where the information is distributed across the vectors positions . In particular , Joshi et al . ( 2016 ) suggest how to embed conventional n-gram statistics into a high-dimensional vector ( HD vector ) using the principles of hyperdimensional computing . Hyperdimensional computing also known as Vector Symbolic Architectures ( Plate , 2003 ; Kanerva , 2009 ; Eliasmith , 2013 ) is a family of bio-inspired methods of manipulating and representing information . The method of embedding n-gram statistics into the distributed representation in the form of an HD vector has demonstrated promising results on the task of language identification while being hardware-friendly ( Rahimi et al. , 2016 ) . In Najafabadi et al . ( 2016 ) it was further applied to the classification of news articles into one of eight predefined categories . The method has also shown promising results ( Kleyko et al. , 2019 ) when using HD vectors for training Self-Organizing Maps ( Kohonen , 2001 ) . However there are no previous studies comprehensively exploring tradeoffs achievable with the method on benchmark NLP datasets when using the supervised classifiers . 3 EVALUATION OUTLINE . 3.1 CLASSIFIERS AND PERFORMANCE METRICS . To obtain the results applicable to a broad range of existing machine learning algorithms , we have performed experiments with several conventional classifiers . In particular , the following classifiers were studied : Ridge Classifier , k-Nearest Neighbors ( kNN ) , Multilayer Perceptron ( MLP ) , Passive Aggressive , Random Forest , Linear Support Vector Classifier ( SVC ) , Stochastic Gradient Descent ( SGD ) , Nearest Centroid , and Bernoulli Naive Bayes ( NB ) . All the classifiers are available in the scikit-learn library ( Pedregosa et al. , 2011 ) , which was used in the experiments . Since the main focus of this paper is the tradeoff between classification performance and computational resources , we have to define metrics for both aspects . The quality of the classification performance of a model will be measured by a simple and well-known metric – F1 score ( please see ( Fawcett , 2006 ) ) . The computational resources will be characterized by three metrics : the time it takes to train a model , the time it takes to test the trained model , and the memory , where the memory is defined as the sum of the size of input feature vectors for train and test splits as well as the size of the trained model . To avoid the dependencies such as particular specifications of a computer and dataset size , the train/test times and memory are reported as relative values ( i.e. , train/test speed-up and memory reduction ) , where the reference is the value obtained for the case of the conventional n-gram statistics.1 3.2 DATASETS . Four different datasets were used to obtain the empirical results reported in this paper : the Chatbot Corpus ( Chatbot ) , the Ask Ubuntu Corpus ( AskUbuntu ) , the Web Applications Corpus ( WebApplication ) , and the 20 News Groups Corpus ( 20NewsGroups ) . The first three are referred to as small datasets . The Chatbot dataset comprises questions posed to a Telegram chatbot . The chatbot , in turn , replied the questions of the public transport of Munich . The AskUbuntu and WebApplication datasets are questions and answers from the StackExchange . The 20NewsGroups dataset comprises news posts labelled into several categories . All datasets have predetermined train and test splits . The first three datasets ( Braun et al. , 2017 ) are available on GitHub.2 The Chatbot dataset consists of two intents : the ( Departure Time and Find Connection ) with 206 questions . The corpus has a total of five different entity types ( StationStart , StationDest , Criterion , Vehicle , Line ) , which were not used in our benchmarks , as the results were only for intent classification . The samples come in English . Despite this , the train station names are in German , which is evident from the text where the German letters appear ( ä , ö , ü , ß ) . The dataset has the following data sample distribution ( train/test ) : Departure Time ( 43/35 ) ; Find Connection ( 57/71 ) . The AskUbuntu dataset comprises five intents with the following data sample distribution ( train/test ) : Make Update ( 10/37 ) ; Setup Printer ( 10/13 ) ; Shutdown Computer ( 13/14 ) ; Software Recommendation ( 17/40 ) ; None ( 3/5 ) . Thus , it includes 162 samples in total . The samples were gathered directly from the AskUbuntu platform . Only questions with the highest scores and upvotes were considered . For the task of mapping the correct intent to the question , the Amazon Mechanical Turk was employed . Beyond the questions labelled with their intent , this dataset contains also some extra information such as author , page URL with the question , entities , answer , and the answer ’ s author . It is worth noting that none of these data was used in the experiments . The WebApplication dataset comprises 89 text samples of eight intents with the following distribution ( train/test ) : Change Password ( 2/6 ) ; Delete Account ( 7/10 ) ; Download Video ( 1/0 ) ; Export Data ( 2/3 ) ; Filter Spam ( 6/14 ) ; Find Alternative ( 7/16 ) ; Sync Accounts ( 3/6 ) ; None ( 2/4 ) . The 20NewsGroups dataset has been originally collected by Ken Lang . It comprises of 20 categories ( for details please see Table 7 in the Appendix ) . Each category has exactly 18 , 846 text samples . Moreover , the samples of each category are split neatly into the train ( 11 , 314 samples ) and test ( 7 , 532 samples ) sets . The dataset comes already prepackaged with the scikitlearn library for Python . 4 METHODS . 4.1 CONVENTIONAL n-GRAM STATISTICS An empty vector s stores n-gram statistics for an input textD . D consists of symbols from the alphabet of size a ; ith position in s keeps the counter of the corresponding n-gram Ai = 〈S1 , S2 , . . . , Sn , 〉 from the set A of all unique n-grams ; Sj corresponds to a symbol in jth position of Ai . The dimensionality of s equals the total number of n-grams in A and calculated as an . Usually , s is obtained via a single pass-through D using the overlapping sliding window of size n. The value of a position in s ( i.e. , counter ) corresponding to a n-gram observed in the current window is incremented by one . In other words , s summarizes how many times each n-gram in A was observed in D. 1 It is worth noting that the speed-ups reported in Section 5 do not include the time it takes to obtain the corresponding HD vectors . Please see the discussion of this issue in Section 6 . 2Under the Creative Commons CC BY-SA 3.0 license : https : //github.com/sebischair/ NLU-Evaluation-Corpora 4.2 WORD EMBEDDINGS WITH SUBWORD INFORMATION . Work by Bojanowski et al . ( 2017 ) demonstrated that words ’ representations can be formed via learning character n-grams , which are then summed up to represent words . This method ( FastText ) has an advantage over the conventional word embeddings since unseen words could be better approximated as it is highly likely that some of their n-gram subwords have already appeared in other words . Therefore , each word w is represented as a bag of its character n-gram . Special boundary symbols “ < ” and “ > ” are added at the beginning and the end of each word . The word w itself is added to the set of its n-grams , to learn a representation for each word along with character n-grams . Taking the word have and n = 3 as an example , ( have ) = [ < ha , hav , ave , ve > , have ] . Formally , for a given word w , Nw ⊂ { 1 , . . . , G } denotes the set of G n-grams appearing in w. Each n-gram g has an associated vector representation zg . Word w is represented as the sum of the vector representations of its n-grams . A scoring function g is defined for each word that is represented as a set of respective n-grams and the context word ( denoted as c ) , as : g ( w , c ) = ∑ g∈Nw z > g vc , where vc is the vector representation of the context word c. Practically , a word is represented by its index in the word dictionary and a set of n-grams it contains .

Summary: This paper introduces a technique to project n gram statistic vectors into a lower dimensional space in order to improve memory efficiency and lower training time. The paper is motivated by the important problem of trying to improve efficiency of existing language models which can be extremely resource intensive. The authors then compare the performance of n gram statistics with HD vectors on 4 datasets to demonstrate that embedding into HD vectors can preserve performance while reducing resource utilization.