HyperEmbed:  Tradeoffs Between Resources and Performance in NLP Tasks with Hyperdimensional Computing enabled embedding of n-gram statistics 

1 INTRODUCTION . Recent work ( Strubell et al. , 2019 ) has brought significant attention by demonstrating potential cost and environmental impact of developing and training state-of-the-art models for Natural Language Processing ( NLP ) tasks . The work suggested several countermeasures for changing the situation . One of them recommends a concerted effort by industry and academia to promote research of more computationally efficient algorithms . The main focus of this paper falls precisely in this domain . In particular , we consider NLP systems using a well-known technique called n-gram statistics . The key idea is that hyperdimensional computing ( Kanerva , 2009 ) allows forming distributed representations of the conventional n-gram statistics ( Joshi et al. , 2016 ) . The use of these distributed representations , in turn , allows trading-off the performance of an NLP system ( e.g. , F1 score ) and its computational resources ( i.e. , time and memory ) . The main contribution of this paper is the systematic study of these tradeoffs on nine machine learning algorithms using several benchmark classification datasets . We demonstrate the usefulness of hyperdimensional computing-based embedding , which is highly time and memory efficient . Our experiments on a well-known dataset ( Braun et al. , 2017 ) for intent classification show that it is possible to reduce memory usage by ∼ 10x and speed-up training by ∼ 5x without compromising the F1 score . Several important use-cases are motivating the efforts towards trading-off the performance of a system against computational resources required to achieve that performance : high-throughput systems with an extremely large number of requests/transactions ( the power of one per cent ) ; resource-constrained systems where computational resources and energy are scarce ( edge computing ) ; green computing systems taking into account the aspects of environmental sustainability when considering the efficiency of algorithms ( AI HLEG , 2019 ) . The paper is structured as follows . Section 2 covers the related work . Section 3 outlines the evaluation and describes the datasets . The methods being used are presented in Section 4 . Section 5 evaluates of the experimental results . Discussion and concluding remarks are presented in Section 6 . 2 RELATED WORK . Commonly , data for NLP tasks are represented in the form of vectors , which are then used as an input to machine learning algorithms . These representations range from dense learnable vectors to extremely sparse non-learnable vectors . Well-known examples of such representations include onehot encodings , count-based vectors , and Term Frequency Inverse Document Frequency ( TF-IDF ) among others . Despite being very useful , non-learnable representations have their disadvantages such as resource inefficiency due to their sparsity and absence of contextual information ( except for TF-IDF ) . Learnable vector representations such as word embeddings ( e.g. , Word2Vec ( Mikolov et al. , 2013 ) or GloVe ( Pennington et al. , 2014 ) ) partially address these issues by obtaining dense vectors in an unsupervised learning fashion . These representations are based on the distributional hypothesis : words located nearby in a vector space should have similar contextual meaning . The idea has been further improved in Joulin et al . ( 2016 ) by representing words with character n-grams . Another efficient way of representing a word is the concept of Byte Pair Encoding , which has been introduced in Gage ( 1994 ) . The disadvantage of the learnable representations , however , is that they require pretraining involving large train corpus as well as have a large memory footprint ( in order of GB ) . As an alternative to word/character embedding , Shridhar et al . ( 2019 ) introduced the idea of Subword Semantic Hashing that uses a hashing method to represent subword tokens , thus , reducing the memory footprint ( in order of MB ) and removing the necessity of pretraining over a large corpus . The approach has demonstrated the state-of-the-art results on three datasets for intent classification . The Subword Semantic Hashing , however , relies on n-gram statistics for extracting the representation vector used as an input to classification algorithms . It is worth noting that the conventional n-gram statistics uses a positional representation where each position in the vector can be attributed to a particular n-gram . The disadvantage of the conventional n-gram statistics is that the size of the vector grows exponentially with n. Nevertheless , it is possible to untie the size of representation from n by using distributed representations ( Hinton et al. , 1986 ) , where the information is distributed across the vectors positions . In particular , Joshi et al . ( 2016 ) suggest how to embed conventional n-gram statistics into a high-dimensional vector ( HD vector ) using the principles of hyperdimensional computing . Hyperdimensional computing also known as Vector Symbolic Architectures ( Plate , 2003 ; Kanerva , 2009 ; Eliasmith , 2013 ) is a family of bio-inspired methods of manipulating and representing information . The method of embedding n-gram statistics into the distributed representation in the form of an HD vector has demonstrated promising results on the task of language identification while being hardware-friendly ( Rahimi et al. , 2016 ) . In Najafabadi et al . ( 2016 ) it was further applied to the classification of news articles into one of eight predefined categories . The method has also shown promising results ( Kleyko et al. , 2019 ) when using HD vectors for training Self-Organizing Maps ( Kohonen , 2001 ) . However there are no previous studies comprehensively exploring tradeoffs achievable with the method on benchmark NLP datasets when using the supervised classifiers . 3 EVALUATION OUTLINE . 3.1 CLASSIFIERS AND PERFORMANCE METRICS . To obtain the results applicable to a broad range of existing machine learning algorithms , we have performed experiments with several conventional classifiers . In particular , the following classifiers were studied : Ridge Classifier , k-Nearest Neighbors ( kNN ) , Multilayer Perceptron ( MLP ) , Passive Aggressive , Random Forest , Linear Support Vector Classifier ( SVC ) , Stochastic Gradient Descent ( SGD ) , Nearest Centroid , and Bernoulli Naive Bayes ( NB ) . All the classifiers are available in the scikit-learn library ( Pedregosa et al. , 2011 ) , which was used in the experiments . Since the main focus of this paper is the tradeoff between classification performance and computational resources , we have to define metrics for both aspects . The quality of the classification performance of a model will be measured by a simple and well-known metric – F1 score ( please see ( Fawcett , 2006 ) ) . The computational resources will be characterized by three metrics : the time it takes to train a model , the time it takes to test the trained model , and the memory , where the memory is defined as the sum of the size of input feature vectors for train and test splits as well as the size of the trained model . To avoid the dependencies such as particular specifications of a computer and dataset size , the train/test times and memory are reported as relative values ( i.e. , train/test speed-up and memory reduction ) , where the reference is the value obtained for the case of the conventional n-gram statistics.1 3.2 DATASETS . Four different datasets were used to obtain the empirical results reported in this paper : the Chatbot Corpus ( Chatbot ) , the Ask Ubuntu Corpus ( AskUbuntu ) , the Web Applications Corpus ( WebApplication ) , and the 20 News Groups Corpus ( 20NewsGroups ) . The first three are referred to as small datasets . The Chatbot dataset comprises questions posed to a Telegram chatbot . The chatbot , in turn , replied the questions of the public transport of Munich . The AskUbuntu and WebApplication datasets are questions and answers from the StackExchange . The 20NewsGroups dataset comprises news posts labelled into several categories . All datasets have predetermined train and test splits . The first three datasets ( Braun et al. , 2017 ) are available on GitHub.2 The Chatbot dataset consists of two intents : the ( Departure Time and Find Connection ) with 206 questions . The corpus has a total of five different entity types ( StationStart , StationDest , Criterion , Vehicle , Line ) , which were not used in our benchmarks , as the results were only for intent classification . The samples come in English . Despite this , the train station names are in German , which is evident from the text where the German letters appear ( ä , ö , ü , ß ) . The dataset has the following data sample distribution ( train/test ) : Departure Time ( 43/35 ) ; Find Connection ( 57/71 ) . The AskUbuntu dataset comprises five intents with the following data sample distribution ( train/test ) : Make Update ( 10/37 ) ; Setup Printer ( 10/13 ) ; Shutdown Computer ( 13/14 ) ; Software Recommendation ( 17/40 ) ; None ( 3/5 ) . Thus , it includes 162 samples in total . The samples were gathered directly from the AskUbuntu platform . Only questions with the highest scores and upvotes were considered . For the task of mapping the correct intent to the question , the Amazon Mechanical Turk was employed . Beyond the questions labelled with their intent , this dataset contains also some extra information such as author , page URL with the question , entities , answer , and the answer ’ s author . It is worth noting that none of these data was used in the experiments . The WebApplication dataset comprises 89 text samples of eight intents with the following distribution ( train/test ) : Change Password ( 2/6 ) ; Delete Account ( 7/10 ) ; Download Video ( 1/0 ) ; Export Data ( 2/3 ) ; Filter Spam ( 6/14 ) ; Find Alternative ( 7/16 ) ; Sync Accounts ( 3/6 ) ; None ( 2/4 ) . The 20NewsGroups dataset has been originally collected by Ken Lang . It comprises of 20 categories ( for details please see Table 7 in the Appendix ) . Each category has exactly 18 , 846 text samples . Moreover , the samples of each category are split neatly into the train ( 11 , 314 samples ) and test ( 7 , 532 samples ) sets . The dataset comes already prepackaged with the scikitlearn library for Python . 4 METHODS . 4.1 CONVENTIONAL n-GRAM STATISTICS An empty vector s stores n-gram statistics for an input textD . D consists of symbols from the alphabet of size a ; ith position in s keeps the counter of the corresponding n-gram Ai = 〈S1 , S2 , . . . , Sn , 〉 from the set A of all unique n-grams ; Sj corresponds to a symbol in jth position of Ai . The dimensionality of s equals the total number of n-grams in A and calculated as an . Usually , s is obtained via a single pass-through D using the overlapping sliding window of size n. The value of a position in s ( i.e. , counter ) corresponding to a n-gram observed in the current window is incremented by one . In other words , s summarizes how many times each n-gram in A was observed in D. 1 It is worth noting that the speed-ups reported in Section 5 do not include the time it takes to obtain the corresponding HD vectors . Please see the discussion of this issue in Section 6 . 2Under the Creative Commons CC BY-SA 3.0 license : https : //github.com/sebischair/ NLU-Evaluation-Corpora 4.2 WORD EMBEDDINGS WITH SUBWORD INFORMATION . Work by Bojanowski et al . ( 2017 ) demonstrated that words ’ representations can be formed via learning character n-grams , which are then summed up to represent words . This method ( FastText ) has an advantage over the conventional word embeddings since unseen words could be better approximated as it is highly likely that some of their n-gram subwords have already appeared in other words . Therefore , each word w is represented as a bag of its character n-gram . Special boundary symbols “ < ” and “ > ” are added at the beginning and the end of each word . The word w itself is added to the set of its n-grams , to learn a representation for each word along with character n-grams . Taking the word have and n = 3 as an example , ( have ) = [ < ha , hav , ave , ve > , have ] . Formally , for a given word w , Nw ⊂ { 1 , . . . , G } denotes the set of G n-grams appearing in w. Each n-gram g has an associated vector representation zg . Word w is represented as the sum of the vector representations of its n-grams . A scoring function g is defined for each word that is represented as a set of respective n-grams and the context word ( denoted as c ) , as : g ( w , c ) = ∑ g∈Nw z > g vc , where vc is the vector representation of the context word c. Practically , a word is represented by its index in the word dictionary and a set of n-grams it contains .

Summary: This paper proposes the use of hyperdimensional (HD) vectors to represent n-gram statistics. The HD vectors are first generated from the whole corpus. Then, it is aggregated or bundled to a vector for each sample as an input of a classifier training. The evaluation is conducted on four datasets: Chatbot, AskUbuntu, WebApplication and 20 News Group using a bunch of classifier including KNN, Random Forest, MLP etc.