MESSAGENET: MESSAGE CLASSIFICATION USING NATURAL LANGUAGE PROCESSING AND META-DATA
Authors: Adar Kahana
Oren Elisha
Venue: Submitted to ICLR 2023 (2023)
Keywords: Message classification · Meta data injection · Deep learning · Natural language processing

Abstract: In this paper we propose a new Deep Learning (DL) approach for message classification. Our method
is based on the state-of-the-art Natural Language Processing (NLP) building blocks, combined
with a novel technique for infusing the meta-data input that is typically available in messages
such as the sender information, timestamps, attached image, audio, affiliations, and more. As we
demonstrate throughout the paper, going beyond the mere text by leveraging all available channels
in the message, could yield an improved representation and higher classification accuracy. To
achieve message representation, each type of input is processed in a dedicated block in the neural
network architecture that is suitable for the data type. Such an implementation enables training all
blocks together simultaneously, and forming cross channels features in the network. We show in the
Experiments Section that in some cases, message’s meta-data holds an additional information that
cannot be extracted just from the text, and when using this information we achieve better performance.
Furthermore, we demonstrate that our multi-modality block approach outperforms other approaches
for injecting the meta data to the the text classifier.

Under review as a conference paper at ICLR 2023
MESSAGENET: MESSAGE CLASSIFICATION USING NATURAL LANGUAGE PROCESSING AND META-DATA
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper we propose a new Deep Learning (DL) approach for message classi- fication. Our method based on the state-of-the-ar Natural Language Processing (NIP) building blocks, combined with novel technique for infusing the meta- data input that is typically available in messages such as the sender information, timestamps, attached image, audio, affiliations, and more. As we demonstrate throughout the paper, going beyond the mere text by leveraging all available chan- nels in the message, could yield an improved representation and higher classifica- tion accuracy. To achieve message representation, each type of input is processed in a dedicated block in the neural network architecture that is suitable for the data type. Such an implementation enables training all blocks together simultaneously. and forming cross channels features in the network. We show in the Experiments Section that in some cases, message's meta-data holds an additional information that cannot be extracted just from the text, and when using this information we achieve better performance. Furthermore, we demonstrate that our multi multi-modality block approach outperforms other approaches for injecting the meta data the the text classifier.
INTRODUCTION
Many real world applications require message classification and regression, such as handling spam emails Karim et al. (2020), ticket routing Han al.) (2020), article sentiment review MMdhat et al. (2014) and more. Accurate message classification could improve critical scenarios such as in call centers (routing tickets based on topic) Han al. (020), alert systems (flagging highly important alert messages) Gupta et al. (2012). and categorizing incoming messages (automatically unclutter emails) Karim et al. (2020); Klimt & Yang (0004). The main distinction between text and message classification is the availability of additional attributes, such as the sender information, timestamps, attached image, audio, affiliations, and more. New message classification contests often appear in the prominent platforms (i.e., Kaggle Kaggle). showing how this topic is sought after. There already many data-sets to explore in this field, but no clear winner algorithm that fits all scenarios with high accuracy, efficiency and simplicity (in terms of implementation and interpretation).
notable advancement in the field of NL.P is the attention based transformers architecture Vaswani al. (2017). This family of methods excels in finding local connections between words, and better understanding the meaning of a sentence. A leading example is the Bidirectional Encoder Represen- tations from Transformers (BERT) et al. (2018) as well as its variations Liu al. (2019): Lan et al. (2019; Sanh et al. (201)), winning certain benchmarks Rajpurkar et al. (2018); Wang et al. (2019). Several packages, such Huggingface Transformers Wolf al. (2019). y make such models accessible and easy to use as well as provide pre-trained versions. In addition, one can use transfer learning Pan Yang (2009) further train BERT on their on data, creating a tailored model for the specific task at hand.
BERT, and often other transformer based models, are designed to handle text. They operate on the words of given text by encoding them into tokens, and by the connections between the tokens they leam the context of sentences. This approach is limited, since sometimes more information can be extracted and used, not necessarily textual. Throughout this paper refer this information as meta-data to distinguish it from the main stream of textual content (though one may recognize the core data, depending the application). For example, a meta-data could be the time stamp

---

Under review as a conference paper at ICLR 2023
of when the text was written, sent, published, etc. Another example is the writer of the text, when dealing with a small list of writers of a corpus. There have been some attempts to incorporate these into BERT models, for example by assigning artificial tokens for writers or for temporal segments (token per month for example) Zhang et al. (002T) This approach is limited since not all meta- data entries suitable for encoding by tokenization. In the example of temporal segments, more segments introduce more tokens, leading to large computational resources consumption, and less segments cause loss of information. Another approach is to concatenate the embeddings, created by the transformer module, with the outputs of an embedding module for the meta-data. In this approach, a transformer for the text is trained (using direct or transfer learning) on the text, and other separate modules (time series embedding, senders embeddings, etc.) are used to embed the deta-data, All the embeddings are then concatenated and used as inputs to a classification network. A drawback of this approach is that the internal network features are not trained from a combination of diffident input streams, and therefore avoid cross dependent features (e.g. the importance of an email is not only determined by its content, but also by who sent it, when, to whom else, attachments, etc.).
To bridge these gaps, we implement transformer based model that is able train with both the text (transformer architecture) and meta-data. We create a new architecture of a blocks based network. Each block handles different kind of inputs. Splitting to blocks enables the flexibility to handle dif- ferent kind of inputs. We present results of the method with a main block based on transformer that handles the text, and an additional block that handles the pre-processed meta-data inputs individu- ally. This method can be extended to support more complex blocks, such as an advanced DL model temporal analysis block extract information from temporal meta- data [eecccec& & Interdonato (2020), additional transformer blocks for multiple text inputs (for example, subject and body of an email), categorical data, and more. To demonstrate the performance of the method we run multiple experiments on publicly available data-sets to show the advantages of using the block architecture, and compare them to the transformer benchmark (BERT), Random Forest (RF) classifier, and Multi-Layer Perceptron (MLP) networks. We achieve competitive results, and in most cases lead those benchmarks, showcasing that there is much to extract from the meta-data compared to just using text for classification tasks.
for images Wang et al. (2017)
2 RELATED WORK
Natural language processing tasks. The publication of BERT Devlin et al. (2018) has been turning point in the text classification domain. The authors demonstrated high accuracy on com- plicated tasks such as question and answer, named entity recognition, and textual entailment Wang ctt al. (2019). Since then, many authors investigated improved architectures and variations such as ROBERTa Liu et al. (2019), ALBERT Lan et al. (2019). DistilBERT [annh et al. (2019), and more. Some focus on better performance on the benchmark tasks, and some create lighter versions of the model that reduce the computational demands while preserving competitive accuracy. Other propositions, like XLNet Yang et al. (2019) and GPT-3 Brown et al. (2020). introduce competing architectures to BERT (also using transformers). The benchmarks for these models are commonly GLUE, SuperGLUE Wang et al. (2019). SQuAD 2.0 Rajpurkar et al. (2018), and more. Text clas- sification a less common benchmark, but the models can be used for this task as shown in this paper.
Accessibility of transformers. Another contributing factor to the growing popularity of transform- ers is the variety of open-source code bases that make it easy for data-scientist to experiment with different architectures and then use it in their applications. The Huggingface transformers package Wolf et al. (2020) is a Python library that can be used to train and fine-tune models, with a large variety of base models to choose from, and straightforward implementation. The GPT-3 Brown et al. (2220) has been published open source and, similar to several other implementations, offers a convenient application programming interface (API). We mention that many librarics that do not machine-learning for text classification exist such NLTK Bird et al. (2009), spaCy Honni- bal & Montani (2017). and more. These are also easily accessible and offer advanced NLP feature extraction and other text analysis tools.
Text classification. There are many tasks in text classification, and each may be considered as a field of study. A popular one is sentiment analysis, aiming to classify texts as positive negative.
2

---

Under review as a conference paper at ICLR 2023
The survey in MMedhat et al. (2014) presents the challenges in this domain and the latest innovations. Another example is the Spam or Ham task, where one tries to differentiate a relevant email from irrelevant ones (like udvertisements, phishing attempts, etc.) Karim et al. (2020). In this work we investigate multi-label classification of messages. For example, classifying the category of a product based purchase review, the category of thread based on the posts, the culinary specialty of restaurant from customer reviews, the type of product from purchase feedback, the category of an incoming email, and so on. For each of these tasks, publicly available data-sets exist and are used in this work to quantify the success of the proposed method. In addition, there are many competitions in Kaggle Kaggle|Kaggle and other machine- learning research benchmark websites, using these data-sets.
Message classification with meta-data. There are two commonly used method to incorporate meta-data with textual information for message categorization when using transformers. The first concatenating the embeddings, computed by the transformer, with the outputs of other embedding systems that are built specifically for the meta-data. In |Rahman et al. (202T), the authors use this approach for visual and audio meta-data. In Ostendorff et al. (2019) the authors use properties of the text as meta-data and show that this approach can also work for the German language. In Xu eet al, (2020) the meta-data is the layout information of scanned documents, and the authors propose an innovative architecture to extract information from both text and layout information. There are many other studies exploring this approach. While it is simple to implement, this strategy has several drawbacks. The training is usually done independently for the text and the meta-data, and the decisions are made as a "voting between classifiers" approach. This may lead to conflicts, since the response of the text to the label may be different from the response of the meta-data and the label, resulting in very low confidence predictions. We compare the performance of the proposed method to this one in the results section. The second popular approach is to assign tokens to the meta-data, and add this to the tokenized input of the transformer. In Zhang et al. (202I), in addition to a hierarchy of labels, the authors introduce method to inject multiple meta-data inputs with varying types (web, references, etc.) as tokens to the embedding vector. Due to the simplicity of embedding the information using tokens, in terms of algorithm and implementation, developers this approach in their codes and it appears in many online notebooks and blogs (open source codes) as well. The main drawback of these methods are the robustness the input data. For example, representing an image as a series of tokens is either done by encoding the image which usually faces loss of information, computational cost. In the numerical experiments presented here, we do not compare to this method since the feature extraction we are using has a varying and potentially high number of features, which lead to computational resource exhaustion when using this method. In this work, we propose a method that can address the issues of the two popular methods, as described in the next section.
by utilizing a large number of tokens that exponentially highers the
3 APPROACH
We propose a method based on blocks to train a linguistic model with meta-data for a specific text classification task. By splitting each type of meta-data input into different blocks, one of-the-art deep-learning architectures to handle each meta-date type uniquely and more efficiently. In addition, the training is done using all block and in unified training loop, adjusting all the weights of all blocks in every optimizer step, so all information from the text and meta-data sources contributes to the learning process.
state-
3.1 BLOCKS ARCHITECTURE
The transformer models, including BERT, can be used for text classification with the input text and corresponding output labels. However, we claim that a lot of information can be found in the meta- data of the text. can be seen in Figure I we use the transformer model single block of neural network. Then, we can add additional blocks for dealing with the meta-data inputs.
With the recent developments in deep-learning, there are many advanced method of extracting in- formation from input signals for classification. For example, in Ang et al. (2020) the authors discuss ways to use deep-learnigg for analysing time series data. Messages typically have temporal ele- ment, such as the time of arrival of an email, the time when a review has been posted, a paper has been published, etc. We propose utilize these advancements together for better model training.
3

---

Under review as a conference paper at ICLR 2023
In Figure [ overall schematic view of the proposed approach is presented. In the first row, a standard transformer architecture illustrated Wolf et al. (2020). The inputs are the tokenized messages, followed by the transformer layers that initially pre-trained. These layers are further trained (using transfer learning) to produce the embeddings, and a classification layer used to predict the categories of the messages. The transformer layers in this illustration presented using dashed lines to express the transfer learning process. The second row presents a meta-data extractor using a deep-neural network. The green layers express layers trained for classification (for example, fully-com colleceddde network architecture). The third row presents another transfer learning architecture. Each row expresses different block of neural-netwer that handles different meta-data inputs.
Translomer ayars
loyerbl
Culput
Embeddinngs matrix
Input
Outpul
block
Dense
Combine
C Coallication
loyer
Meta deta block
Cutput
Mela-dala fealure exhrcc fon
Derse
Oulpul
Meta-deta
Oulpul
Mero-data unttecddnggs
Input
Figure I: A sketch diagram of the block method. The first row illustrates a transformer architecture handle textual input, while the second and third rows illustrate neural networks for handling specific meta-data input. All the modules are then combines to create a unified prediction. Blue tiles are static and green tiles are trainable layers. The dashed green tiles illustrate the transfer learning layers
3.1.1 REPRESENTATION
Using the propose approach, the expected representation of the data is much different than the one achieved by the prominent approaches mentioned in section 2 Since the blocks are trained simul- taneously, information from the meta-data may impact the training of the textual block (the transformer block), and vice-versa. Therefore, a new and different representation is achieved. As toy illustrative example, let say that single sample has text suggesting a specific class but meta-data suggesting another, the textual block would train differently (and the textual representa- tion would be different, taking this account), and the weights of the textual block would be able to capture this difference. We illustrate the different in Figure 2 In Figure 2a we illustrate using tokens inject the meta-data into the transformer layers, producing an embedding (e (e1-eM1). Figure2b we demonstrate concatenation of the embedding produced by the transformer layers (e1-eM2) with vector representation (v) of the meta-data. In Figure 2c the proposed blocks approach is illustrated. The trainable blocks are wrapped in green to emphasize that training is done together, creating unified representation of the message, compared to 2b where the transformer layers and the dense concatenated. We emphasize
layers are trained independently, and the output representations that the blocks may have different architectures and may produce different vector representations, to produce better representation.

---

Under review as a conference paper at ICLR 2023
9x
%
%
Rxx
PNG
Transformer layers
[CLS] my dog is cule [SEP] he likes play # ##ing (SEP) [Sender] [Time]...
(a) Illustration of injecting meta-data as tokens
%y
Yyu
Transformer lcyers
Dense layers
[CLS] my dog cute [SEP] likes play ##igg [SEP]
Sender, Time...
(b) Illustration of concatenation of meta-data
G
Transformer layers
Cony. layers
Dense layers
Dense layers
Image data
Audio data
Block #n data
[SEP)
(c) Illustration of the blocks approach
Figure 2: Overview of the two common methods (a, b) and the proposed approach (c), emphasizing that the blocks are trained together, so information from one block may affect the training of the other blocks (as opposed to (b) for example, where the blocks are trained separately and concatenated)
3.1.2 COMBINE
After the blocks, we propose to take combination of the layers (e.g. averaging the outputs, sum- ming the outputs, concatenating the outputs, etc.). This combination may also be trained part of the training loop. From the experimental tests we find that the best performance was achieved by having the combine step as weighted concatenation of the outputs, producing an output sized as the number of classes times the number of blocks. We then use a few dense layers to produce an output sized as the number of classes, and apply softmax activation for the classification.
3.2 TRANSFER LEARNING
The text classification blocks often transformer models. These models have complex architec- tures that achieve high performance on the available benchmark data-sets. To utilize the strength of these models, one may train network with the same architecture but with their specific data-set. However, this tends to have high computational costs, and require large volume of data which is so often available. Therefore, many of the state-of-ther -fee------h-rtttt transformer models, including BERT, have an open source pre-trained versions available online, where one can download the pre-trained version and use a small data-set to train the model for only a few more iterations on that data. Since the pre-trained model already trained for a long time to understand textual embeddings, this transfer learning method adapts the model to the new data-set with lighter training (relative to pre-training).
Transfer learning is applicable for image data-sets, audio data-sets, time series analysis, and many more|Pan & Yang 2009). The core block the proposed method is operating on text, and the current stat-of-the-art methods for text classification are transfer learning, usually with transformers. That said, one may choose to implement other blocks using transfer learning as well. Using the proposed approach, the transfer learning can be done all blocks simultaneously.
4 EXPERIMENTS
For the numerical experiments train the network of the proposed approach with two blocks: the transformer block operating on the main text, and b) the meta-data block which is a fully-connected block operating dimensional meta-data vector. We emphasize that more advanced blocks
5

---

Under review as a conference paper at ICLR 2023
can be used, but even this simple architecture provided results that demonstrated the value from the meta-data and the blocks approach. In addition, it is simple to distinguish between the transformer only architecture, and the transformer and meta-data architecture in this way, giving better ex- plainability of the contribution of the meta-data.
4.1 DATA-SETS
Amazon reviews. The Amazon product reviews data-set He & McAuley (2016) contains large number of product reviews, varying by product category (label). The main text is the description of the review. We extract meta-data from the Amazon data-set comes from the reviewer field (sender), the review creation date and time (timestamp), and the overall satisfaction (enumerated). This data- set has in total 82.83 million product reviews. We sub-sample subset of the reviews, by choosing the first 100,000 reviews of each category and then selecting the ones with the longest texts. We eventually save 4,687 reviews for training, 521 for validation, and 2,064 for testing (eliminating text-less messages). The final data-set is close to balanced (roughly the same number of reviews per category).
Yelp Open Data-set. The Yelp restaurant reviews data-set Yelp (014) contains restaurant reviews, varying by culinary class (label). The main text the restaurant review. We extract meta-data from the reviewer field (sender), the review creation date and time (timestamp), is the review use- ful/funny/cool (cnumerated), and the number of stars (enumerated), We clean the data-set similar to the Amazon one, and save 4,612 reviews for training, 513 for validation, and 2,050 for testing, roughly balanced as well.
Reddit. The Reddit data-set BBuumgartner et al. 2020) contains a large volume of Reddit posts, each belongs to a specific sub-Reddit (label). Specifically, we use the data-set of version 2 from 2010. The main text is the post content. We extract meta-data from the post uploader field (sender), the post upload date and time (timestamp), whether the post has comments (enumerated), and whether has attachments (enumerated). We clean the data-set as well and save 4,950 posts for training, 550 for validation, and 2,200 for testing. We focus on 12 sub-Redditt (12 unique labels) and the data-set roughly balanced.
Enron Email Data-set. The Enron email data-set Klimt & Yang 2004) contains a corpus of emails. Although this data-set publicly available, we had access to limited internal version that has been tagged based on email category (label). The main text is the email body. We extract meta-data from the email sender field (sender), and the email reception date and time (timestamp). We save 4,500 emails for training, 500 for validation, and 2,000 for testing. This data-set is not balanced.
4.2 FEATURE EXTRACTOR
To conduct the numerical experiments we first discuss a genuine feature extractor we use to extract the meta-data information from the available fields of the data-set. For each type of meta-data we extract corresponding features. Since different data-sets have different fields, we use generic modules to extract information from different fields of the same purpose, such as sender field (in emails data-set) or reviewer field (in restaurant reviews data-set). An overview of the meta-data features we extract from the data-set is given in Table [ Note, that the different data-sets have different fields. Some of them do not have an enumerated field, and some have more than one.
4.3 RESULTS
To demonstrate the success of the proposed framework, use simple 2-blocks network. The first block is transformer based block, specifically a pre-trained BERT model, and 2 fully-connected layers. That way we can examine the contribution of the meta-data carefully. We emphasize that with more advanced handling of the meta-data and introduction of more blocks, higher accuracy expected. The transformer models are implemented using the Huggingface Transformers pack- age Wolf et al. (2019). In addition, we used the ktrain package, specifically the text classification modules, and we-wrote them to have a message classification class.
We use the data-sets mentioned in Section 4.1 We compare multiple methods in terms of accuracy:
1. Pre-trained BERT with an additional fully-connected layer.

---

Under review as a conference paper at ICLR 2023
Field type
Features
Number of slots
Description
One hot vector,
each slot represents if the sender is one of the 100 senders not Same with sender top affiliations Compute the frequiency of
Top senders
120
Sender
Top affiliations
120
Sender frequency
1
messages received from this sender
One hot vector, each slot represents if the message arrived
Day
7
at that day "indiatting if the message arrived within working hours not Creating a histogram of 50 bins and a corresponding 50 slots one hot vector where each slot represents if the message arrived the corresponding bin time
Timestamp
Working hours
Rush hour
50
One hot vector with in the slot representing the option and 0 otherwise The numeric value
Enumerated
Enumerated value
#Options
Numeric
Numeric value
Table Review meta-data features extraction per field
2. Pre-trained BERT with random forest classifier.
3. Pre-trained BERT with concatenated meta-data and an additional fully- cally-connected layer.
4. Pre-trained BERT with concatenated meta-data and a random forest classifier.
5. Transfer learning BERT with an additional fully-connected layer.
6. Transfer learning BERT with random forest classifier.
7. Transfer learning BERT with concatenated meta-data and an additional fully-connected layer.
8. Transfer learning BERT with concatenated meta-data and random forest classifier.
9. The proposed method (BERT and meta-data) with output averaging.
10. The proposed method (BERT and meta-date with output weighting.
Methods 1-4 use embeddings from a pre-trained BERT model (without transfer learning), and train either an extra fully- culle-co layer random forest classifier Breiman (2001) for the classifica- tion. We do this either without (1-2) or with (3-4) concatenation of the meta-data to the embeddings, following the common methods in the literature. Methods 5-8 are similar, but the BERT transform- ers layers are further trained on the specific task data. Methods 3,4,7,8 are the reference methods of concatenating the BERT model embedding with meta-data embedding. Methods 9-10 the proposed ones, exploring the effect of combining the transformer output with the meta-data block output, once with averaging the block outputs and once with fully-connected layer learn the weights after concatenating the output representations of each block. The latter involves training to learn this weight, which implemented so that happens in the same training process all other weights in the network. The results are given in Table 2 By observing the results, we see that the proposed method competitive with other methods and even outperforms most of them.
An interesting result is observed for the Enron emails data-set. We notice that the pre-trained meth- ods performed better than the transfer-leariing based methods. This can be explained by the mis- matches between the textual information and the meta-data information. While the text suggests one class, the meta-data may suggest a different one. Methods 1-8, in this case, act as voting between classifiers. However, text and meta-data and leams a better representation that can take into account these differences, performing better than all other methods.
mentioned in section 3.1.1 the proposed method is influenced by both

---

Under review as a conference paper at ICLR 2023
Methodif
Amazon reviews
Yelp Open Data-set
Reddit
Enron Emails
1
0.66
0.29
0.56
0.49
2
0.61
0.22
0.52
0.49
3
0.65
0.24
0.46
0.48
4
0.61
0.22
0.5
0.5
5
0.74
0.39
0.61
0,47
6
0.73
0.38
0.6
0,47
7
0.71
0.38
0.62
0,47
8
0.73
0.39
0.6
0.47
9
0.7
0.3
0.62
0.47
10
0.77
0.4
0.62
0.53
Table 2: Results table
CONCLUSION
In this paper we proposed new framework for training classification models. The proposition relies on the availability of additional, not necessarily textual, data channels such as attached images, audio, sender and timestamp, etc. We proposed architecture with which aforementioned additional information using different blocks, along with the text, and train a neural network perform message classification more accurately. We demonstrate the strength of this method using a set examples, varying by data-set and classification algorithm, and show that the proposed method outperforms the reference related works.
utilize the
REFERENCES
John-Syin Ang, Kok-Wiy Why Ng. and Fang-Fang Chua. Modeling time series data with deep learning: A review, analysis, evaluation and future trend. In 2020 8th International Conference on Infor- mation Technology and Multimedia (ICIMU), pp. 32-37, 2020, doi: 10.1009/CCMU499871.2000 9243546.
Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. The pushshift reddit dataset, 2020. URL httpssss//a.x....oooot/o
08435,
Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with Python: analyzing text with the natural language toolkit. O'Reeilly Media, Inc.", 2009.
Leo Breiman. Random forests. Machine learning, 45(1):5-32, 2001.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Asso- ciates, Inc., 2020. URL
neurips httpps.cccppppee pcpppappr//22///1leeee Papeeee.eaaPaap.pppo pdf
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2018. URL https::/arxxv..rrg
Ming-Wei
0bs/1810.048055
Dheeraj Gupta, P.S. Joshi, A.K. Bhattacharjee, and R.S. Mundada. Ids alerts classification using knowledge-brie evaluation. In 2012 Fourth International Conference Communication Sys- tems and Networks (COMSNETS 2012), pp. 1-8, 2012.
Jianglei Han, Jing Li, and Aixin Sun. Uftr: A unified framework for ticket routing, 2020. URL
https::/aae org/ abs 2tt.//20.000000
8

---

Under review as a conference paper at ICLR 2023
Ruining He and Julian McAuley. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In proceedings of the 25th international conference on world wide web, pp. 507-517, 2016.
Matthew Honnibal and Ines Montani. spaCy 2: Natural language understanding with Bloom em- beddings, convolutional neural networks and incremental parsing. To appear, 2017.
Dino Ienco and Roberto Interdonato. Deep multivariate time series embedding clustering via attentive-gated autoencoder. In Pacific-Asia Conference Knowledge Discovery and Data Min- ing. pp. 318-329. Springer, 2020.
Kaggle. https//wwww.agglee.omm
Asif Karim, Sami Azam, Bharanidharan Shanmugam, and Krishnan Kannoorpatti. Efficient clus- tering of emails into spam and ham: The foundational study of comprehensive unsupervised framework. IEEE Access, 8:154759-154788 2020.
Bryan Klimt and Yiming Yang. The enron corpus: new dataset for email classification research. In Jean-Françen Boulicaut, Floriana Esposito, Fosca Giannotti, and Dino Pedreschi (eds.), Machine Learning: ECML 2004, pp. 217-226, Berlin, Heidelberg, 2004. Springer Berlin Heidelberg.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervised learning of language representations, 2019, URL
https://arxit wttp:oooggooooooo
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019. URL httpthtttt///rttttt wttp://w..wga/bb///..oa//
192
Walaa Medhat, Ahmed Hassan, and Hoda Korashy. Sentiment analysis algorithms and applications: A survey. Ain Shams engineering journal, 5(4): 2014.
Malte Ostendorff, Peter Bourgonje, Maria Berger, Julian Moreno-Schneidee Georg Rehm, and Bela Gipp. Enriching bert with knowledge graph embeddings for document classification. arXiv preprint 09400000900 2019.
Simno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering. 22(10):1345-135) 2009.
Tanzila Rahman, Mengyu Yang, and Leonid Sigal. Tribert: Full-body human-centric audio-visual representation learning for visual sound separation. arXiv preprint arXiv:2110.13412. 2021.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad, 2018. URL
03822
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter, 2019. URL http:/tpps//aavi..og////a///////
08
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz. Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In the Proceedings ICLR., 2019.
Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang, Cheng Li, Honggang Zhang, Xiaogang Wang, and Xiaoou Tang. Residual attention network for image classification. In Proceedings of the IEÉE conference on computer vision and pattern recognition, pp. 3156-3164, 2017.

---

Under review as a conference paper at ICLR 2023
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: State-of- the-art natural language processing, 2019. URL
http///tww....rg///////11/111111 03771
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. As- sociation for Computational Linguistics. URL httph:://///..... hwtp::/wwwwwwwww..bb...b.. aclweb. org/anthology/
2020. emnip-demos 6
Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. Layoutlm: Pre- training of text and layout for document image understanding. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, pp. 1192-1200, 2020.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc Le. Xlnet: Generalized autoregressive pretraining for language understanding. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL httttss//pppooceeig......t ctt.errr.......ppppe///////ae...
pdff
Yelp. Yelp open dataset,
2014.
Yu Zhang, Zhihong Shen, Yuxiao Dong, Kuansan Wang, and Jiawei Han. Match: Metadata-awan text classification large hierarchy. In Proceedings of the Web Conference 2021, pp. 3246- 3257, 2021.
10


Reviews:
[{"review_id": "oADF3YYqUwR", "reviewer": "Reviewer_7oAd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new approach to message classification. Specifically:\n\n* It is based on SOTA NLP building blocks.\n* It has a novel technique to infuse metadata.\n\nThis paper shows that:\n\n* Adding metadata increase the performance of their model.\n* Their ulti-modality building block outperforms other methods. ", "summary_of_the_review": "This paper is clear but lacks novelty. I am leaning towards rejecting it.", "strength_and_weaknesses": "Strengths:\n\n* This paper leverages additional information from the metadata to boost their model performance.\n* Their multi-modality building block outperforms other methods.\n\nWeaknesses:\n\n* Leveraging metadata and ensembling BERTs have been used for both research prototyping and industry best practices. I am not sure where the novelty is.\n* Using additional information from the metadata can introduce biases. I don't see any effort from the authors to debias their model and data.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "XBhs2BPTsc", "reviewer": "Reviewer_FKbD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a framework for message classification, built with standard transformer (or more precisely, BERT, as used in all experiments) along with meta-data as augmented input. Experiments were conducted on standard text datasets with meta-data of sender and timing information.", "summary_of_the_review": "Not too much technical novelty to be presented at a top ML/NLP conference", "strength_and_weaknesses": "Strength:\n- The use of metadata makes sense in some scenarios\n\nWeaknesses:\n- The motivation does not seem to be clear on the specific benchmarks: why would the time or sender information useful for classifying product categories or topic or other categorical labels?\n- Readers might not learn much information from this submission since the presented framework is rather standard while one can draw very few conclusions from the current experiments\n- No comparison with task-specific baselines: the specific datasets have all been used in previous studies while there are no comparisons on previous results, making the last sentence of the conclusion section poorly justified.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "ToMjmfarNCZ", "reviewer": "Reviewer_rydk", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to use meta-data for message classification. Specifically, the authors extract meta-data from popular benchmarks, and add them to the models used in training. The results show improvement, but the reader is not sure of the metrics used. The method resembles multi-modal models that already exist, and does not compare to existing methods (e.g. other papers).", "summary_of_the_review": "The paper proposes a simple multi-modal method to add meta-data information to message classification. The authors do not compare to existing data, and there is no analysis or discussion of the results. The method is not novel, and the paper overall could use some proofreading.", "strength_and_weaknesses": "Strengths:\n- Good problem to tackle and many datasets.\n\nWeaknesses:\n- There is no need to reserve a subsection (3.2) or a paragraph in Related Work (Accessibility of Transformers) to explain the use of pre-trained models, as this is of widespread use. Please reserve the technical details for a \"Training Details\" section under Experiments, and remove them from everywhere else for easier reading.\n- There should be a discussion of the results: are we using macro-F1 score or accuracy? Some of the results are very close together, and there should be an analysis to show why. For example, which meta-data features are important and which ones are not?\n- There is no comparison with prior work, but rather a table of results with small technical changes.\n\nAdditional notes:\n- Please fix the typos, as there are many repeated words and spelling mistakes\n- Please fix the citations -- most of them should be in parentheses (those that are included as part of the discourse should not)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "G5_Y4lC5f4", "reviewer": "Reviewer_uH2i", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a mechanism to combine text and metadata in messages. The authors propose using a \"block\" (a custom architecture or pipeline) for each metadata field which is supposedly superior to placing metadata as [SEP] separated sentences or a single pipeline for the whole metadata.", "summary_of_the_review": "- Paper presents a very trivial architecture as an \"improvement\" over various BERT based pipelines. The architecture is really not that sophisticated and seems like a straightforward thing any practitioner would try. No real technical insights.", "strength_and_weaknesses": "In my opinion, this paper is not at level of an ICLR paper. The ideas are fairly straightforward and probably among the first things attempted by ML practitioners at any single task. I do not think there are any significant technical ideas one can take from this paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject