Patent ID: 12210833

DETAILED DESCRIPTION OF THE INVENTION

In the following detailed description of the preferred embodiments, reference is made to the accompanying drawings, which form a part thereof, and within which are shown by way of illustration specific embodiments by which the invention may be practiced. It is to be understood that one skilled in the art will recognize that other embodiments may be utilized, and it will be apparent to one skilled in the art that structural changes may be made without departing from the scope of the invention. Elements/components shown in diagrams are illustrative of exemplary embodiments of the disclosure and are meant to avoid obscuring the disclosure. Any headings, used herein, are for organizational purposes only and shall not be used to limit the scope of the description or the claims. Furthermore, the use of certain terms in various places in the specification of for illustration and should not be construed as limiting.

Reference in the specification to “one embodiment,” “preferred embodiment,” “an embodiment,” or “embodiments” means that a particular feature, structure, characteristic, or function described in connection with the embodiment is included in at least one embodiment of the disclosure and may be in more than one embodiment. The appearances of the phrases “in one embodiment,” “in an embodiment,” “in embodiments,” “in alternative embodiments,” “in an alternative embodiment,” or “in some embodiments” in various places in the specification are not necessarily all referring to the same embodiment or embodiments. The terms “include,” “including,” “comprise,” and “comprising” shall be understood to be open terms and any lists that follow are examples and not meant to be limited to the listed items.

Definitions:

As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” include plural referents unless the content clearly dictates otherwise. As used in this specification and the appended claims, the term “or” is generally employed in its sense including “and/or” unless the context clearly dictates otherwise.

As used herein, the term, “computer readable medium,” may refer to a computer readable signal medium or a computer readable storage medium.

As used herein, the term, “computer readable storage medium” may be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

As used herein, the term, “computer readable signal medium,” may include a propagated data signal with computer readable program PIN embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program PIN embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wire-line, optical fiber cable, radio frequency, etc., or any suitable combination of the foregoing. Computer program PIN for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C#, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages.

Aspects of the present invention may be described below with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, computing device, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer program instructions may also be loaded onto a computing device, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

As used herein, “about” means approximately or nearly and in the context of a numerical value or range set forth means ±15% of the numerical.

As used herein, the term “application” may refer to any program known in the art which may comprise a text that may be configured to convey a tone of a user. For ease of reference, the exemplary embodiment described herein refers to a social media program, but this description should not be interpreted as exclusionary of other programs.

As used herein, the term “text” may refer to a word, emoji, emoticon, gif, image, video, and/or any content known in the art which may convey a tone. For ease of reference, the exemplary embodiment described herein refers to a word, but this description should not be interpreted as exclusionary of other content.

As used herein, the term “aspect embedding” may refer to a word, emoji, emoticon, and/or any text known in the art which may convey sarcasm of a batch of text. For ease of reference, the exemplary embodiment described herein refers to a word, but this description should not be interpreted as exclusionary of other texts.

As used herein, the term “attention module” may refer to a mechanism to discover patterns in the input that are crucial for solving the given task.

As used herein, the term “self-attention module” may refer to an attention module and mechanism for sequences which helps learn the task-specific relationship between different elements of a given sequence to produce a better sequence representation.

All numerical designations, including ranges, are approximations which are varied up or down by increments of 1.0, 0.1, 0.01 or 0.001 as appropriate. It is to be understood, even if it is not always explicitly stated, that all numerical designations are preceded by the term “about”. It is also to be understood, even if it is not always explicitly stated, that the structures described herein are merely exemplary and that equivalents of such are known in the art and can be substituted for the structures explicitly stated herein.

Wherever the term “at least,” “greater than,” or “greater than or equal to” precedes the first numerical value in a series of one or more numerical values, the term “at least,” “greater than” or “greater than or equal to” applies to each of the numerical values in that series of numerical values. For example, greater than or equal to 1, 2, or 3 is equivalent to greater than or equal to 1, greater than or equal to 2, or greater than or equal to 3.

Wherever the term “no more than,” “less than,” or “less than or equal to” precedes the first numerical value in a series of one or more numerical values, the term “no more than,” “less than” or “less than or equal to” applies to each of the numerical values in that series of numerical values. For example, less than or equal to 1, 2, or 3 is equivalent to less than or equal to 1, less than or equal to 2, or less than or equal to 3.

Sarcasm Detection:

The present disclosure pertains to a system and a method of automatically detecting a tone of a batch of text of an application, such as a social media program, by leveraging a multi-head self-attention architecture. In an embodiment, as shown inFIG.1, the system architecture may include distinct components of routing data through modules comprising data pre-processing, multi-head self-attention, gated recurrent units (hereinafter “GRU”), and classification, resulting in a modeling system with improved interpretability and accuracy. Accordingly, developing models that can explain their predictions (with high interpretability) is crucial to understanding and extending use of deep learning models, enabling a wide range of applications with machine intelligence at its backbone. Existing deep learning network architectures, such as convolutional and recurrent neural networks, are not inherently interpretable and require additional visualization techniques. To avoid this, inherently interpretable self-attention modules are applied that allows the identification of elements in the input that are crucial for a given task.

As such, the present invention includes a system and method of automatically detecting a tone of a batch of text within an application. Accordingly, the detection of the tone of the batch of text of the application is automatically optimized, such that, based on a score provided, the batch of text is automatically displayed as either indicative of sarcasm and/or non-sarcasm. The system and method will be described in greater detail in the sections herein below.

As shown inFIG.1, in an embodiment, during the initial pre-processing step, a server converts a batch of text into at least one aspect embedding, which are used to train a deep learning model. Additionally, in this embodiment, a tokenizer may be applied to convert at least one aspect embedding into a dimensional embedding. Accordingly, in some embodiments, a pre-trained language module may also be applied to convert at least one aspect embedding into the at least one dimensional embedding. In this embodiments, the at least one dimensional embedding may form the input to the multi-head self-attention module, which in some embodiment, may identify at least one aspect embedding in the batch of text that indicates the presence or absence of sarcasm. In this embodiment, after routing through the multi-head self-attention module, in an embodiment, the GRU layer may aid in learning long-distance relationships among the words highlighted by the multi-head self-attention module. The GRU layer may create a sarcasm output, such that the sarcasm output may encode an entire sequence. Moreover, in some embodiments, the sarcasm output may then be used in a fully-connected layer with sigmoid activation in order to obtain the final sarcasm score, such that a presence of sarcasm is detected. Finally, the final sarcasm score may be transmitted to a classification module, via a processor. In an embodiment, the sarcasm output may then be converted into an initial tone prediction via the classification module. In this embodiment, the initial tone prediction may comprise a binary structure (e.g., [0,1]). As such, the classification module may then be configured to compare the initial tone prediction with a ground-truth label. In this embodiment, the classification module may comprise the ground-truth label. Accordingly, the ground-ruth label may also comprise a binary structure (e.g., [0,1]). In this embodiment, after the classification module compares the initial tone prediction with the ground-truth label, a final tone score may be computed, such that the final tone score may comprise a binary structure (e.g., [0,1]), such that 0=No Sarcasm, and 1=Sarcasm. In some embodiments, the initial tone prediction, the ground-truth label, and/or the final tone score may comprise standardized digits and probability metrics. Accordingly, each module and step will be described herein below in greater detail.

Referring toFIG.2, an exemplary process-flow diagram is provided, depicting a method of detecting a presence of sarcasm within a batch of text of an application, according to an embodiment of the present disclosure. The steps delineated in the exemplary process-flow diagram ofFIG.2are merely exemplary of an order of adjusting a color scheme on a computing device. The steps may be carried out in another order, with or without additional steps included therein.

As shown inFIG.2, the method200includes a step202of converting the batch of text, via a processor of a computing device, into at least one aspect embedding. In an embodiment, during data pre-processing, at least one aspect embedding may be used to process the batch of text. Accordingly, at least one aspect embedding may include a clustering of text based on the local context of particular text within the batch of text. Additionally, in some embodiments, the at least one aspect embedding may include a clustering of text based on a global context, such that the aspect embedding may consider an association between a particular word and every other word in a sentence. For example, aspect embeddings that rely on local context may include but are not limited to Continuous Bag of Words (hereinafter “CBOW”), Skip Grams, Word2Vec, and/or any local context that is known in the art. In addition, aspect embeddings that capture global context include but are not limited to global vectors for word representation (hereinafter “GloVe”), FastText, Embeddings from Language Models (hereinafter “ELMO”) Bidirectional Encoder Representations from Transformers (hereinafter “BERT”), and/or any global context known in the art. In some embodiments, the at least one aspect embedding may be used to capture an overall context of the batch of text, such that the context is indicative of sarcasm within the batch of text.

Next, during step204of method200, after pre-processing and conversion into at least one aspect embedding, in an embodiment, the at least one aspect embedding may be transmitted to a multi-head self-attention module, via a processor of a computing device. In this manner, given at least one aspect embedding, a tokenizer may be applied to obtain dimensional embeddings D for at least one aspect embedding within a given batch of text. In some embodiments, the tokenizer may be applied in addition to pre-trained models to obtain at least one dimensional embedding D for at least one aspect embedding within the given batch of text. In an embodiment, the at least one dimensional embedding (S={e1, e2, . . . , eN},S ∈N×D) may conform the at least one aspect embedding to the model. Accordingly, in order to detect sarcasm in sentence S, in some embodiments, specific aspect embeddings may be identified, such that essential cues to tone may be provided, for example as sarcastic connotations and negative emotions. In some embodiments, the cues may be dependent on the local context and/or global context of the at least one aspect embedding. In this manner, the importance of these cue-words depends on multiple factors that are based on different contexts. In this embodiment, the multi-head self-attention may be leveraged to identify these cue-words from the at least one aspect embedding.

Furthermore, during step204and step206of method200, as shown inFIG.2, in an embodiment, an attention module may be a mechanism configured to discover patterns in the input that are crucial for solving the given task. In deep learning models, self-attention may be an attention mechanism for sequences which helps learn the task-specific relationship between different elements of a given sequence to produce a better sequence representation. In this embodiment, in the self-attention module, there may be three linear projections: Key (K), Value (V), and Query (Q) of the given at least one dimensional embedding, may be generated, such that K, Q, V ∈N×D. Accordingly, in this embodiment, an attention map may be computed based on the similarity between K, Q, and the output of the self-attention module. As such, in this embodiment, A ∈N×Dmay also be the self-attention value between V and the learned softmax attention (QKT), as provided in the equation below:
A=softmax(QKT/√{square root over (D)})

In some embodiments, the multi-head self-attention module may provide multiple copies of the self-attention module, such that the multiple copies are used in parallel. Furthermore, in these other embodiments, each head may capture different relationships between the at least one aspect embedding in the batch of text and may identify a keyword, such that the keyword aids in classification. In this embodiment, the self-attention module may use a series of multi-head self-attention layers (hereinafter “#L”) with multiple heads (“#H”) in each layer. In some embodiments, the self-attention module may use at least 1 #L, 2 #L, 3 #L, 4 #L, 5 #L, 6 #L, 7#L, and/or 8 #L with at least 1 #H, 2 #H, 3 #H, 4 #H, and/or 5 #H.

In an embodiment, as multi-head self-attention module finds the aspect embedding within the batch of text that may be important in detecting sarcasm, some aspect embeddings may be proximate to each other or may be spaced apart within the input batch of text. Referring again toFIG.2, during step208of method200, a GRU may be used to learn a long-distance relationships between at least one aspect embedding within a batch of text. In some embodiments, the GRUs may be designed to dynamically remember and forget the information flow using Reset gates (rt) and Update gates (zt) to solve the vanishing gradient problem, that is normally found within standard recurrent neural networks. Additionally, in this embodiment, a single layer of bi-direction a GRU may be used to process the self-attention value A of the batch of text in order to make use of the contextual information from local and global contexts. In addition, during step208, using the self-attention value, A ∈N×D(e.g., the output of the self-attention module), the GRU may compute hidden states H={h1, h2, . . . , hN}, H ∈N×Dfor at least one aspect embedding of the batch of text found within the input, A ∈N×D, as provided in the equation below:
rt=σ(WrAt+Urht-1+br)
zt=σ(WzAt+Uzht-1+bz)
ht=tanh(WhAt+Uh(rt⊙ht-1)+br)
ht=zt⊙ht+(1−zt)⊙ht-1

In an embodiment, the GRU module may encompass a sigmoid function. Accordingly, in this embodiment, ⊙ may represent a σ, such that it may be the element-wise sigmoid function, and W, U, and b are the trainable weights and biases rt, zt,ht, {tilde over (h)}t∈d, where d is the size of the hidden dimension. Accordingly, in this embodiment, the GRU module may create a sarcasm output. In some embodiments, the sarcasm output of the GRU module may comprise a vector. Additionally, in some embodiments, the final hidden state, hN, may be the sarcasm output from the GRU module.

Next, as shown inFIG.2, at step210, in an embodiment the sarcasm output of the GRU may be transmitted to a classification module, via the processor of the computing device. In some embodiments, during this step of the model, a single fully-connected feed-forward layer may be used with sigmoid activation. In this manner, at step212, the classification module may compute the initial tone prediction from the sarcasm output of the GRU, hN. In this manner, the initial tone prediction of the fully connected layer is a probability score y ∈[0,1] computed as shown in equation provided below:
y=σ(WhN+b)

Where W ∈d×1are the weights of this layer, b is the bias term, and y is the initial tone prediction.

The method then proceeds to step214, and results in either step216or step218, depending on whether sarcasm may be detected within the at least one aspect embedding of the batch of text. Accordingly, at step214, in an embodiment, the classification module may query the initial ton prediction, such that a binary cross entropy (BCE) loss between the initial tone prediction y (also referred to as the “sarcasm prediction output”) and a ground-truth label ŷ may be calculated as shown in the equation provided below:
loss(y, ŷ)=ŷlog(y)+(1−ŷ) log(1−y)

Where ŷ ∈{0,1} is the binary label (for example, 1:Sarcasm and 0:No-Sarcasm) and y is the initial tone prediction. Accordingly, in some embodiments, the equation provided above may be used to train modules.

During step216, in an embodiment, the classification module of the computing device may determine that a substantial match does exist between the initial tone prediction and the ground-truth label. As such, during step216, the processor may execute instructions to generate a notification comprising a non-sarcasm score for the at least one aspect embedding of the batch of text of the application when the application is displayed. Accordingly, the display associated with the computing device includes the non-sarcasm score. In some embodiments, the non-sarcasm score may be zero (0) based on the binary label provided by the comparison between the initial tone prediction and ground-truth label of the classification module.

During step218, in an embodiment, the processor of the computing device determines that a substantial match does not exist between the initial tone prediction and the ground-truth label. As such, during step218, the processor executes instructions to generate a notification comprising a sarcasm score for at least one aspect embedding of the batch of text of the application when the application is displayed. Accordingly, the display associated with the computing device includes the sarcasm score. In some embodiments, the sarcasm score, may be one (1), based on the binary label provided by the comparison between the initial tone prediction and the ground-truth label of the classification module.

Sarcasm Architecture Interpretation:

The present disclosure may further include attention maps. In an embodiment, an attention map may be created for, such that the individual heads of the self-attention layers may be used to provide the learned attention weights for each time-step in the input. Accordingly, each time-step may be at least on aspect embedding and a per-aspect attention weight may be visualized for sample batches of text with and without sarcasm from the application. In this embodiment, the multi-head self-attention module may comprise the #L preset to 5 and the #H preset to 8 per attention. As shown inFIGS.3-4, in some embodiments, the attention analysis for at least two batches of text with sarcasm, as shown inFIG.3, and without sarcasm, as shown inFIG.4may be analyzed. Additionally, each column, as shown inFIGS.3-4may correspond to a single attention layer, such that the attention weights between the at least one aspect embedding in each head are represented using colored edges. In some embodiments, the darkness of an edge may indicate the strength of the attention weight. For example, CLS represents a classification token, and SEP represents a separator token. In addition,FIGS.5-6depicts alternative exemplary embodiments of an attention analysis map using a sample batch of text to detect sarcasm, according to an embodiment of the present disclosure. In some embodiments, the rows may correspond at least five attention layers, and the columns may correspond to at least eight heads in each layer. As shown inFIGS.5-6, in some embodiments, the at least one aspect embedding receiving the most attention may vary between different heads in each layer and also across layers.

Referring again toFIG.3, in an embodiment, when a batch of text includes sarcasm, at least one aspect embedding (“sarcasm aspect”) within the batch of text may receive more attention than another aspect embedding within the batch of text. For example, the at least one aspects embedding may include words such as “just,” “again,” “totally,” along with exclamation points, may have darker edges connecting them with at least one other aspect embedding in the batch of text. Accordingly, the at least one aspect embeddings within the batch of text which may be targeted due to having a hint at sarcasm may receive higher attention than another aspect embedding within the batch of text. In addition, in some embodiments, each aspect embedding may be attended by a different head in at least the first three (3) layers of self-attention. In these other embodiments, in the final two (2) layers, the attention may be spread out to at least one other aspect embedding within the batch of text, such that the redundancy of these layers in the model may be indicated. Contrarily, in an embodiment, as shown inFIG.4, when a batch of text contains no sarcasm, at least one aspect embedding may not be highlighted by any head in any layer. In some embodiments, each aspect embedding may be attended by a different head in at least the first two (2), four (4), and/or five (5) layers of self-attention. Additionally, in some embodiments, the at least one aspect embedding within the batch of text which are targeted due to having a hint at sarcasm may receive a lower attention than aspect embeddings within the batch of text. Moreover, in these other embodiments, the specific aspects may have lighted edges connecting them with every other aspect in the batch of text.

FIG.7depicts a sarcastic and non-sarcastic attention analyses including predictive results for sample batch of text, according to an embodiment of the present disclosure. In an embodiment, the attention weight for at least one aspect embedding may be computed by first considering the maximum attention the at least one aspect embedding receives across at least one layer of the multi-layer self-attention module. Next, the multi-head self-attention module may average the aspect weights across at least one head in the at least one layer. Finally, in this embodiment, the aspect weight for the aspect may be averaged over all the aspect embeddings in the batch of text. Accordingly, in this embodiment, the stronger the highlight for the at least one aspect embedding, the higher the attention weight may be placed on the at least one aspect embedding by the multi-head self-attention module, while the multi-head self-attention module is classifying the sentence. In some embodiments, at least one aspect embedding in the batch of text with higher weights show that the model can detect sarcastic characteristics of the aspect in the batch of text. For example, as shown inFIG.7, in some embodiments, the at least one aspect embedding, which may include words such as “totally,” “first,” and “ever” from the batch of text, as well as the aspect embeddings, such as “even,” “until,” and “already” from the batch of text, may receive a higher weight as the words that exhibit sarcasm in the batch of text, as identified by the at least one multi-head self-attention module. Furthermore, in some embodiments, in the batch of text, which is classified as non-sarcastic, the weights for the at least one aspect embedding may be low in comparison to at least one aspect embedding which may comprise sarcastic characteristics from the batch of text. In this manner in some embodiments, the weaker the highlight for the at least one aspect embedding, the higher the attention may be placed on the aspect embedding by the multi-head self-attention module, while the multi-head self-attention module is classifying the sentence.

In an embodiment multi-head self-attention-based neural network architecture may be used to detect tone in a batch of text. Accordingly, the multi-head self-attention may be additionally used to highlight at least one aspect embedding in the batch of text which provide crucial cues for tone detection. In addition, in some embodiments, GRUs may aid in learning the long-distance relationships among the at least one highlighted aspect embeddings in the batch of text. As such, the sarcasm prediction output from the GRU may be passed through a fully-connected classification layer of a classification module to obtain the final non-sarcasm notification and/or the final sarcasm notification. As shown below, several experiments were conducted on multiple datasets from varied data sources and show significant improvement over the state-of-the-art models by all evaluation metrics. The results from ablation studies and analysis of the trained model, including analyses of the learned attention weights used to interpret the trained model, show that the model may automatically identify at least one aspect embedding in the batch of text which may provide cues for tone, optimizing tone detection of the batch of text.

The following examples are provided for the purpose of exemplification and are not intended to be limiting.

EXAMPLES

Various existing datasets, as shown in Table 1, provided below, were used to test the system and method described in detail above. Each dataset includes a data source and the sample counts in train and test values, and each set is sourced from varied online platforms, including but not limited to social networking platforms and discussion forums.

TABLE 1Non-Dataset SourceTrainTestTotalSarcasticSarcasticTWITTER ™,1.3685881.9563081.6482013*Online Dialogues,3.7549384.6922.3462.3462016**TWITTER ™,51.1893.74254.93125.87229.0592017***REDDIT ™,154.70264.666219.368109.684109.6842018****News Headlines,22.8955.72428.61913.63414.9852019******In the TWITTER ™, 2013 dataset, the batches of text (referred to as “tweets”) that contain sarcasm are identified and labeled by a human annotator solely based on the contents of the text. These batches of text do not depend on prior conversational context and are limited to each single batch of text. Aspect embeddings that does not include sarcasm or those which required prior conversational context are labeled as non-sarcastic. As a pre-processing step, URLs (uniform resource locators) and usenames are removed from the batches of text.**In the Online Dialogues, 2016 dataset, which is a part of the Internet Argument Corpus, includes annotated quote-response pairs for sarcasm detection. Batches of text are assigned classifiers including general sarcasm, hyperbole, and rhetorical. In these quote-response pairs, a quote is a dialogic parent to the response. Therefore, a response post can be mapped to the same quote post or the post earlier in the thread. In the experiments described herein, the quoted text is used as a context for sarcasm detection.***In the TWITTER ™, 2017 dataset, batches of text (referred to as “tweets”) are collected by a specific account. The dataset not only contains the tweets and the replies thereto, but also the mood of the posting party at the time of tweeting. The tweets/re-tweets of the posting parties are used as the content, and the replies to the tweets are used as the context. Similar to the TWITTER ™, 2013 dataset, batches of text in this dataset are pre-processed by removing URLs and replacing usernames.****In the REDDIT ™, 2018 dataset, a self-annotated corpus for sarcasm, SARC 2.0, contains comments from a particular online discussion forum. Often during online communication, such as those communications that occur on discussion forums, users will self-annotate a publication with “\s” to denote a sarcastic intent. These self-annotated publications are altered to remove the “\s” denotation, and only the original comment is used without using any parent or child comments. Two variants of the dataset, “Main Balanced” and “Political,” are used in the experiments, with the “Political” dataset being linked to a particular subforum dedicated to political discussions.*****In News Headlines, 2019 dataset, headlines of news stories are collected from two sources: a sarcastic publication under the trade name THE ONION ™, and a news Organization published under the trade name HUFFPOST ™. In this dataset, the headlines of stories are used as the content, and the text within the articles is used as the context.

To tokenize and extract at least one aspect embedding for the input batch of text, publicly available resources are used. Specifically, tokenizer and pre-trained weights from the “BERT-base-uncased” model are used to convert a portion of the batch text to tokens, and to subsequently convert tokens to at least one aspect embeddings. The pre-trained BERT model is trained with inputs of maximum length N=512 by truncating longer inputs and padding shorter inputs with a special token <pad>. To extract the at least one aspect embedding, the weights of the pre-trained BERT model are frozen and inputs are truncated or padded (with token <pad>) based on the input length.

The 768-dimensional output for each word in the input from the final hidden layer of the BERT model is considered as the aspect embeddings. The at least one aspect embedding for the portion of the batch of text are passed through a series of multi-head self-attention layers #L, with multiple heads #H in each of the layers. The output from the self-attention layer is passed through a single bi-directional GRU layer with its hidden dimension d=512. The 512-dimensional output feature vector from the GRU layer is passed through the fully connected layer to yield a 1-dimensional output (“sarcasm output”). A sigmoid activation is applied to the sarcasm output and BCE loss is used to compute the loss between the ground truth and the predicted probability score.

The parameters in the model include weights from the Multi-Head Attention, GRU, and Fully Connected layers, as described above. When using the BERT model for extracting the at least one aspect embedding, the model is initialized with pre-trained weights and frozen while training. An Adam optimizer is used to train the model with approximately 13 million parameters, using a learning rate of 1 e−4, a batch size of 64, and a dropout set of 0.2. For reach experiment, #H=8 and #L=3 were preset.

Sarcasm Detection was posed as a classification problem using Precision (ratio of the number of correctly predicted sarcastic sentences to the total number of predicted sarcastic sentences), Recall (ratio of correctly predicted sarcastic sentences to the actual number of sarcastic sentences in the ground-truth), F1-Score (harmonic mean of precision and recall), and Accuracy as evaluation metrics to test the performance of the trained models. A threshold of 0.5 was used on the predictions from the model to compute these scores. Apart from these standard metrics, the Area Under the ROC Curve (AUC score) was also calculated, which is threshold independent.

Example 1

Sarcasm Multi-Head Self-Attention Architecture Compared to Closest Art

As shown in Tables 2-6, provided below, the results of the system and method based on publicly available datasets are presented and compared to existing methods. In each experiment, the system and method described herein outperformed the prior art, indicating the enhancement in accuracy associated with the model and optimization of sarcasm detection, as described above.

Referring again to Tables 2-6, attention maps were created for each experiment, using the individual heads of the self-attention layers to provide the learned attention weights for each time-step in the input. In the model, each time-step is an aspect and the per-aspect attention weights are visualized for sample batches of text with and without sarcasm from the REDDIT™, 2018 dataset. In the model, #L was preset to 5 and #H was preset to 8 per attention. As shown inFIGS.3-4, the attention analysis for two sample sentences with sarcasm and without sarcasm (FIG.4) are analyzed. Each column inFIGS.3-4corresponds to a single attention layer, and attention weights between words in each head are represented using colored edges. The darkness of an edge indicates the strength of the attention weight. CLS represents a classification token, and SEP represents a separator token. In addition,FIGS.5-6represent another visualization that provides an overview of attention across all the heads and layers in the model. The rows correspond to five attention layers, and the columns correspond to eight heads in each layer. As shown inFIGS.5-6, the words receiving the most attention vary between different heads in each layer and also across layers.

Referring specifically toFIG.3, for a batch of text that includes sarcasm, certain aspects receive more attention than others. For instance, aspect embeddings such as “just,” “again,” “totally,” and exclamation points have darker edges connecting them with every other word in a sentence. These are the aspects in the batch of text that hint at sarcasm and, as expected, these receive higher attention than others. In addition, each cue aspect is attended by a different head in the first three layers of self-attention. In the final two layers, the attention is spread out to every word in the sentence, indicating the redundancy of these layers in the model. Contrarily, a sample batch of text having no sarcasm is shown inFIG.4; no aspect embedding is highlighted by any head in any layer.

Turning toFIG.7, the distribution of attention over the words in a sentence for six sample sentences is visualized. An attention weight for an aspect is computed by first considering the maximum attention it receives across layers, and then averaging the weights across multiple heads in the layer. Finally, the weights for an aspect are averaged over all the words in the sentence. The stronger the highlight for a word, the higher the attention weight placed on it by the model while classifying the sentence. Aspects from the sarcastic batches of text with higher weights show that the model can detect sarcastic cues from the batch of text. For example, the words “totally,” “first,” and “ever” from the first sentence, as well as the aspects “even,” “until,” and “already” from the third sentence, receive a higher weight as the words that exhibit sarcasm in the sentences, as identified by the model. In all the samples that are classified as non-sarcastic, the weights for the individual words are very low in comparison to cue-words from the sarcastic sentences.

TABLE 2ModelPrecisionRecallF1AUCNBOW71.262.364.1—Vanilla CNN71.067.168.5—Vanilla LSTM67.367.267.2—Attention LSTM68.768.668.7—Bootstrapping62.044.051.0—EmotIDM——75.0—Fracking Sarcasm88.387.988.1—GRNN66.364.765.4—ELMo-BiLSTM75.975.075.9—ELMo-BiLSTM FULL77.873.575.3—ELMo-BiLSTM AUG68.470.869.4—A2Text-Net91.791.090.097.0This Work97.999.698.799.6Improvement over+6.2+8.6+8.7+2.6Closest Art

TABLE 3ModelPrecisionRecallF1AUCSarcasm Magnet73.371.772.5—Sentence-level Attention74.975.074.9—Self Matching Networks76.372.574.4—A2Text-Net80.380.280.188.4This Work80.981.881.288.6Improvement over+0.6+1.6+1.1+0.2Closest Art

TABLE 4Main-BalancedPoliticalModelAccuracyF1AccuracyF1Bag-of-Words63.064.059.060.0CNN65.066.062.063.0CNN-SVM68.068.070.6567.0CUE-CNN70.069.069.070.0CASCADE77.077.074.075.0SARC 2.075.0—76.0—ELMo-BiLSTM72.0—78.0—ELMo-BiLSTM FULL76.076.072.072.0This Work81.081.080.080.0Improvement over+4.0+4.0+2.0+5.0Closest Art

TABLE 5ModelPrecisionRecallF1AUCNBOW66.066.066.0—Vanilla CNN68.468.168.2—Vanilla LSTM68.363.960.7—Attention LSTM70.069.669.6—GRNN62.261.861.2—CNN-LSTM-DNN66.166.765.7—SIARN72.171.871.8—MIARN72.972.972.7—ELMo-BiLSTM74.874.774.7—ELMo-BiLSTM FULL76.076.076.0—This Work77.477.277.283.4Improvement over+1.2+1.4+1.2Closest Art

TABLE 6ModelPrecisionRecallF1AccuracyAUCHybrid———89.7—A2Text-Net86.386.286.2—93.7This Work91.991.891.891.697.4Improvement over+5.6+5.6+5.6+1.9+3.7Closest Art

Example 2

Multi-Head Self-Attention Architecture with Fixed Heads and Variable Layers

In addition, as shown in Table 7, provided below, the Sarcasm Corpus v2 Dialogues dataset was used in an ablation study (“Ablation 1”), in which the number of self-attention layers (#L) are varied, and the number of heads per layer are fixed (#H=8). From the results presented in Table 7, as the number of self-attention layers increases (#L=0, 1, 3, 5), the improvement in the performance of the model due to the additional layers becomes saturated. Due to current memory constraints, it was not feasible to have more than five self-attention layers in the model; however, it should be appreciated that with greater computing power, more self-attention layers can be used. These results show that the multi-head self-attention model achieves a 2% improvement over the baseline model where only a single GRU layer is used without any self-attention layers.

TABLE 7#L-LayersPrecisionRecallF10 (GRU only)75.675.675.61 Layer76.276.176.13 Layers77.477.277.25 Layers77.677.677.6

Example 3

Multi-Head Self-Attention Architecture with Fixed Layers and Variable Heads

Additionally, as shown in Table 8, provided below, the Sarcasm Corpus v2 Dialogues dataset was used in an ablation study (“Ablation 2”), in which the number of heads per layer (#H) are varied, and the number of self-attention layers are fixed (#L=3). From the results presented in Table 8, the performance of the model increases with the increase in the number of heads per self-attention layer.

TABLE 8#H-HeadsPrecisionRecallF11 Head74.974.574.44 Heads76.976.876.88 Heads77.477.277.2

Example 4

Multi-Head Self-Attention Architecture with Multiple Aspect Embeddings

A further ablation study (“Ablation 3”), as shown in Table 9, provided below, was performed in which the system and method described herein was trained with different aspect embeddings, including Glove-6B, Glove-840B, ELMO, and FastText, and compared to existing models. In these experiments, #H was set to 8 and #L was set to 3, using each dataset described above. The results showed improvements over the prior art, indicating that the model can achieve improved results regardless of the at least one aspect embedding selected during pre-processing.

TABLE 9ModelEmbeddingsPrecisionRecallF1AUCMIARN—72.972.972.7—ELMo-BiLSTM FULLELMO76.076.076.0—BERT77.477.277.283.4ELMO76.776.776.780.8This WorkFastText75.775.775.781.6Glove 6B76.076.076.082.3Glove 840B77.077.077.082.9

REFERENCES

Shamay-Tsoory, S. G.; Tomer, R.; Aharon-Peretz, J. The neuroanatomical basis of understanding sarcasm and its relationship to social cognition. Neuropsychology 2005, 19,288.

Skalicky, S.; Crossley, S. Linguistic Features of Sarcasm and Metaphor Production Quality. In Proceedings of the Workshop on Figurative Language Processing, New Orleans, LA, USA, 6 Jun. 2018; pp. 7-16.

Kreuz, R. J.; Caucci, G. M. Lexical influences on the perception of sarcasm. In Proceedings of the Workshop on Computational Approaches to Figurative Language, Association for Computational Linguistics, Rochester, NY, USA, 26 Apr. 2007; pp. 1-4.

Joshi, A.; Sharma, V.; Bhattacharyya, P. Harnessing context incongruity for sarcasm detection. In Proceedings of the 53rd Annual Meeting of the ACL and the 7th IJCNLP, Beijing, China, 26-31 Jul. 2015; pp. 757-762.

Ghosh, A.; Veale, T. Magnets for sarcasm: Making sarcasm detection timely, contextual and very personal. In Proceedings of the 2017 Conference on EMNLP, Copenhagen, Denmark, 7-11 Sep. 2017; pp. 482-491.

Ilic, S.; Marrese-Taylor, E.; Balazs, J.; Matsuo, Y. Deep contextualized word representations for detecting sarcasm and irony. In Proceedings of the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, Brussels, Belgium, 31 Oct. 2018; pp. 2-7.

Ghosh, D.; Fabbri, A. R.; Muresan, S. Sarcasm analysis using conversation context. Comput. Linguist. 2018, 44, 755-792.

Xiong, T.; Zhang, P.; Zhu, H.; Yang, Y. Sarcasm Detection with Self-matching Networks and Low-rank Bilinear Pooling. In Proceedings of the World Wide Web Conference, San Francisco, CA, USA, 13-17 May 2019; pp. 2115-2124.

Liu, L.; Priestley, J. L.; Zhou, Y.; Ray, H. E.; Han, M. A2text-net: A novel deep neural network for sarcasm detection. In Proceedings of the 2019 IEEE First International Conference on Cognitive Machine Intelligence (CogMI); IEEE: New York, NY, USA, 2019; pp. 118-126.

Carvalho, P.; Sarmento, L.; Silva, M. J.; De Oliveira, E. Clues for detecting irony in user-generated contents: Oh . . . !! it's so easy. In Proceedings of the 1st International CIKM Workshop on Topic-Sentiment Analysis for Mass Opinion; Association for Computing Machinery: New York, NY, USA, 2009; pp. 53-56.

González-Ibánez, R.; Muresan, S.; Wacholder, N. Identifying sarcasm in Twitter: A closer look. In Proceedings of the 49th Annual Meeting of the ACL: Human Language Technologies: Short Papers, Portland, OR, USA, 19-24 Jun. 2011; Volume 2, pp. 581-586.

Tsur, O.; Davidov, D.; Rappoport, A. ICWSM—A great catchy name: Semi-supervised recognition of sarcastic sentences in online product reviews. In Proceedings of the Fourth International AAAI Conference on Weblogs and Social Media, Washington, DC, USA, 23-26 May 2010.

Davidov, D.; Tsur, O.; Rappoport, A. Semi-supervised recognition of sarcastic sentences in twitter and amazon. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning; Association for Computational Linguistics: Stroudsburg, PA, USA, 2010; pp. 107-116.

Riloff, E.; Qadir, A.; Surve, P.; De Silva, L.; Gilbert, N.; Huang, R. Sarcasm as contrast between a positive sentiment and negative situation. In Proceedings of the 2013 Conference on EMNLP, Seattle, WA, USA, 18-21 Oct. 2013; pp. 704-714.

Wallace, B. C.; Charniak, E. Sparse, contextually informed models for irony detection: Exploiting user communities, entities and sentiment. In Proceedings of the 53rd Annual Meeting of the ACL and the 7th IJCNLP, Beijing, China, 26-31 Jul. 2015; pp. 1035-1044.

Poria, S.; Cambria, E.; Hazarika, D.; Vij, P. A Deeper Look into Sarcastic Tweets Using Deep Convolutional Neural Networks. In Proceedings of the COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, Osaka, Japan, 11-16 December 2016; pp. 1601-1612.

Amir, S.; Wallace, B. C.; Lyu, H.; Carvalho, P.; Silva, M. J. Modelling Context with User Embeddings for Sarcasm Detection in Social Media. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, Berlin, Germany, 11-12 Aug. 2016; pp. 167-177.

Hazarika, D.; Poria, S.; Gorantla, S.; Cambria, E.; Zimmermann, R.; Mihalcea, R. CASCADE: Contextual Sarcasm Detection in Online Discussion Forums. In Proceedings of the 27th International Conference on Computational Linguistics, Santa Fe, NM, USA, 20-26 Aug. 2018; pp. 1837-1848.

Rajadesingan, A.; Zafarani, R.; Liu, H. Sarcasm detection on twitter: A behavioral modeling approach. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, Shanghai, China, 2-6 Feb. 2015; pp. 97-106.

Zhang, M.; Zhang, Y.; Fu, G. Tweet sarcasm detection using deep neural network. In Proceedings of the COLING 2016, The 26th International Conference on Computational Linguistics: Technical Papers, Osaka, Japan, 11-16 Dec. 2016; pp. 2449-2460.

Ptáček, T.; Habernal, I.; Hong, J. Sarcasm detection on czech and english twitter. In Proceedings of the COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, Dublin, Ireland, 23-29 Aug. 2014; pp. 213-223.

Wang, Z.; Wu, Z.; Wang, R.; Ren, Y. Twitter sarcasm detection exploiting a context-based model. In Proceedings of the International Conference on Web Information Systems Engineering; Springer: Berlin/Heidelberg, Germany, 2015; pp. 77-91.

Joshi, A.; Tripathi, V.; Bhattacharyya, P.; Carman, M. Harnessing sequence labeling for sarcasm detection in dialogue from tv series ‘friends’. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, Berlin, Germany, 11-12 Aug. 2016; pp. 146-155.

Ghosh, A.; Veale, T. Fracking sarcasm using neural network. In Proceedings of the 7th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis; Association for Computational Linguistics: San Diego, CA, USA, 2016; pp. 161-169.

Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, L.; Polosukhin, I. Attention is all you need. In Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA, 4-9 Dec. 2017; pp. 5998-6008.

Wolf, T.; Debut, L.; Sanh, V.; Chaumond, J.; Delangue, C.; Moi, A.; Cistac, P.; Rault, T.; Louf, R.; Funtowicz, M.; et al. In Proceedings of the HuggingFace's Transformers: State-of-the-art Natural Language Processing. arXiv 2019, arXiv:1910.03771.

Mikolov, T.; Chen, K.; Corrado, G.; Dean, J. Efficient estimation of word representations in vector space. arXiv 2013, arXiv:1301.3781.

Mikolov, T.; Sutskever, I.; Chen, K.; Corrado, G. S.; Dean, J. Distributed representations of words and phrases and their compositionality. In Proceedings of the Advances in Neural Information Processing Systems, Lake Tahoe, NV, USA, 5-10 Dec. 2013; pp. 3111-3119.

Pennington, J.; Socher, R.; Manning, C. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on EMNLP, Doha, Qatar, 25-29 Oct. 2014; pp. 1532-1543.

Joulin, A.; Grave, E.; Bojanowski, P.; Mikolov, T. Bag of Tricks for Efficient Text Classification. In Proceedings of the 15th Conference of the European Chapter of the ACL, Valencia, Spain, 3-7 Apr. 2017; pp. 427-431.

Peters, M. E.; Neumann, M.; Iyyer, M.; Gardner, M.; Clark, C.; Lee, K.; Zettlemoyer, L. Deep contextualized word representations. In Proceedings of the NAACL-HLT, New Orleans, LA, USA, 1-6 Jun. 2018; pp. 2227-2237.

Devlin, J.; Chang, M. W.; Lee, K.; Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of NAACL: Human Language Technologies, Minneapolis, MN, USA, 2-7 Jun. 2019; pp. 4171-4186.

Zhou, B.; Khosla, A.; Lapedriza, A.; Oliva, A.; Torralba, A. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27-30 Jun. 2016; pp. 2921-2929.

Selvaraju, R. R.; Cogswell, M.; Das, A.; Vedantam, R.; Parikh, D.; Batra, D. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22-29 Oct. 2017; pp. 618-626.

Oraby, S.; Harrison, V.; Reed, L.; Hernandez, E.; Riloff, E.; Walker, M. Creating and Characterizing a Diverse Corpus of Sarcasm in Dialogue. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Los Angeles, CA, USA, 13-15 Sep. 2016; pp. 31-41.

Walker, M. A.; Tree, J. E. F.; Anand, P.; Abbott, R.; King, J. A Corpus for Research on Deliberation and Debate. In Proceedings of the LREC, Istanbul, Turkey, 23-25 May 2012; pp. 812-817.

Khodak, M.; Saunshi, N.; Vodrahalli, K. A Large Self-Annotated Corpus for Sarcasm. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, 7-12 May 2018.

Misra, R.; Arora, P. Sarcasm Detection using Hybrid Neural Network. arXiv 2019, arXiv:1908.07414.

Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; et al. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Proceedings of the Advances in Neural Information Processing Systems 32, Vancouver, BC, Canada, 8-14 Dec. 2019.

Farías, D. I. H.; Patti, V.; Rosso, P. Irony detection in twitter: The role of affective content. In Proceedings of the ACM Transactions on Internet Technology (TOIT); Association for Computing Machinery: New York, NY, USA, 2016; pp. 1-24.

Tay, Y.; Luu, A. T.; Hui, S. C.; Su, J. Reasoning with Sarcasm by Reading In-Between. In Proceedings of the 56th Annual Meeting of the ACL, Melbourne, Australia, 15-20 Jul. 2018; pp. 1010-1020.

Clark, K.; Khandelwal, U.; Levy, O.; Manning, C. D. What Does BERT Look at? An Analysis of BERT's Attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Florence, Italy, 1 Aug. 2019; pp. 276-286.

All referenced publications are incorporated herein by reference in their entirety, to the same extent as if each were incorporated by reference individually. Furthermore, where a definition or use of a term in a reference, which is incorporated by reference herein, is inconsistent or contrary to the definition of that term provided herein, the definition of that term provided herein applies and the definition of that term in the reference does not apply.

The advantages set forth above, and those made apparent from the foregoing description, are efficiently attained. Since certain changes may be made in the above construction without departing from the scope of the invention, it is intended that all matters contained in the foregoing description or shown in the accompanying drawings shall be interpreted as illustrative and not in a limiting sense.

It is also to be understood that the following claims are intended to cover all of the generic and specific features of the invention herein described, and all statements of the scope of the invention which, as a matter of language, might be said to fall therebetween.