Title: UNSUPERVISED SENTENCE EMBEDDING USING DOC-UMENT STRUCTURE-BASED CONTEXT Anonymous

Abstract: We present a new unsupervised method for learning general-purpose sentence embeddings. Unlike existing methods which rely on local contexts, such as words inside the sentence or immediately neighboring sentences, our method selects, for each target sentence, influential sentences in the entire document based on a document structure. We identify a dependency structure of sentences using metadata or text styles. Furthermore, we propose a novel out-of-vocabulary word handling technique to model many domain-specific terms, which were mostly discarded by existing sentence embedding methods. We validate our model on several tasks showing 30% precision improvement in coreference resolution in a technical domain, and 7.5% accuracy increase in paraphrase detection compared to baselines.
Recently, several researchers have proposed dependency-based embedding methods using a dependency parser to consider discontinuous intra-sentence relationships (

1. INTRODUCTION:

Distributed representations are ever more leveraged to understand text (Mikolov et al., 2013a; b; Levy & Goldberg, 2014; Pennington et al., 2014) . Recently, Kiros et al. (2015) proposed a neural network model, SKIP-THOUGHT, that embeds a sentence without supervision by training the network to predict the next sentence for a given sentence. However, unlike human reading with broader context and structure in mind, the existing approaches focus on a small continuous context of neighboring sentences. These approaches work well on less structured text like movie transcripts, but do not work well on structured documents like encylopedic articles and technical reports.

To better support semantic understanding of such technical documents, we propose a new unsupervised sentence embedding framework to learn general-purpose sentence representations by leveraging long-distance dependencies between sentences in a document. We observe that understanding a sentence often requires understanding of not only the immediate context but more comprehensive context, including the document title, previous paragraphs or even related articles as shown in Figure 1. For instance, all the sentences in the document can be related to the title of the document (1(a)). The first sentence of each item in a list structure can be influenced by the sentence introducing the list (1(b)). Moreover, html documents can contain hyperlinks to provide more information about a certain term (1(c)). With the contexts obtained from document structure, we can connect ransomware with payment (1(a)) and the four hashes with Locky (1(b)).

Locky ransomware on aggressive hunt for victims:

Millions of spam emails spread new ransomware variant on the day it first appeared. A new variant of ransomware known as Locky (detected by Symantec as Trojan.Cryptolocker.AF) has been spreading quickly since it first appeared on Tuesday (February 16). The attackers behind Locky have pushed the malware aggressively, using massive spam campaigns and compromised websites. …… Ransomware is computer malware that installs covertly on a victim's computer, executes a cryptovirology attack that adversely affects it, and demands a ransom payment to decrypt it or not publish it. 

Look Into Locky Ransomware:

Locky is a new ransomware that has been released (most probably) by the Dridex gang. Not surprisingly, it is well prepared, which means that the threat actor behind it has invested sufficient resources for it, including its mature infrastructure. Let's take a look.

Analyzed samples • 7a23368ee84781d7584e058a9922f324:

o payload: 74dde1905eff75cf3328832988a785de <main focus of this analysis • d9df60c24ceca5c4d623ff48ccd4e9b9 • e7aad826559c8448cd8ba9f53f401182

(b) Sections and List:

Similarities to Dridex?

These spam campaigns have many similarities to campaigns used to spread the Dridex financial Trojan. The sheer size of the campaigns, their disguise as financial documents such as invoices, and the use of malicious macros in attached Word documents are all hallmarks of the Dridex group.

Dridex: Financial Trojan aggressively spread in millions of spam emails each day:

Built to harvest the banking credentials of victims, the virulent Dridex is now one of the most dangerous pieces of financial malware in circulation. Our approach leveraging such structural elements has several advantages. First, it can learn from technical documents containing several subtopics that may cause sudden context changes. Some sentences have dependences to distant ones if a different perspective of the topic is introduced. Using We validate our model on several NLP tasks using a Wikipedia corpus. When trained with the Wikipedia corpus, our model produces much lower loss than SKIP-THOUGHT in the target sentence prediction task, confirming that training with only local context does not work well for such documents. We also compare the performance of the learned embedding on several NLP tasks including coreference resolution and paraphrase identification. For coreference resolution, our model shows roughly 30% improvement in precision over a state-of-the-art deep learning-based approach on cybersecurity domain, and produces 7.5% increase in accuracy compared with SKIP-THOUGHT for paraphrase identification.

The main contributions of the paper include:

• We propose a general-purpose sentence embedding method which leverages long distance sentence dependencies extracted from the document structure. • We developed a rule-baed dependency annotator to automatically determine the document structure and extract all governing sentences for each sentence. • We also present a new OOV handling technique based on the document structure.

• We have applied our methods to several NLP applications using cybersecurity datasets. The experiments show that our model consistently outperform existing methods.

2. RELATED WORK:

Distributed representation of sentences, which is often called sentence embedding, has gained much attention recently, as word-level representations (Mikolov et al., 2013a; b; Levy & Goldberg, 2014; Pennington et al., 2014) are not sufficient for many sentence-level or document-level tasks, such as machine translation, sentiment analysis and coreference resolution. Recent approaches using neural networks consider some form of dependencies to train the network. Dependencies can be continuous (relating two adjacent words or sentences) or discontinuous (relating two distant words or sentences), and intra-sentence (dependency of words within a sentence) or inter-sentence (dependency between sentences). Many sentence embedding approaches leverage these dependencies of words to combine word embeddings, and can be categorized as shown in 1.

One direct extension of word embedding to sentences is combining words vectors in a continuous context window. Kusner et al. (2015) (Mikolov et al., 2013a; b) .

dependencies. Ma et al. (2015) proposes a dependency-based convolutional neural network which concatenate a word with its ancestors and siblings based on the dependency tree structure. Tai et al. (2015) proposes tree structured long short-term memory networks. These studies show that dependency-based (discontinuous) networks outperform their sequential (continuous) counterparts.

Unlike these approaches, considering only intra-sentence dependencies, SKIP-THOUGHT (Kiros et al., 2015) joins two recurrent neural networks, encoder and decoder. The encoder combines the words in a sentence into a sentence vector, and the decoder generates the next sentence. Our approach is similar to SKIP-THOUGHT since both approaches are unsupervised and use inter-sentential dependencies. However, SKIP-THOUGHT considers only continuous dependency.

Furthermore, we propose a new method to handle OOV words in sentence embedding based on the position of an OOVword in a sentence and the dependency type of the sentence. To our knowledge, there has been no sentence embedding work incorporating OOV words in formulating the training goal. Most existing systems map all OOV words to a generic unknown word token (i.e., < unk >).

Santos & Zadrozny (2014) and Horn (2017) build an embedding of an OOV word on the fly that can be used as input to our system, but not to set the training goal. Luong et al. (2015) propose a word position-based approach to address the OOV problem for neural machine translation (NMT) systems. Their methods allow a neural machine translation (NMT) system to emit, for each unknown word in the target sentence, the position of the corresponding word in the source sentence. However, their methods are not applicable to sentence embedding, as they rely on an aligned corpus. Also, our approach considers not only word positions but also the dependency types to represent OOV words in a finer-grained OOV level.

3. DOCUMENT STRUCTURED-BASED CONTEXT:

Previous methods use intra-sentence dependencies such as dependency tree, or immediately neighboring sentences for sentence embedding. However, we identify more semantically related content to a target sentence based on the document structure as shown in Figure 1 . In this section, we describe a range of such inter-sentence dependencies that can be utilized for sentence embedding and the techniques to automatically identify them. We use the following notations to describe the extraction of document structure-based context for a given sentence. Suppose we have a document D = {S 1 , . . . , S |D| }, which is a sequence of sentences. Each sentence S i is a sequence of words: s i,1 , . . . , s i,|Si| . For each target sentence S t ∈ D, there can be a subset G ⊂ D that S t depends on (For simplicity, we use G to denote a S t specific set). We call such a sentence in G a governing sentence of S t , and say G i governs S t , or S t depends on G i . Each G i is associated with S t through one of the dependency types in D described below.

3.1. TITLES:

The title of a document, especially a technical document, contains the gist of the document, and all other sentences support the title in a certain way. For instance, the title of the document can clarify the meaning of a definite noun in the sentence. Section titles play a similar role, but, mostly to the sentences within the section. We detect different levels of titles, starting from the document title to chapter, section and subsection titles. Then, we identify the region in the document which each title governs and incorporate the title in the embedding of all the sentences in the region. To identify titles in a document, we use various information from the metadata and the document content.

Document Metadata (D T M )::

We extract a document title from the <title> tag in a HTML document or from the title field in Word or PDF document metadata. Since the document title influences all sentences in a document, we consider a title obtained from D T M governs every sentence in D.

Heading Tag (D T Hn ): The heading tags <h1> to <h6> in HTML documents are often used to show document or section titles. We consider all the sentences between a heading tag and the next occurrence of the same level tag are considered under the influence of the title. 

• • • • • • • • 8".

The range of each section can be easily identified from the TOC. If the document is a HTML file, each line in the TOC tends to have a hyperlink to the corresponding section. For non-HTML documents, we can extract the page number from the TOC (e.g., page 8) and locate the corresponding content if the document includes the page numbers.

Header and Footer (D T R ): Technical documents often contain the document or section titles in the headers or footers. Thus, if the same text is repeated in the header or in the footer in many pages, we take the text as a title and consider all the sentences appearing in these pages belong to the title.

Text Styles (D T S ): Titles often have a distinctive text style. They tend to have no period at the end and contain a larger font size, a higher number of italic or bold text, and a higher ratio of capitalized words compared to non-title sentences. We first build a text style model for sentences appearing in the document body, capturing the three style attributes. If a sentence ends without a period and any dimension of its style model has higher value than that of the text style model, we consider the sentence as a title. Then, we split the document based on the detected titles and treat each slice as a section.

3.2. LISTS:

Authors often employ a list structure to describe several elements of a subject. These list structures typically state the main concept first, and, then, the supporting points are described in a bulleted, numbered or in-text list as illustrated in Figure 2 . In these lists, an item is conceptually more related to the introductory sentence than the other items in the list, but the distance can be long because of other items. Once list items are identified, we consider the sentence appearing prior to the list items as the introductory sentence and assume that it governs all the items in the list.

The categories of the products State Farm offers are as follows:

• We have property and casualty insurance.

• We offer comprehensive types of life and health insurances. • We have bank products. To extract numbered or bulleted lists, we use the list tags (e.g., <ul>, <ol>, <li>) for HTML documents. For non-HTML documents, we detect a number sequence (i.e., 1, 2, ...) or bullet symbols (e.g., -, •) repeating in multiple lines.

In-text List (D LT ): We also identify in-text lists such as "First(ly), . . .. Second(ly), . . .. Last(ly), . . ." by identifying these cue words. We consider the sentence appearing prior to the list items as the introductory sentence and assume that it governs the list items.

3.3. LINKS:

Hyperlinks (D H ): Some sentences contain hyperlinks or references to provide additional information or clarify the meaning of the sentence. We can enrich the representation of the sentence using the linked document. In this work, we use the title of the linked document in the embedding of the sentence. Alternatively, we can use the embedding of the linked document.

Footnotes and In-document Links (D F ): Footnotes also provide additional information for the target sentence. In an HTML document, such information is usually expressed with in-document hyperlinks, which ends with "#dest". In this case, we identify a sentence marked with "#dest" and add a dependency between the two sentences.

3.4. WINDOW-BASED CONTEXT (D W n )::

We also consider the traditional sequential dependency used in previous methods (Kiros et al., 2015; Gan et al., 2017) . Given a document D = {S 1 , . . . , S |D| }, the target sentence S t is considered to be governed by n sentences prior to (n < 0) or following (n > 0) S t . In our implementation, we use only one left sentence. 

4. NEURAL NETWORK MODELS:

Similarly to SKIP-THOUGHT (Kiros et al., 2015) , we train our model to generate a target sentence S t using a set of governing sentences G. However, SKIP-THOUGHT takes into account only the window-based context (D W n ), while our model considers diverse long distance context. Furthermore, we handle out-of-vocabulary (OOV) words based on their occurrences in the context. Our model has several encoders (one encoder for each G i ∈ G), a decoder and an OOV handler as shown in Figure 3 . The input to each cell is a word, represented as a dense vector. In this work, we use the pre-trained vectors from the CBOW model (Mikolov et al., 2013b) , and the word vectors can be optionally updated during the training step. Unlike existing sentence embedding methods, which include only a small fraction of words (typically high frequency words) in the vocabulary and map all other words to one OOV word by averaging all word vectors, we introduce a new OOV handler in our model. The OOV handler maps all OOV words appearing in governing sentences to variables and extend the vocabulary with the OOV variables. More details about OOV handler is described in Section 5.

We now formally describe the model given a target sentence S t and a set G of its governing sentences. We first describe the encoders that digest each G i ∈ G. Given the i-th governing sentence G i = (g i,1 , . . . , g i,|Gi| ) let w(g i,t ) be the word representation (pre-trained or randomly initialized) of word g i,t . Then, the following equations define the encoder for S i .

hi,t = RC(w(gi,t), hi,t-1; θE), hi

= h i,|G i | λi = σ(Udi + g), h0 = i λi(u dep(i) hi + a dep(i) ) + (1 -λi)(W dep(i) hi + b) (1)

where RC is a recurrent neural network cell (e.g., LSTM or GRU) that updates the memory h i,t ; θ E is the parameters for the encoder RC; λ i is an OOV weight vector that decides how much we rely on out-of-vocabulary words; d i denotes the OOV features for G i ; U and g are linear regression parameters; σ(•) is the sigmoid function; u dep and a dep are dependency-specific weight parameters; W and b are a matrix and a bias for a fully connected layer; and h0 is the aggregated information of G and is passed to the decoder for target sentence generation. Now, we define the decoder as follows:

o t , ht = RC(o t-1 , ht-1 ; θ D ), y t = softmax(V o t + c) (2)

where RC is a recurrent neural network cell that updates the memory ht and generates the output o t ; θ D is a set of parameters for the decoder RC; softmax(•) is the softmax function; and V o t + c transforms the output into the vocabulary space. That is, V o t + c generates logits for words in the vocabulary set and is used to predict the words in the target sentence.

To strike a balance between the model accuracy and the training time, we use K randomly chosen governing sentences from G for all target sentence. We use the cross entropy between y t and o t as the optimization function and update θ E , W dep(i) , b, V, c, θ D and optionally w(•).

Algorithm 1: Building OOV map Function BuildOOVMap (G, V ) Input : A governing sentence set G = {G 1 , . . . , G |G| } and a vocabulary V 0 Output:

OOV Map foreach G i ∈ G do OOVWords i ← {w j ∈ G i |w j / ∈ V, j = 1, . . . , η} W2Var i ← {w j → O i (j)|w j ∈ OOVWords i , j = 1, . . . , |OOVWords i |} return i W2Var i

5. OUT-OF-VOCABULARY (OOV) MAPPING:

Incorporating all the words from a large text collection in deep learning models is infeasible, since the amounts of memory use and training time will be too costly. Existing sentence embedding techniques reduce the vocabulary size mainly by using only high frequency words and by collapsing all other words to one unknown word. The unknown word is typically represented by the average vector of all the word vectors in the vocabulary or as a single dimension in a bag-of-word representation. However, this frequency-based filtering can lose many important words including domain-specific words and proper nouns resulting in unsatisfactory results for technical documents. Specifically, OOV word handling is desired in the following three places: (1) input embeddings to encode the governing sentences (G); (2) input embeddings to decode the target sentence (S t ); and

(3) output logits to compute the loss with respect to S t . In this work, we apply the most commonly used approach, i.e., using the average vector of all the words in the vocabulary to represent all OOV words, to generate the input embeddings of G or S t for the encoder and the decoder. To handle the OOV words in the output logits, we propose a new method using two vocabulary sets. We first select N most frequent words in the training corpus as an initial vocabulary V 0 . Note that N (typically, tens of thousands) is much smaller than the vocabulary size in the training corpus (typically, millions or billions). The OOV mapper reduces the OOV words into a smaller vocabulary V OOV of OOV variables that can represent certain OOV words given a context (e.g., an OOV variable may indicate the actor in the previous sentence).

We note that only the OOV words appearing in governing sentences influence in model training, and many semantically important words tend to appear in the beginning or at the end of the governing sentences. Thus, we use OOV variables to represent the first and the last η OOV words in a governing sentences. Specifically, we denote a j-th OOV word in the i dependency governing sentence by an OOV variable O i (j) ∈ V OOV . This idea of encoding OOV words based on their positions in a sentence is similar to Luong et al. (2015) . However, we encode OOV words using the dependency type of the sentence as well as their position in the sentence.

Our OOV handler performs the following steps. First, we build an OOV map to convert OOV words to OOV variables and vice versa. Algorithm 1 summarizes the steps to build a map which converts the first η OOV words into OOV variables. To model the last η OOV words, we reverse the words in each G i , and index them as w -1 , w -2 , . . ., then pass them to BuildOOVMap to construct

O i (-1), O i (-2), . . . , O i (-η).

Note that the mapping between OOV words and OOV variables is many-to-many. For example, suppose "We discuss Keras first' is a target sentence S t , and, "Slim and Keras are two tools you must know" is extracted as the document title by the dependency type D T S , "PLA's weekly review: Slim and Keras are two tools you must know" is extracted as the document title by D T M for S t , and, words 'Slim', 'Keras' and 'PLA' are OOV words. Then, we map the 'Slim' and 'Keras' from the first title to OOV variable O T S (1) and O T S (2) and 'PLA', 'Slim' and 'Keras' from the second title to O T M (1), O T M (2), and O T M (3) respectively. As a result, 'Keras' in S t is mapped to O T S (1) and O T M (3).

Once we have the OOV mapping and the augmented vocabulary, we can formulate an optimization goal taking into account the OOV words with a vocabulary with a manageable size. The optimization goal of each RNN cell without OOV words is to predict the next word with one correct answer. In contrast, our model allows multiple correct answers, since an OOV word can be mapped to multiple OOV variables. We use the cross entropy with soft labels as the optimization loss function. The weight of each label is determined by the inverse-square law, i.e., the weight is inversely proportional to the square of the number of words associated with the label. This weighting scheme gives a higher weight to less ambiguous dependency.

One additional component we add related to OOV words is a weight function for the governing sentences based on occurrences of proper nouns (λ i in Equation 1). Instead of equally weighing all governing sentences, we can give a higher weight to sentences with proper nouns, which are more likely to be OOV words. Thus, we introduce a feature vector representing the number of OOV proper nouns in the i-th governing sentence (d i in Equation 1). Currently, the features include # of OOV words whose initials are uppercased, # of OOV words that are uppercased, and # of OOV words with any of the letters are uppercased. Together with the linear regression parameters, U and g, the model learns the weights for different dependency types.

6. EXPERIMENTS:

In this section, we empirically evaluate our approach on various NLP tasks and compare the results with other existing methods. We trained the proposed model (OURS) and the baseline systems on 807,647 randomly selected documents from the 2009 Wikipedia dump, which is the latest Wikipedia dump in HTML format, after removing the discussion and resource (e.g., images) articles among. Since our approach leverages HTML tags to identify document structures, our model use the raw HTML files. For the baseline systems, we provide plain text version of the same articles. All models were train for 300K steps with 64-sized batches and the Adagrad optimizer (Duchi et al., 2011) . For the evaluation, we use up-to 8 governing sentences as the context for a target sentence. When a sentence has more than 8 governing sentences, we randomly choose 8 sentences. We set the maximum number of words in a sentence to be 30 and pad each sentence with special start and end of sentence symbols. We set η to 4, resulting in |V OOV | = 80.

6.1. TARGET SENTENCE PREDICTION:

Unlike most other approaches, our model and SKIP-THOUGHT (Kiros et al., 2015) can learn application-independent sentence representations without task-specific labels. Both models are trained to predict a target sentence given context. The prediction is a sequence of vectors representing probabilities of words in the target sentence. For a quantitative evaluation between the two models, we compare the prediction losses by using the same loss function, namely cross entropy loss. We randomly chose 640,000 target sentences for evaluation and computed the average loss over the 640K sentences.

We compare SKIP-THOUGHT with two versions of our model. OURS denotes our model using the document structure-based dependencies and the OOV handler. OURS-DEP denotes our model with the OOV handler but using only local context like SKIP-THOUGHT to show the impact of the OOV handler. Table 2 shows the comparison of the three models. The values in the table are the average loss per sentence. We measure the average loss value excluding OOV words for SKIP-THOUGHT, as it cannot handle OOV words. However, for our models, we measure the loss values with and without OOV words. As we can see, both OURS-DEP and OURS significantly outperform SKIP-THOUGHT resulting in 25.8% and 26.9% reduction in the loss values respectively.

6.2. PARAPHRASE DETECTION:

Further, we compare our model with SKIP-THOUGHT on a paraphrase detection task using the Microsoft Research Paraphrase corpus (Microsoft, 2016) . The data consists of 5,801 sentence pairs extracted from news data and their boolean assessments (if the pair of sentences are paraphrases of each other or not), which were determined by three assessors using majority voting. The goal is correctly classifying the boolean assessments and accuracy (# correct pairs / # all pairs) is measured. We used 4,076 pairs for training and 1,725 pairs for testing. Since the data sets contain sentence pairs only and no structural context, we evaluate only the effectiveness of the trained encoder. To compare the qualities of sentence embeddings by the two models, we use the same logistic regression classifier with features based on embedded sentences as in (Kiros et al., 2015) . Given a pair of sentences S 1 and S 2 , the features are the two embeddings of S 1 and S 2 , their entry-wise absolute difference, and their entry-wise products. Our model shows a 5% points higher accuracy than SKIP-THOUGHT in paraphrase detection (Table 3 ), demonstrating the effectiveness of our encoder trained with the structural dependencies. Note that SKIP-THOUGHT trained on the Wikipedia corpus performs worse than a model trained on books or movie scripts due to more sophisticated and less sequential structure in Wikipedia documents.

6.3. COREFERENCE RESOLUTION:

Traditionally, the coreference resolution problem is considered as a supervised pairwise classification (i.e., mention linking) or clustering problem (coreference cluster identification) relying on an annotated corpus (Haghighi & Klein, 2010; Durrett et al., 2013; Clark & Manning, 2016a; b; Lee et al., 2017) . While, recently, there have been an impressive improvement in coreference resolution, existing coreference models are usually trained for general domain entity types (i.e., 'Person', 'Location', 'Organization') and leverage metadata that are not available in technical documents (e.g., Speaker). D'Souza & Ng (2015) and Choi et al. (2014) have shown that general domain coreference resolution models do not work well for domain specific entity types.

While our system is not intended to be a coreference resolution tool, the rich sentence embedding can be used for unsupervised coreference resolution allowing it applicable to any domain. Although building a dedicated coreference resolution method to a given domain can produce better results, we claim that our approach can build a good starting set of features without supervision for a new domain. Specifically, we treat the coreference resolution problem as an inference problem given the context. To apply our model, we assume that entity mentions are detected in advance (any mention detection tool can be used), and, for a pronoun or a generic entity reference (e.g., a definite noun phrase), we select a list of candidate referents that conform to the mention types allowed by the pronoun or the definite noun. We apply the mention type-based filtering to reduce the search space, but, a span-based approach as in Lee et al. (2017) can be used as well. Then, we replace the entity reference with each of the candidate referents and compute the loss of the new sentence. Finally, we choose the referent with the lowest loss value as the result, if the ratio of its loss to the original sentence loss value is less than a threshold value θ.

To show the effectiveness of the unsupervised coreference resolution method, we compare our approach with the Stanford Deep Coreference Resolution tool (Clark & Manning, 2016b ) using a set of cybersecurity-related documents. The evaluation data consists of 563 coreferences extracted from 38 Wikipedia articles about malware programs which were not included in the training document set. We conducted experiments for several cybersecurity related entity types such as 'Malware' and 'Operating System' in addition to general entity types including 'Person' and 'Organization'. For the evaluation, we set θ to 0.99 and 1.00.

Table 4 summarizes the results of the two systems. Our model achieves higher precision and recall than DEEPCOREF. Since DEEPCOREF was trained for a general domain, its overall performance on domain specific documents is very low. Figure 4 shows the two systems' performance on different entity types. As we can see, OURS works well for domain specific entities such as 'Malware' and 'Vulnerability', while DEEPCOREF shows higher precision for 'Person' and 'Organization'. The reason OURS performs worse for 'Person' and 'Organization' is because the security documents have only a few mentions about people or organizations, and we did not use carefully crafted features as in DEEPCOREF. In Table 5 , we show the relative importance of the different dependencies compared to the sequential dependency (D W,-1 ), which is used in other methods. The values show their relative importance levels when the importance of D W,-1 is set to 1. As we can see, all levels of document and section titles, except the fourth level subsection title, play a much more significant role than the sequential dependency. The reason the title from the metadata, (D T M ), does not have a high weight as the title from the heading 1 tag (D T H1 ) is that the metadata contains extra text, "-Wikipedia", in the title for Wikipedia articles (e.g., "George W. Bush -Wikipedia" instead of "George W. Bush"). Further, hyperlinks (D H ), indocument links (D F ) and formatted lists (D LF ) are all shown to have a similar influence as the sequence dependency. The remaining dependencies, D T C , D T R , D T S , and D LT are scarcely found in the Wikipedia corpus, and thus, did not converge or were not updated. 

7. CONCLUSION:

In this paper, we presented a novel sentence embedding technique exploiting diverse types of structural contexts and domain-specific OOV words. Our method is unsupervised and applicationindependent, and it can be applied to various NLP applications. We evaluated the method on several NLP tasks including coreference resolution, paraphrase detection and sentence prediction. The results show that our model consistently outperforms the existing approaches confirming that considering the structural context generates better quality sentence representations.

:



Figures and Tables:

Figure 1: Examples of long distance dependencies between sentences

Figure 2: A sample text with a bulleted list

Figure 3: Our model architecture.

Figure 4: Performance per entity types

Categorization of sentence embedding methods. * denotes unsupervised methods. context results in insufficient input to the neural network. Using long distance dependencies, we can provide a broader context. Second, we can consider out-of-vocabulary (OOV) words using their information extracted from the structural context. The vocabulary in a neural network is always limited due to costly training time and memory use. An existing method discarding low frequency words results in losing important keywords in the technical domain.

Table content: <table><tr><td/><td>Continuity</td><td/></tr><tr><td/><td>Continuous</td><td>Discontinuous</td></tr><tr><td>Range</td><td>Intra-sentence Kusner et al. (2015); Kalchbrenner et al. (2014); Kim (2014); Wieting &amp;</td><td>Socher et al. (2011); Ma et al. (2015); Tai et al.</td></tr><tr><td/><td>Gimpel (2017); Conneau et al. (2017);</td><td>(2015)</td></tr><tr><td/><td>Palangi et al. (2016); Le &amp; Mikolov</td><td/></tr><tr><td/><td>(2014)*</td><td/></tr><tr><td/><td>Inter-sentence Kiros et al. (2015)*</td><td>Our work*</td></tr><tr><td colspan="2">only small neighboring</td><td/></tr></table>



Table content: <table/>

Comparison of our models and SKIP-THOUGHT for

Table content: <table><tr><td/><td/><td/><td colspan="2">Table 3: Comparison of para-</td></tr><tr><td colspan="2">target sentence prediction</td><td/><td colspan="2">phrase detection accuracy</td></tr><tr><td>Method</td><td>All Words</td><td>In-vocabulary</td><td>Method</td><td>Accuracy</td></tr><tr><td/><td/><td>Words</td><td>OURS</td><td>0.72</td></tr><tr><td>OURS</td><td>0.1456</td><td>0.1394</td><td>SKIP-THOUGHT</td><td>0.67</td></tr><tr><td>OURS-DEP</td><td>0.1467</td><td>0.1415</td><td/><td/></tr><tr><td colspan="2">SKIP-THOUGHT N/A</td><td>0.1907</td><td/><td/></tr></table>

Overall performance on coreference resolution

Table content: <table><tr><td/><td/><td/><td>0.6</td><td/><td/><td/><td/></tr><tr><td/><td/><td/><td>0.4</td><td/><td/><td/><td/></tr><tr><td>Method</td><td colspan="2">Precision Recall</td><td>0.2</td><td/><td/><td/><td/></tr><tr><td>OURS (θ = 0.99)</td><td>0.47</td><td>0.12</td><td/><td/><td/><td/><td/></tr><tr><td>OURS (θ = 1.00) DEEPCOREF</td><td>0.35 0.13</td><td>0.17 0.10</td><td>0</td><td>Precision Person Recall</td><td colspan="2">Precision Organization Recall</td><td colspan="2">Precision Malware Recall</td><td>Precision Vulnerability Recall</td></tr><tr><td/><td/><td/><td/><td/><td>Ours (Ө=0.99)</td><td colspan="2">Ours (Ө=1.00)</td><td>DEEPCOREF</td></tr></table>

The weights of different dependency types. * indicates non converging dependencies. dep D T M D T H1 D T H2 D T H3 D T H4 D T H5 D T C |u dep /u D W,-1 | 1.00

Table content: <table><tr><td/><td/><td>2.30</td><td>2.30</td><td>2.30</td><td>0.24</td><td>1.40</td><td>2.94*</td></tr><tr><td>dep</td><td>D T R</td><td>D T S</td><td>D LT</td><td>D LF</td><td>D H</td><td>D F</td></tr><tr><td colspan="4">|u dep /u D W,-1 | 1.23* 0.08* 2.67*</td><td>1.00</td><td>1.00</td><td>1.00</td></tr></table>

Peer review (iclr2018):
 This paper extends the idea of forming an unsupervised representation of sentences used in the SkipThought approach by using a broader set of evidence for forming the representation of a sentence. Rather than simply encoding the preceding sentence and then generating the next sentence, the model suggests that a whole bunch of related "sentences" could be encoded, including document title, section title, footnotes, hyperlinked sentences. This is a valid good idea and indeed improves results. The other main new and potentially useful idea is a new idea for handling OOVs in this context where they are represented by positional placeholder variables. This also seems helpful. The paper is able to show markedly better results on paraphrase detection that skipthought and some interesting and perhaps good results on domain-specific coreference resolution. On the negative side, the model of the paper isn*t very excitingly different. It*s a fairly straightforward extension of the earlier SkipThought model to a situation where you have multiple generators of related text. There isn*t a clear evaluation that shows the utility of the added OOV Handler, since the results with and without that handling aren*t comparable. The OOV Handler is also related to positional encoding ideas that have been used in NMT but aren*t reference. And the coreference experiment isn*t that clearly described nor necessarily that meaningful. Finally, the finding of dependencies between sentences for the multiple generators is done in a rule-based fashion, which is okay and works, but not super neural and exciting. Other comments: - p.3. Another related sentence you could possibly use is first sentence of paragraph related to all other sentences? (Works if people write paragraphs with a "topic sentence" at the beginning. - p.5. Notation seemed a bit non-standard. I thought most people use .sigma for a sigmoid (makes sense, right?), whereas you use it for a softmax and use calligraphic S for a sigmoid.... - p.5. Section 5 suggests the standard way to do OOVs is to average all word vectors. That*s one well-know way, but hardly the only way. A trained UNK encoding and use of things like character-level encoders is also quite common. - p.6. The basic idea of the OOV encoder seems a good one. In domain specific contexts, you want to be able to refer to and re-use words that appear in related sentences, since they are likely to appear again and you want to be able to generate them. A weakness of this section however is that it makes no reference to related work whatsoever. It seems like there*s quite a bit of related work. The idea of using a positional encoding so that you can generate rare words by position has previously been used in NMT, e.g. Luong et al. (Google brain) (ACL 2015). More generally, a now quite common way to handle this problem is to use "pointing" or "copying", which appears in a number of papers. (e.g., Vinyals et al. 2015) and might also have been used here and might be expected to work too. - p.7. Why such an old Wikipedia dump? Most people use a more recent one! - p.7. The paraphrase results seem good and prove the idea works. It*s a shame they don*t let you see the usefulness of the OOV model. - p.8. For various reasons, the coreference results seem less useful than they could have been, but they do show some value for the technique in the area of domain-specific coreference.