Role of Context in Unsupervised Sentence Representation Learning: the Case of Dialog Act Modeling
Authors: Rastislav Hronsky
Emmanuel Keuleers
Venue: EMNLP 2023 Findings (2023)
Keywords: unsupervised learning
sentence representation
dialog act modeling

Abstract: Unsupervised learning of word representations involves capturing the contextual information surrounding word occurrences, which can be grounded in the observation that word form is largely disconnected from word meaning. While there are fewer reasons to believe that the same holds for sentences, learning through context has been carried over to learning representations of word sequences. However, this work pays minimal to no attention to the role of context in inferring sentence representations.
In this article, we present a dialog act tag probing task designed to explicitly compare content-, and context-oriented sentence representations inferred on utterances of telephone conversations (SwDA). Our results suggest that there is no clear benefit of context-based sentence representations over content-based sentence representations. However, there is a very clear benefit of increasing the dimensionality of the sentence vectors in nearly all approaches.

Role of Context in Unsupervised Sentence Representation Learning: the Case of Dialog Act Tagging
Rastislav Hronsky
Emmanuel Keuleers
Jheronimus Academy Data Science
Tilburg University
Sint Janssingel 92, 521 DA ss-Hertogenbosch Warandelaan 2, 5037 AB Tilburg
Netherlands
Netherlands
r. hronsky@tue.nl
E.A. #.A.Keullersst..bburgnvvvvrrrr
Abstract
that it works well for words (Kiros al., 2015).
In the present article, we examine whether a contextual signal is of any significant benefit to general- purpose sentence representation.
Unsupervised learning of word representations involves capturing contextual information surrounding word occurrences, which can be grounded in the observation that word form is largely disconnected from word meaning. While there are fewer reasons to believe that the same holds for sentences, learning through context has been carried over learning rep- resentations of word sequences. However, this work pays minimal to no attention the role of context in inferring sentence representations. In this article, we present a dialog act tag probing task designed explicitly compare content-, and context- -oriented sentence representations inferred on utterances of telephone conversa- tions (SwDA). Our results suggest that there is no clear benefit of context- based sentence rep- resentations over content-based sentence rep- resentations. However, there is a very clear benefit of increasing the dimensionality of the sentence vectors in nearly all approaches.
To accommodate this goal, we infer two sets of ufterance representations from telephone conversa- tion transcripts, one by a model trained to encode the transcripts' content, and one by model trained to encode the transcripts context in Thought like fashion. We evaluate the learned representa- tions with linear probe classifying the dialog act tags. Below, we discuss these choices more exten- sively.
The reason for the linear probing technique is to avoid any further representation learning in the downstream task training, such that the resulting accuracy numbers solely reflect the unsupervised representations. This technique is commonly used to assess to what extent sentence representations exhibit certain linguistic properties, such as part-of- speech or syntax (Conneau et al., 2018; Chrupata and Alishahi, 2019; Perone et al., 2018; Belinkov, 2022a).
1 Introduction
Unsupervised methods for constructing word rep- resentations vary widely in their implementation (Landauer and Dumais, 1997; Mikolov et al., 2013a; Pennington et al., 2014), but share the core property that they try to capture the contextual in- formation surrounding word occurrences. This can be grounded in the observation that word form is largely disconnected from word meaning (De Saus- sure, 1916) and that, therefore, trying to extract meaning from form directly would be pointless. However, there are fewer reasons to believe that representing the meaning of fully composed sentence requires the description of its context in the way that word does. Still, much effort has been devoted to developing and evaluating context-oriented methods for sentence representa- tion (Kiros et al., 2015; Logeswaran and Lee, 201 8; Cer et al., 2018). Interestingly. the thcoretical justi- fication for the approach -roughly paraphrased- is
We chose the Switchboard Dialog Act Corpus (SwDA) (Jurafsky, 1997) corpus for the following properties: (1) its size allows for inferring repre- sentations with both traditional, statistical learn- ing methods and modern neural network based methods; (2) at the same time, the corpus is fairly narrow- -domin,, enabling small models converge to meaningful representations 1 (3) it contains ut- terance labels that we consider suitable for prob- ing task with respect to contextuality: while a dia- log act lag unquestionably depends on the utterance contents, the surrounding utterances likely provide good signal about its label too. See example utterances from the corpus in Table 1.
'As opposed to some other well-known sentence-level tasks, such as NLI (Bowman et al., 2015), SICK (Marelli et al., 2014), SST-2 (Socher et al., 2013), that require "universal" sentence representations.

---

But I, my grandparents were looking into it before.
to transform the sparse word-count matrices into dense vectors.
SD
SD
So I know what they' ve said. Uh-huh.
With the subsequent success of simple neu- ral networks such as word2vec (Mikolov et al., 201 3a,b; Goldberg and Levy, 2014) GloVe (Pen- nington et al., 2014) at representing words and phrases, seemingly naive method of averaging these word vectors was shown to produce use- ful senemne/pragrap/dooummmn representations (Arora et al., 2017).
^H
Well, I I'm trying to think.
Table I: Example dialog act tags and utterances from the SwDA corpus. (SD* Statement expanding yn answer, SD = Suatement-non-opiniee B = Backchannel, ^H = Hold before unswerfagreement).
With further success of neural networks in text processing, the research on the topic grew markedly. Among the unsupervised methods, recur- sive auto-encoders (RAEs) (Socher et al., 201 1a,b; Pollack, 1990) were trained reconstruct sen- tences via recursively encoding nodes in a depen- dency parse tree and performed well at a para- phrasing task (Dolan et al., 2004). Tree-LSTMs (Tai et al., 2015) later succeeded the RAEs in the same recursive set up but using an LSTM encoder (Hochreiter and Schmidhuber, 1997), and evaluated predicting semantic relatedness between two sentences (Marelli al., 2014) and classifying sen- timent (Socher et al., 2013). The current state-of- the-art unsupervised, resentation systems are based on pre-trained Trans- former language models (Vaswani et al., 2017) and fine-tuned via contrastive learning (Gutmann and Hyvärinen, 2012; Chen et al., 2020), i.e., pulling representations of alternative sentence formulations closer together and those of unrelated, random sen- tence pairs apart (Gao al., 2021; Chuang et al., 2022).
Using context to enhance dialog act predictions is not a novel idea (Raheja and Tetreault, 2019; Li and Wu, 2019; Kalchbrenner and Blunsom, 2013). To ascertain the usefulness of context in this task, checked for the reduction in en- tropy of tag in "unigram" setting vs. con- ditioned on previous tag, a "bigram" setting. The results were H (curreetttag) = 2.1 and H (currenttag) previoustag) = 1.86bit, mean- ing that such a context-enhanced baseline would, at least, increase the chance-level accuracy.
For the content-oriented representation variants, the representations of utterances were inferred via classic topic modeling methods, word vector aver- aging, or an LSTM-based auto-encoder (see Sec- tion 3 for details). The resulting sentence vectors were then used as input for a Skip-Thought-like training regime, where the goal was to learn their context-oriented version by projecting them such that they predictive of surrounding utterances. Additionally, we trained a version where the utter- ance encoding and projection are trained jointly, with an L.STM-based encoder of identical propor- tions with the one used for auto-encoding.
sentence rep-
The success of skip- sarp-gram-ae gram-based word represen- tations inspired researchers to transfer the idea to sentence representation. Skip-Thought (Kiros et al., 2015) is large. LSTM-based (Hochreiter and Schmidhuber, 1997), sentence encoder that trained produce representations predictive of sur- rounding sentences. and Lee, 2018) is a similar system that differs that it casts the problem as contrastive learning (Gutmann and Hyvärinen, 2012). The Universal Sentence Encoder (USE) is Transformer-bast (Vaswani et al., 2017) system introduced by Cer et al. (2018) that uses the Skip-Thought one of the tasks in multi-task learning set-up (includ- ing some supervised tasks). The task of decid- ing whether two sentences are consecutive or not (NSP task) was also used in training of BERT (De- vlin et al., 2019). The latest dialog processing sys-
2 Related Work
In the literature review, we summarize some of the most important work in sentence representa- tion and relate it to the issues outlined in previous paragraphs.
(Logeswaran
Before the deep learning era of NLP, text- processing methods relied mainly on lexical fea- tures of sentences and were invariant to word or- der. Using word frequency transformations, e.g., if-idf (Ramos et al., 2003), has proven effective base the representations on more semantically laden words. Techniques such as singular value de- composition (SVD) (Landauer and Dumais, 1997; Landauer et al., 1998), or topic modeling (Blei et al., 2003; Blei, 2012; Hofmann, 2001) were used

---

tems, built on top of Transformer language models (Vaswani et al., 2017), typically learn predict contextual utterrances within the contrastive learn- ing framework and achieve superior performance on dialog tasks compared to universal representa- tions (Wu et al., 2020; Zhou et al., 2022; Zeng et al., 2023; Oh et al., 2023).
3. Mean of GloVe word vectors (Pennington al., 2014)*
4. LSTM-based auto-encoder (Hochreiter and Schmidhuber, 1997)
5. Mean of arbitrarily assigned word vectors
While sentence contextuality is used in of the above-mentionel studies, there is little to no focus on assessing the necessity for using context to de- rive sentence representations. An ablation of the NSP task is presented by Devlin et al. (2019), sug- gesting a slight performance benefit over version where the task is omitted. However, Devlin al. (2019) only evaluate the fine-tuning pipeline, while the raw BERT representations were shown per- form poorly overall (Reimers and Gurevych, 2019). Further, results by Liu et al. (2019) are contradic- tory and do not credit a performance improvement to the NSP task. Lastly, Zhou et al. (2022) addi- tionally compare the representations resulting from dropout augmentations (Gao et al., 2021) to those of consecutive utterrances and found significant benefit in favor of the latter augmentations. Their results are perplexing, however, since the dropout version they implemented performed even worse than that of SimCSE (Gao et al., 2021), which also uses dropout augmentation, and should in fact per- form worse, not being tailored for dialog.
We also varied the final dimension of the sen- tence vector, namely D (6, 12, 25), either by set- ting the number_of_ parameter, setting the auto-encoder bottleneck size, or truncating GloVe vectors. The auto-encoder was based on 2-layer, bidirectional Long Short-Term Memory (LSTM) neural network with hidden size of Dhidden = 61, dropout of p 0.1. The variant is one where we randomly assigned feature vector from an 8- dimensional raster to each vocabulary entry (a pro- cedure by which we want to emulate the usage of 1-hot vector encoding, but with dense, real-valued vectors), and created the final sentence representa- tion by averaging corresponding word vectors.
To create the context based representations, we devised learning regimen for 2 linear layers for each variant. One layer was used to project the sen- tence content vector to the context space (encode), and the other layer was used to predict the content vectors of surrounding sentences from the context vector (decode).
Finally, we trained a Skip-Thought-like model, where each sentence was represented by sequence of word vectors (obtained via the same procedure as in variant 5), optionally truncated and padded to 20 tokens, processed via LSTT (identical with the auto-encoder in variant 4), and projected such that, conditioned on the resulting vector, decoder- LSTM reconstructs the surrounding sentences well via teacher forcing.
Using the natural language inference (NLI) task has been effective in several general sentence rep- resentation lcarning systems (Conneau et al., 2017; Reimers and Gurevych, 2019; Zhou et al., 2022). However, along with machine translation models, these systems fall outside the category of unsuper- vised sentence representations.
3 Method
In all of the context-based variants, we varied the context size between and sentences, each side.
We trained several variants of content- based repre- sentations, their context-based counterparts, and an end-to-end, Skip-Thoughh like system, trained to encode and predict sentences.
3.1 Dataset and Evaluation
To infer and evaluate the representations, we used the Switchboard Dialog Act Corpus (SwDA) (Ju- rafsky, 1997), with exception of the GloVe vectors, where we used the pre-trained glove-twilter-25 vec- tors. The SwDA contains transcripts of casual tele- phone conversations in which the two parties were instructed to discuss a certain topic, e.g. "care of the elderly". Every utterance is assigned label
We evaluated the following variants of content- based sentence representation 2.
1. Latent Semantic Analysis (LSA)*
2. Latent Dirichlet Allocation (LDA)* (Blei et al., 2003)
PEntries marked with use the GenSim implementation.
We used the pre-trained glove-twitter-25 version.

---

Train
Dev
Test
Context
Utterances
212,145
26,518
26,519
Content
utt.
2 utt.
Tokens
1,833M
227K
227K
Algo.
Dim.
Accuracy [%]]
Tokens per utt.
8.64
8.56
8.32
Maj. class
35.00
SD
35.47%
33.52%
35.00%
GLOVE
51.62
50.37
50.14
%/Other
27.45%
25.77%
25.60%
12
52.82
52.60
52.55
B
18.52%
21.54%
23.02%
25
57.29
57.24
56.52
SV
13.76%
14.60%
13.73%
LDA
51.10
51.16
51.05
AA
4.81%
4.57%
2.65%
12
53.32
52.99
52.73
25
56.82
56.36
56.13
Table 2: Summary of the SwDA dataset. In the upper section of the table, we summarize the corpus statis- ties. In the lower section list the percentages of dialog act labels per corpus split (SD==Statemennt-non opinion", B="Backchannel", "V=" "V=""Saaeementoppiiionn"" AA=" Agrec//c/ccpp
LSA
51.07
50.62
50.58
12
51.94
51.44
51.04
25
52.02
52.11
52.12
Random
36.52
41.55
41.83
12
46.99
47.17
46.47
25
48.46
47.69
47.59
Auto-enc.
53,88
52.27
51.63
describing the "role" of the act, in similar way that part-of-speech tags describe roles of words in sentences"
12
54.46
53.89
52.34
25
52.43
53.18
52.57
Skip-Th.
6
51.13
39.66
To pre-process the texts, (I) we tokenized the ut- terances with the reference Morfessor 2.0 (Virpioja et al., 2013) implementation, (2) long utterances were broken down into multiple shorter utterances with sliding-window of 20 words and step-size of 18 words, and (3) split the corpus into train, dev, and test split with respective propor- tions of 80%, 10%, and 10% of the utterances. For an overview of the final corpus statistics, including the dialog act tags, see Table 2.
12
53.33
52.38
25
55.21
57.54
Table 3: Results. The three right-most columns contain the accuracy percentages for the content-only, context of distance I, and context distance 2 variants.
for LSA, each algorithm achieved the highest score with Drodden 25.
For each representation type, we trained lin- ear regresregressson -based classifier with 10-fold validaaoliotionn (on the training set) to predict the dialog act labels. Altogether, there were 42 distinct dialog acts. To simplify the probe, we only selected the 10 most frequent labels to classify, while group- ing the remaining labels under a single category.
There were no big differences and no discern- ing patterns in the performance of the contextually projected variants.
The only case where the training regime of pre- dicting two neighboring sentences on each side scored the best was with the end-to-end, Skip- Thought like variant, which also achieved the over- all highest score.
4 Results
Discussion
The classification accuracy results are presented in Table 3. We established baseline performance the frequency of the majority class. The only variant performing below the majority baseline was the Random variant (Diidden 6), which used means of arbitrarily assigned word vectors. The other variants achieved scores in the range of up to 15- - 21% above the majority baseline. Except
All tested models performed well over the major- ity baseline, including the averaged random word vectors which exceeded the majority baseline by a large margin when coupled with high dimensional- ity. One reason for this might be that, when aver- aging word vectors of higher dimension, there is a higher chance that some important lexical features will remain separable from the noise (also appli- cable to GloVe). Only in case of LSA and Auto- Encoder, the effect of dimensionality was small. For the latter, using a variational or contrastive learning set-up up might allow for better scaling
*See the Coder's manual (Jurafsky, 1997) for more infor- mation.
*We opted for ridge regression, instead of logistic regres- sion, hecause of high colincarity (except for LDA and LSA) across feature vectors. See correlation matrices in the ap- pendix Scction A. We used the Scikit-Leam implementation.

---

performance with capacity, Based on the patterns observed here, increasing the dimensionality in- creases performance on the probe, with or without context.
sentence representations, end-to-end approaches based on reconstructing surrounding sentences are a promising way forward.
Limitations
We did not find either of these aspects discussed in the mainstream related work (Conneau et al., 2018; Belinkov, 2022b). The question of how far increasing the dimensionality helps performance should be explored in future work.
Since this work only presents evidence from sentence task, the conclusions drawn task- specific. Although we believe the chosen task be fairly general, one can easily think of tasks that would likely challenge the drawn conclusions, e.g.. recognition of humor or sarcasm. However, these are rather rare cases and it is questionable, where the best position of the boundary between unsu- pervised and task-specific lies. Additionally, the selected range of dimensions is limited to small amount, and at larger scales the patterns of results might change.
The presented results on the dialog act tag probing task are in line with related literature in that sequence/document representations with word embeddings and classical topic modeling ap- proaches tough-to-beat baseline for many neural networks (Conneau al., 2018; Reimers and Gurevych, 2019). Our results suggest that transforming existing, content-oriented represen- tations to context-oriented ones provides minimal to no meaningful improvement in representation. An improvement of small, but perhaps meaningful magnitude is one from the best performing Auto- Encoder to Skip- Thought variant 3%)) suggesting that some benefit in this training scheme may be found for sentences, too.
However, our experimental design limits us in drawing conclusions about what the Skip-Thought set-up benefited from specifically: the more gen- eral sentence representations may simply be artifact of the more difficult learning task (predict- ing neighboring sentences), as opposed being result of form disambiguation via context, the way it works with words. Future work should employ methodology that separates the contribution of the competing sources of signal more clearly.
When compared to the obvious benefits of con- learning word representations, it striking that the benefits for learning sentence representa- tions appear to be very small at best.
This observation is suggestive of a more serious blind-spot in NLP: because context-baas based repre- sentations work well for words, many researchers assume that the same holds for any level of analysis. This is akin claiming arbitrariness of the sign any level of language processing.
6 Conclusion
We presented evidence that explicitly examines the differences between content-, and context-oriented unsupervised sentence representations. While re- sults suggest that dimensionality. rather than con- text, the determining factor building good

---

References
you can cram into single vector: Probing sentence embeddings for linguistic properties. arXiv preprint arXiv:1805: 1805.01070.
Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A
simple but tough-to-Hn -o-bttt baseline for sentence em- beddings. In International conference on learning representations.
Ferdinand De Saussure. 1916. Nature of the linguistic sign. Course in general linguisties, 1:65-70.
Yonatan Belinkov. 2022a. Probing classifiers: Promises, shortcomings, and advances. Computational Linguis- tics, 48(1)2207221
Jacob Devlin, Ming Wei Chang, and Kenton Lee. 2019. Google, kt, language, ai: Bert: pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of NAACL-HLT, pages 4171- 4186.
Yonatan Belinkov. 2022b. Probing classifiers: Promises, shortcomings, and advances. Computational Linguis- tics, (8(1:207722.
William Dolan, Chris Quirk, Chris Brockett, and Bill Dolan. 2004. Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources.
David Blei. 2012. Probabilistic topic models. Com- munications of the ACM, 55(4))77-84.
David M Blei, Andrew Y Ng. and Michael Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research, 3(am))99331022.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Process- ing, pages 6894 6910, Online and Punta Cana, Do- minican Republic. Association for Computational Linguisties.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015, A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics.
Yoav Goldberg and Omer Levy. 2014. word2vec explained: deriving mikolov et al.'s negative- sampling word-embedding method. arXiv preprint 1rXiv:1402.317
Daniel Cer, Yinfei Yang. Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Steve Yuan, Chris Tar, al. 2018. Universal sentence encoder for english. In Proceedings of the 2018 conference on empiri- cal methods in natural language processing: system demonstrations, pages 169-174.
Mario Huajardo-Cespede
Michael U Gutmann and Aapo Hyvärinen, 2012. Noise- contrastive estimation of unnormalized statistical models, with applications to natural image statistics. Journal of machine learning research, 13(2).
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In In- ternational conference on machine learning, pages 1597-1607. PMLR.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 18): 1780.
Thomas Hofmann. 2001. Unsupervised learning by probabilistic latent semantic analysis. Machine learn- ing. 42(1-2):177.
Grzegorz Chrupata and Afra Alishahi. 2019. Correlat- ing neural and symbolic representations of language. arXiv preprint (0XXvv:905.06444
Dan Jurafsky. 1997. Switchboard swbd-damsl shallow- discourse-function annotation coders manual. Tech- nical report.
Yung-Sung Chuang. Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljacic. Shang- Wen Li, Scott Yih, Yoon Kim, and James Glass. 2022. DifiCSE: Difference- based contrastive learning for sentence embeddings. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4207-4211 Seattle, United States. Association for Computational Lin- guistics.
Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent convolutional neural networks for discourse compo- sitionality. arXiv preprint arXiv: 330.3384..
Ryan Kiros, Yukun Zhu, Russ Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. Advances in neural information processing systems, 28.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loie Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv:1705.023664
Thomas K Landauer and Susan T Dumais. 1997. A solu- tion plato's 's problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological review, 104(2):211.
Thomas K Landauer, Peter W Foltz, and Darrell Laham. 1998. introduction to latent semantic analysis. Discourse processes, 25(2-3);259-284,
Alexis Conneau, German Kruszewski, Guillaume Lam- ple, Lote Barrault, and Marco Baroni, 2018. What

---

Wei Li and Yunfang Wu. 2019. Multi-level gated re- current neural network for dialog act classification. arXiv preprint arXiv: arXiv:191000150 1910.01822.
Richard Socher, Eric Huang, Jeffrey Pennin, Christo- pher D Manning, and Andrew Ng. 2011a. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. Advances in neural informa- lion processing systems, 24.
Yinhan Liu, Myle Ott. Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 7907. 1692.
Richard Socher, Jeffrey Pennington, Eric H Huang. An- drew Y Ng, and Christopher D Manning. 201 lb. Semi-supervised recursive autoencoders for predict- ing sentiment distributions. In Proceedings of the 2011 conference empirical methods in natural language processing, pages 151-161.
Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representa- tions. arXiv preprint (tt:::/80302893
Richard Socher, Alex Perelygin, Jean Wu. Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empiri- cal methods in natural language processing, pages 1631-1644
Marco Marelli, Luisa Bentivogli, Marco Baroni, Raf- faella Bernardi, Stefano Menini, and Roberto 7am- parelli. 2014. Semeval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. Proceedings of the 8th international workshop on semantic evaluation (SemEval 2014), pages 1-8.
Kai Sheng Tai, Richard Socher, and Christopher Man- ning. 2015. Improved semantic representations from tree-structured long short-term memory networks. arXiv prepriat arXiv:1503.00077
Tomas Mikolov, Kai Chen, Greg Corrado, and Jef- frey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances neural information processing systems, 30.
Tomas Mikolov, Nya Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013b. Distributed representa- tions of words and phrases and their compositionality. Advances in neural information processing systems, 26.
Sami Virpioja, Peter Smit, Stig-Ame Grönroos. Mikko Kurimo, al. 2013. Morfessor 2.0: Python im- plementation and extensions for morfessor baseline. Technical report.
Minsik Oh, Jiwei Li, and Guoyin Wang. 2023. Tadse: Template-awes aware dialogue sentence embeddings.
Chien-Sheng Wu, Steven C.H. Hoi, Richard Socher. and Caiming Xiong. 2020. TOD-BERT Pre-trained natural language understanding for task- di- alogue. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 917-929, Online, Association for Computational Linguistics
Jeffrey Pennington, Richard Sacher, and Christopher D Manning, 2014, Glove: Global vectors for word rep- resentation. In Proceedings of the 2014 conference on empirical methody in natural language processing (EMNLP), pages 1532-154
Christian Perone, Roberto Silveira, and Thomas Paula. 2018. Evaluation of sentence embeddings in downstream and linguistic probing tasks. arXiv prepriat wxXiv:18060055
Weihao Zeng, Keqing He, Yejie Wang, Chen Zeng, Jin- gang Wang. Yunsen Xian, and Weiran Xu. 2023. Fu- turetod: Teaching future knowledge to pre-trained language model for task- dialogue.
Jordan B Pollack. 1990. Recursive distributed represen- tations. Artificial Intelligence, 46(1-22 1611-2):77-105.
Zhihan Zhou, Dejiao Zhang, Wei Xiao, Nicholas Ding- wall, Xiaofei Ma, Andrew Amold, and Bing Xiang. 2022. Learning dialogue representations from con- secutive utterances. In Proceedings of the 2022 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 754-768, Seattle, United States. Association for Computational Linguistics.
Vipul Raheja and Joel Tetreault. 2019. Dialogue act classification with context-aware self-attention. arXiv preprint
Juan Ramos al. 2003. Using tf-idf to determine word relevance document queries. Proceddings of the first instructional conference on machine learning. volume 242, pages 29 48. Citeseer.
Nils Reimers and Iryna Gurevych, 2019, Sentence-bert: Sentence embeddings using siamese nert-networks. arXiv preprini arXiv:1908. 10084.

---

A
Correlation Matrices
Pearson correlation between features (dim=6)
AE
GLOVE-TWITTE-25
RAND8
LSA
LDA
2
3
3
1 2
0
1
150LATE
2
2
2
2
3
3
3
4
GLOVETWIITER-25
RAND8
LSA
LDA
0
3
CONTEXT-1
1
2
2
2
2
3
3
GLOVE-WIITEE-25
RAND8
LSA
LDA
1
0 3
3
1 2
10
CONTEXT-2
05
2
2
2
2
3
3
3
4
005
5
-1.0
Figure I: Correlation matrices of feature vectors across
representation variants with dimension of 6.
Pearson correlation between features (dim=12)
AE 2
TLOTE-
RAND8
LSA
LDA
10
2 4
6
8
10
8 10
2 4 6 10
6
8
0
2
2
2
ISOLATED
4
4
6
8
10
SG
GLOVE-TWITTE--25
RAND8
LSA
LDA
0
10
2 4 6
8
2 4
6
8 10
6
8
10
0
0
CONTEXT-1
2
2
2
8
10
10
10
10
SG
GLOVE-TWITEE-22
RAND8
LSA
LDA
10
2
0
2
6
8 10
0
10
CONTEXT-2
2
00
8
005
10
10
10
-1.0
Figure 2: Correlation matrices feature vectors actoss representation variants with dimension of 12.

---

Pearson correlation between features (dim=25)
AE
GLOVE-TWITEE--2 10 15 20
RAND8
LSA
LDA
5 10
15
20
5
10
15
20
0
10
15
20
10 15
20
0
5
5
ISOLATED
10
10
10
15
15
15
15
15
SG
TLOVER 25
RAND8
LSA
LDA
5 10
15
20
10
15
20
5
10 20
10 15
5
10
15
20
CONTEXT-1
5
5
15
15
15
15
15
SG
GLOVE-TWITTER-25
RAND8
LSA
LDA
5 10
20
15
20
15
5 10 15 20
10
CONTEXT-2
5
5
10
0.0
15
15
15
15
005
20
20
-1.0
Figure Correlation matrices of feature vectors across representation variants with dimension 25.


Reviews:
[{"review_id": "z9G3Ak8oDc", "reviewer": "Reviewer_wEG6", "summary": "", "questions": "1. What are your thoughts about the issue of dimensionality? Why do you think it can influence the performance so much?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, thank you for your feedback. We assumed that contextual information can be informative for this task mainly based on various other studies that develop classifiers (at least partially) based on it, for example [1, 2, 3]. We also checked the reduction in entropy when “guessing” a single isolated label vs. predicting it given the previous label (a “bigram” setting), and the results were $H(currentlabel) = 2.1bit$ and $H(currentlabel | previouslabel) = 1.86bit$. Such a context-enhanced baseline would, at the least, result in higher chance-level accuracy.\n\nRegarding the dimensionality, it seems to be very case-specific. For the approaches that aggregate word-vectors, it might be that, as the “word signature” gets bigger, the chance increases that a presence of a certain word (i.e. lexical feature) can be more easily recovered from the final representation. In case of the auto-encoder, the seeming failure to scale with dimension may be due to its naive implementation (variational loss or a de-noising setup would probably learn more robustly). The Skip-Thought setup, an inherently more difficult task compared to the AE, benefited from higher dimension most when context was 2 (the most difficult version of the task). An alternative explanation to why this approach provided better probe scores is simply that neural network-based unsupervised representation learning scales best when (1) the representation capacity increases while (2) the task difficulty increases. To give more conclusive answers about the role of context, a further exploration would be needed, possibly including experiments with masking/denoising/contrastive learning, etc. \n\n[1] Dialogue Act Classification with Context-Aware Self-Attention (Raheja, Tetreault, 2019)\n\n[2] Recurrent convolutional neural networks for discourse compositionality (Kalchbrenner, Brunson, 2013)\n\n[3] Multi-level gated recurrent neural network for dialog act classification (Li, Wu, 2016)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper represents a valuable attempt to identify to what extent context plays a role in using unsupervised methods for learning sentence representation. At the same time, this study is a good way to attest how these models adopt dynamics that move away from human-like procedures, being influenced by other parameters. From this point of view, the findings about the importance of dimensionality and how it can improve performance, with or without context, is very interesting and it can be a good starting point for further investigations. The main objective of the paper is very clear and so are the final results obtained. ", "reasons_to_reject": "Probably the greatest weakness of this research may lie in the corpus used. The size and heterogeneity of the data are adequate for the purpose of this study, but it seems that an analysis of each conversation (and its constituent utterances) was not carried out in order to understand whether the contexts (represented by the surrounding sentences) could really be informative. This could therefore influence the models in favouring a content-based rather than context-based representation of sentences. \nFor this reason, it would be preferable to carry out an a priori study on the dataset to make it more cognitively grounded. For example, human judgements could be collected to make sure that contexts are really informative in identifying the utterance under investigation. ", "questions_to_authors": "1. What are your thoughts about the issue of dimensionality? Why do you think it can influence the performance so much?", "paper_topic_and_main_contributions": "The authors investigated to what extent context plays a role in using unsupervised methods for learning sentence representation. \nThey used a corpus made up of utterances of telephone conversations with dialog act labels. First, they trained a bidirectional LSTM to extract content-based representations (content vectors) of sentences. Then, they trained another Skip-Thought-like model for creating for each utterance the context-based representations of its surrounding sentences.\nFinally, they evaluate such representations with a probing task by testing the models in predicting the correct dialog act labels.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The concept of 'teacher forcing' at line 209 is taken for granted, but I think you should explain it, maybe with a footnote. Moreover, I think that there is a typo at line 301.\n\n"}, "extra_text": {}}, {"review_id": "DunpR9wIAg", "reviewer": "Reviewer_F3K4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, thank you for your feedback. For this study, we deliberately intended to use a task (and dataset) from a narrow domain with only few semantic categories, such that the experiment can be performed with little compute from scratch. Indeed, we did not consider some popular NLP tasks for this purpose, e.g., NLI, SICK, SST(2)), because such representations would have to be way more “universal” to be useful (in practice, most likely implemented via a (large) language model, requiring more expensive experiments). We recognise that there are more expressive, state-of-the-art models that achieve high scores on these tasks, but we did not regard the model’s overall achieved performance as compromising our objective of contrasting the two sources of signal. Nonetheless, we acknowledge that we should mention them in the related work section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well written.\n2. Experimental method is sound.", "reasons_to_reject": "1. It's a short paper, but it may benefit from an evaluation on an additional task (i.e. NLI) which is more common.\n2. Mentioning the relation between dimensionality and performance as a contribution is too marginal. In addition, the discussion in regard to LSA and Auto-Encoder seems insufficient.\n3. The paper does not cite or perform experiments with LM-based unsupervised sentence representation learning methods, starting with SimCSE and dialogue embeddings such as DSE which are state-of-the-art in unsupervised sentence embeddings. They create embeddings with BERT models. The related works section should be revised (citations below), with more experiments.\n4. Appendix A should provide an analysis of the correlation matrix, instead of a blank space.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper experiments with dialog act probing task to compare content and context-oriented sentence representations on the SwDA dataset. The results suggest no clear benefit between context-based sentence representations over content-based sentence representations. Increasing the dimensionality of the sentence vectors helps in all approaches.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- SimCSE: Simple Contrastive Learning of Sentence Embeddings https://arxiv.org/abs/2104.08821\n- DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings https://arxiv.org/abs/2204.10298\n- Learning dialogue representations from consecutive utterances https://arxiv.org/abs/2205.13568\n- TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue https://arxiv.org/abs/2004.06871\n- TaDSE: Template-aware Dialogue Sentence Embeddings https://arxiv.org/abs/2305.14299\n- FutureTOD: Teaching Future Knowledge to Pre-trained Language Model for Task-Oriented Dialogue https://arxiv.org/abs/2306.10315"}}, {"review_id": "0xAfWH2E2r", "reviewer": "Reviewer_TQzd", "summary": "", "questions": "1. Do you have any insights or intuitions as to why 1) dimensionality would matter so much, and 2) context-based methods do not benefit sentences as much as words?\n\n2. Do qualitative inspections of predicted dialog act assignment show patterns of what context-based methods capture well and content-based do not, and viceversa?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, thank you for your feedback.\n\nRegarding the importance of dimensionality (same answer as the one to the reviewer wEG6), it seems to be very case-specific. For the approaches that aggregate word-vectors, it might be that, as the “word signature” gets bigger, the chance increases that a presence of a certain word (i.e. lexical feature) can be more easily recovered from the final representation. In case of the auto-encoder, the seeming failure to scale with dimension may be due to its naive implementation (variational loss or a de-noising setup would probably learn more robustly). The Skip-Thought setup, an inherently more difficult task compared to the AE, benefited from higher dimension most when context was 2 (the most difficult version of the task). An alternative explanation to why this approach provided better probe scores is simply that neural network-based unsupervised representation learning scales best when (1) the representation capacity increases while (2) the task difficulty increases. To give more conclusive answers about the role of context, a further exploration would be needed, possibly including experiments with masking/denoising/contrastive learning, etc.\n\nRegarding why context-based methods do not benefit sentences as much as words, we conjecture that this is related to structural characteristics of language. In the introduction, we pointed to Saussure's arbitrariness of the sign: word form by itself does not encode meaning. In other words, semantic similarity between two words cannot be predicted based on the individual components of those words (sounds or characters). For sentences, this arbitrariness does not seem to hold to the same extent: words appearing in a sentence do contribute to the sentence's meaning and having words in common will contribute to sentence similarity in a way that having letters in common will not for words.\n\nRegarding qualitative inspections of dialog act assignment: When we looked at the confusion matrices, we did notice some differences. \nComparison between the models trained with auto-encoding (AE), vs. skip-thought (ST; =reconstructing neighbouring utterances), specifically AE dim=12 and no context, vs. ST dim=25 and context 1 (two models that achieved similar performance: 54.46 vs. 55.21). First, AE correctly identified more “Acknowledge (Backchannel); B” utterances, which are typically very short (e.g. “uh-huh”). On one hand, such an utterance is probably not very informative in terms of what could precede it or come next, so the contextual representation probably encodes the same information as other utterances that are de facto noise. On the other hand, for the AE, encoding short content like “uh-huh” is likely an easy task. Second, the ST correctly distinguished between more SD (statement-non-opinion) vs. SV (statement-opinion) labeled utterances (with the AE, these classifications collapsed to the majority class SD). Upon further inspection, we didn’t notice a marked contextual signal that would explain this difference, so this might in fact be a simple artefact of a better/harder learning objective in the case of ST. Lastly, when comparing the context-adjusted representations to their original (content-based) counter-parts (e.g. LDA, GloVe, LSA), we did not see any qualitative differences. \n\nThese observations are indicative of a methodological challenge that may be worth addressing in future work: when training neural network-based models with a reconstruction objective and comparing identical architectures optimised with differently sampled target signal, the differences in the representations might still be confounded between the usefulness of the target signal and the degree to which the training regimen can actually extract the useful information.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper provides a focused contribution that is well-justified and discussed. The points made are original and well explained. \n\nThe experiments provide initial results challenging a common methodological assumption in the field, that can give rise to further explorations in this area. ", "reasons_to_reject": "The scope of the empirical results is limited to a single task and corpus. \n\nMissing insights from qualitative analyses of the predictions. ", "questions_to_authors": "1. Do you have any insights or intuitions as to why 1) dimensionality would matter so much, and 2) context-based methods do not benefit sentences as much as words?\n\n2. Do qualitative inspections of predicted dialog act assignment show patterns of what context-based methods capture well and content-based do not, and viceversa?", "paper_topic_and_main_contributions": "The paper assess the gains of using sentence representations that are learned to be predictive of the discourse context, focusing on the case of dialog act classification. They train linear probing classifier from sentence representations obtained based on sentence content only or also based on context. The results point to only minimal improvements of context-based methods, challenging the assumption that such methods are beneficial for sentence representations only because beneficial for word representations. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Reading Table 3, with context = 0 meaning content-based representation, is not very intuitive.\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This short paper investigates the contribution of context in unsupervised methods for building sentence representations, by comparing content-based methods (e.g. topic modeling, word vector averaging, LSTM autoencoders) with their context-based version, which was obtained by using the same vectors as the input for a Skip Thought-like training regime. The evaluation, conducted on a dialog act tagging task on the Switchboard corpus via a linear probing classifier, reveal that the advantage of the latter training regime is not significant, while there is a clear benefit in the simple increase of the dimensionality of the sentence vector.

All the reviewers agree that this is a clear and well-conducted study. On the weak side, I tend to agree with Reviewer 2, who pointed out that the contribution of this paper is incremental, and it would be beneficial to add some extra experiments with more recent sentence representations method.
Decision: Accept-Findings