Question: Do the notable differences in approaches depend on the nature of the evaluation metric?
Evidence: Distributed representations - dense real-valued vectors that encode the semantics of linguistic units - are ubiquitous in today's NLP research. For single-words or word-like entities, there are established ways to acquire such representations from naturally occurring (unlabelled) training data based on comparatively task-agnostic objectives (such as predicting adjacent words). These methods are well understood empirically BIBREF0 and theoretically BIBREF1 . The best word representation spaces reflect consistently-observed aspects of human conceptual organisation BIBREF2 , and can be added as features to improve the performance of numerous language processing systems BIBREF3 .
By contrast, there is comparatively little consensus on the best ways to learn distributed representations of phrases or sentences. With the advent of deeper language processing techniques, it is relatively common for models to represent phrases or sentences as continuous-valued vectors. Examples include machine translation BIBREF8 , image captioning BIBREF9 and dialogue systems BIBREF10 . While it has been observed informally that the internal sentence representations of such models can reflect semantic intuitions BIBREF11 , it is not known which architectures or objectives yield the `best' or most useful representations. Resolving this question could ultimately have a significant impact on language processing systems. Indeed, it is phrases and sentences, rather than individual words, that encode the human-like general world knowledge (or `common sense') BIBREF12 that is a critical missing part of most current language understanding systems.
We address this issue with a systematic comparison of cutting-edge methods for learning distributed representations of sentences. We constrain our comparison to methods that do not require labelled data gathered for the purpose of training models, since such methods are more cost-effective and applicable across languages and domains. We also propose two new phrase or sentence representation learning objectives - Sequential Denoising Autoencoders (SDAEs) and FastSent, a sentence-level log-linear bag-of-words model. We compare all methods on two types of task - supervised and unsupervised evaluations - reflecting different ways in which representations are ultimately to be used. In the former setting, a classifier or regression model is applied to representations and trained with task-specific labelled data, while in the latter, representation spaces are directly queried using cosine distance.
We observe notable differences in approaches depending on the nature of the evaluation metric. In particular, deeper or more complex models (which require greater time and resources to train) generally perform best in the supervised setting, whereas shallow log-linear models work best on unsupervised benchmarks. Specifically, SkipThought Vectors BIBREF13 perform best on the majority of supervised evaluations, but SDAEs are the top performer on paraphrase identification. In contrast, on the (unsupervised) SICK sentence relatedness benchmark, FastSent, a simple, log-linear variant of the SkipThought objective, performs better than all other models. Interestingly, the method that exhibits strongest performance across both supervised and unsupervised benchmarks is a bag-of-words model trained to compose word embeddings using dictionary definitions BIBREF14 . Taken together, these findings constitute valuable guidelines for the application of phrasal or sentential representation-learning to language understanding systems.
We observe notable differences in approaches depending on the nature of the evaluation metric.
Answer: Yes, they do.