[Review] Advances in Semantic Textual Similarity

This post follows the great research blog of Google
https://ai.googleblog.com/

## Introduction
Recent dramatic growth in natural language understanding research made great achievements. In this post, they have explained their approaches in two research papers recently published.
1. [Learning Semantic Textual Similarity from Conversations](https://arxiv.org/abs/1804.07754)
2. [Universal Sentence Encoder](https://arxiv.org/abs/1803.11175)
And they made the pre-trained model accessible on [**TensorflowHub**](https://www.tensorflow.org/hub/)

## Learning Semantic Textual Similarity from Conversations
In this paper, they introduced novel approach to understand the semantic textual similarity based on the natural conversation. The concept is that the sentences are likely to be semantically similar if they have a similar distribution of responses. For instance, "How are you?" and "How old are you?" are meaning totally different and indeed the response should be different. But ""How old are you?" and "What is your age?" are meant same thing, asking age.

<img width="427" alt="Screen Shot 2018-05-18 at 8.28.50.png" src="https://qiita-image-store.s3.amazonaws.com/0/240801/60586839-f5e7-9d13-9b53-e7b82f8690c5.png">

In this work, they have aimed to have learn the similarity between sentence through the response classification task; given a conversational input, we wish to classify the correct response from a batch of randomly selected responses.
![image3.gif](https://qiita-image-store.s3.amazonaws.com/0/240801/f73d2a6b-fdbc-48a7-997c-d72e4740b40f.gif)
However, the ultimate goal is to let a model be able to return the embeddings representing a variety of natural language relationships, including similarity and relatedness.

## Universal Sentence Encoder
They have made the three models related with the sentence encoding architecture for NLP tasks on TF Lite.
Most of them are inspired by some existing reputed techniques in NN domain though, they have dispensed the decoder system and built the model with only encoders.

1. Using DAN(Deep Average Network) in encoding
2. Using Transformer in encoding
3. Multi DAN models for encoding

Through the validation, they have found that compute time for the model using Transformer increases noticeably as sentence length increases, whereas the compute time for the DAN model stays nearly constant as sentence length is increased.

<img width="652" alt="Screen Shot 2018-05-22 at 8.52.52.png" src="https://qiita-image-store.s3.amazonaws.com/0/240801/6864577a-8f13-2c45-970b-37892b934bd5.png">

### DAN
Simply averaging the input word vectors and propagate them to feed forward network.
<img width="563" alt="Screen Shot 2018-05-22 at 8.57.31.png" src="https://qiita-image-store.s3.amazonaws.com/0/240801/fad1a42f-8b15-c074-4c35-e52525b31136.png">
reference: https://www.cs.umd.edu/~miyyer/pubs/2015_acl_dan.pdf

### Transformer
It is the neural network architecture based on a self-attention mechanism.

<img width="420" alt="Screen Shot 2018-05-22 at 9.00.40.png" src="https://qiita-image-store.s3.amazonaws.com/0/240801/cbe7860f-da61-1817-bedb-65fcae2808ac.png">
<img width="722" alt="Screen Shot 2018-05-22 at 9.01.37.png" src="https://qiita-image-store.s3.amazonaws.com/0/240801/f2b98dcd-0c0f-923b-7b74-b508d3f6d6a1.png">
reference: https://arxiv.org/pdf/1706.03762.pdf

Self dot product attention
$Attention(Q, K, V) = softmax(\frac{QK^T}{\sqrt{d_k}})$
<img width="522" alt="Screen Shot 2018-05-22 at 9.03.48.png" src="https://qiita-image-store.s3.amazonaws.com/0/240801/cad52613-ef9d-7de3-9cf7-1593428b21eb.png">


## References
Skip-Thought Vector
https://papers.nips.cc/paper/5950-skip-thought-vectors.pdf