Deep Learning Specialization (Coursera) 自習記録 (RNN W2)


# はじめに
[Deep Learning Specialization](https://www.coursera.org/specializations/deep-learning) の RNN, Week 2 (RNN W2) の内容です。

# (RNN W2L01) Word Representation
## 内容

### Word representation
- 1-hot representation
- $V = [a, aaron, ... , zulu, \<\textrm{UNK}\>] $ ($|V| = 10000$)
- 例えば man は 10000 要素のベクトルの 5391 番目が 1, それ以外がゼロ ; $O_{5381}$

### Featurized representation : word embedding

|   | Man | Woman | King | Queen | Apple | Orange | ... |
|---|---|---|---|---|---|---|---|
| Gender | -1 | 1 | -0.95 | 0.97 | 0.00 | 0.00 |
| Royal | 0.01 | 0.02 | 0.93 | 0.95 | -0.01 | 0.00 |
| Age | 0.03 | 0.02 | 0.7 | 0.69 | 0.03 | -0.02|
| Food | 0.09 | 0.01 | 0.02 | 0.01 | 0.95 | 0.97 | 
| Size | | | | | | |
| Cost | | | | | | |
| Verb | | | | | | |
| ... |  | | | || |

- この縦のベクトルを Man なら $e_{5381}$ と表現する

### Visualization word emebeddings

- t-SNE アルゴリズム


# (RNN W2L02) Using Word Embeddings
## 内容

- Named entity recognition example
- Transfer learning and  word embedding
    1. Learn word embeddings from large text corpus (1B - 100B words) (Or download pre-trained embedding online)
    2. Transfer embedding to new task with smaller training set (Say, 100k words)
    3. Optional : Continue to finetune the word embeddings with the new data
- Face encoding と考え方が似ている 

# (RNN W2L03) Properties of word embedding
## 内容
### Analogies
- Man → Woman as King → ?
- $e_{man} - e_{woman} \sim e_{king} -e_{?}$ となる ? を探す

### Analogies using word vectors
- Find word $w$

```math
\textrm{arg} \max_w \textrm{sim} \left( e_w, e_{king} - e_{man} + e_{women} \right)
```
### Cosine similarity

```math

\textrm{sim}\left( u, v \right) = \frac{u^T v}{\|u\|_2 \|v\|_2}

```

- ユークリッド距離 ($\||u - v\||^2$) でも良い (非線形になるけど)


# (RNN W2L04) Embedding matrix
## 内容

- $E o_j = e_j$ ; embedding for word $j$
- In practice, use special function to look up an embeddings. いちいち行列とベクトルの掛け算をして $e_j$ を求めるのは計算コストが高いので...



# (RNN W2L05)
## 内容
### Neural language model
- 例文 ; I want a glass of orange ____.
- 直前 4 words だけ使う (4 ; hyperparameter) と，入力は 300 * 4 = 1200 次元。ここから Softmax で単語を選ぶ

### Other context/target unit
- 例文 ; I want a glass of orange juice to go along with my cereal.
- Context の選び方
    - Last 4 words (前述)
    - 4 words on left & right
    - last 1 word
    - nearby 1 word

# (RNN W2L06)
## 内容
### Skip-grams
- 例文 ; I want a glass of orange juice to go along with my cereal.

|Context|Target|
|---|---|
|orange|juice|
|orange|glass|
|orange|my|

### Model
- Vocab size = 10k
- context $c$ ("orange") → target $t$ ("juice")
- $o_c$ → $E$ → $e_c$ → 〇(softmax) → $\hat{y}$

```math

p\left( t | c \right) = \frac{e^{\theta_t^T e_c}}{\sum_{j=1}^{10000} e^{\theta_j^T e_c}}

```

- $\theta_t$ ; parameter associate with output $t$


```math
L\left( \hat{y}, y \right) = -\sum_{i=1}^{10000} y_i \log\hat{y}_i

```

### Problems with softmax classification
- $e_c$ の探索に時間がかかる → Hierarchical softmax


# (RNN W2L07) Negative sampling
## 内容
### Defining a new language problem
- 例文 ; I want a glass of orange juice to go along with my cereal.

|context|word|target|
|---|---|---|
|orange|juice|- (positive)|
|orange|king|o (negative)|
|orange|book|o|
|orange|the|o|

- ランダムに word を選ぶ
- 単語数 $k$ は
    - $k = 5 \sim 20$ ; smaller data set
    - $k = 2 \sim 5$ ; larger data set

### Model
- softmax

```math

p\left( y=1 | c, t \right) = \sigma\left(\theta_t^T e_c \right)

```

- 全部の word でトレーニングするのではなく，ランダムに選ばれた $k$ 個の word でトレーニングする

### Selecting negative example

```math

p(w_i) = \frac{f(w_i)^\frac{3}{4}}{\sum_{j=1}^{10000} f(w_j)^\frac{3}{4}}
```
## 感想
- 実はよく分かっていない


# (RNN W2L08) GloVe word vectors
## 内容

- $X_{ij}$ ; #times $i$ appears in context of $j$

### Model

```math

\textrm{minimize} \sum_{i=1}^{10000} \sum_{j=1}^{10000} f(X_{ij}) \left( \theta_i^T e_j + b_i - b_j^\prime - \log X_{ij}  \right)^2 \\
f\left( X_{ij} \right) = 0 \ \ \textrm{if} \ X_{ij} = 0 \\
e_w^{(final)} = \frac{e_w + \theta_w}{2} 

```

- $\theta_i$, $e_j$ are symmetric



# (RNN W2L09) Sentiment classification
## 内容
### Sentiment classification

| Sentence | # stars|
|---|---|
| The dessert is excelent | $\star \star \star \star $ |
| Service was quite slow | $\star \star $ |
| Good for a quick meal, but nothing special | $\star \star \star$ |

- 単純に各単語の $e_w$ の平均を取って softmax してはダメ (言葉の順番も重要)

### RNN for sentiment classification
- many-to-one の考え方を使う


# (RNN W2L10) Debiasing word embeddings
## 内容
### The problem of bias in word embeddings

| | judge |
|---|---|
|Man : Woman as King : Queen| no problem |
| Man : Computer_Programmer as Woman : Homemaker | problem |
| Father : Doctor as Mother : Nurse | problem |

- Word embeddings can reflect gender, ethnicity, age, sexual orientation, and other bias of text used to train the model.

### Addressing bias in word embedding
1. Identify bias direction
    - $e_{he} - e_{she}$
    - $e_{male} -e_{female}$
2. Neutralize : For every word that is not definitional, project to get rid of bias
3. Equalize pairs
    - grandfather, grandmother
    - boy, girl

# 参考
- [Deep Learning Specialization (Coursera) 自習記録 (目次)](https://qiita.com/satomshr/items/dc4ae752c558798bbef6)