Machines Learning 学习笔记(Week6)

##Evaluating a Hypothesis

**1. Model Selection**
Break down our dataset into three sets:
- Traning set: 60%
- Cross Validation set: 20%
- Test set: 20%

Suppose we have several hypothesis functions with different polynomial degrees. To select the best model:

1. use the training set to optimize the parameters for each hypothesis function

2. use the validation set to find the best function (of the polynomial degree $d$) with the least error

3. use the test set with $J_{test}(Θ^{(d)})$ to estimate the generalization error (test set error)

**2. Test Set Error**

- Linear regression:

```math
J_{test}(Θ) = \frac{1}{2m_{test}} \sum_{i=1}^{m_{test}} \bigl(h_Θ(x_{test}^{(i)})-y_{test}^{(i)} \bigr)^2
```

- Classification ~ Misclassification error:

```math
err\bigl(h_Θ(x),y\bigr) = 
\begin{array}{ll}
1 & if \, h_Θ(x)\geq0.5 \, and \, y=0 \, or \, h_Θ(x)\leq0.5 \, and \, y=1  \\
0 & otherwise
\end{array}
```

The average test error for the test set is:
Test Error = $\frac {1}{m_test} \sum_{i=1}^{m_{test}}err\bigl(h_Θ(x_{test}^{(i)}),y_{test}^{(i)} \bigr)$
This gives us the proportion of the test data that was misclassified.

##Bias vs. Variance

**1. Degree of the Polynomial d and B/V**
![dbv.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/620849/16f2bdc4-0b77-3c93-eb60-679fe1183adc.png)

**2. Regularization and B/V**
![rbv.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/620849/84e9fd15-edc9-4db0-24cd-815bfa2eb316.png)

How to choose $\lambda$ :

1. Create a list of lambdas (i.e. λ∈{0,0.01,0.02,0.04,0.08,0.16,0.32,0.64,1.28,2.56,5.12,10.24});
2. Create a set of models with different degrees or any other variants.
3. Iterate through the λs and for each λ go through all the models to learn some Θ.
4. Compute the cross validation error using the learned Θ (computed with λ) on the $J_{CV}(Θ)$ without regularization or λ = 0.
5. Select the best combo that produces the lowest error on the cross validation set.
6. Using the best combo Θ and λ, apply it on $J_{test}(Θ)$​ to see if it has a good generalization of the problem.

**3. Learning Curves**

- Experiencing high bias:
![lbv1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/620849/87acd8a5-0f28-4de7-0000-b8573551a3cb.png)

- Experiencing high variance:
![lbv2.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/620849/e37a3bd4-9b03-fa07-458e-70baa432549f.png)

**4. What to Do Next to Improve**
Our decision process can be broken down as follows:

- **Getting more training examples:** Fixes high variance
- **Trying smaller sets of features:** Fixes high variance
- **Adding features:** Fixes high bias
- **Adding polynomial features:** Fixes high bias
- **Decreasing λ:** Fixes high bias
- **Increasing λ:** Fixes high variance

##Error Analysis

- Start with a simple algorithm, implement it quickly, and test it early on your cross validation data.
- Plot learning curves to decide if more data, more features, etc. are likely to help.
- Manually examine the errors on examples in the cross validation set and try to spot a trend where most of the errors were made.

**Choose Error Metrics:**
<img width="403" alt="Screen Shot 2020-05-24 at 23.30.01.png" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/620849/47c362ca-8292-2a1d-8c9b-3893be8a2725.png">
<img width="429" alt="Screen Shot 2020-05-24 at 23.22.23.png" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/620849/7edf0d1f-691f-9ba4-ccfd-ba18b4328465.png">
*Precision, Recall and F1 Score are good metrics paricularly when dealing with skewed data.

**Using large data sets usually helps!**
<img width="477" alt="Screen Shot 2020-05-24 at 23.32.11.png" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/620849/98df7a57-b2aa-ec60-f1bd-99c60eb7a169.png">
>It’s not who has the best algorithm that wins.
It’s who has the most data.