Lesson8: Ouliers：外れ値 まとめ Intro to Machine learning@Udacity

#講義内容
外れ値が発生する原因とその後の処理は
Sensor Malfunction/Data Entory errors・・・無視 
Freak Event・・・有効活用：詐欺検知 

無視する場合の外れ値検知と除去の方法
下記図の①～③を繰り返す。このアルゴリズムで外れ値除去をしていくのが基本らしい。
![outlier.jpg](https://qiita-image-store.s3.amazonaws.com/0/333380/f1590dbc-3835-aa0a-3fa5-22c2b03b68d1.jpeg)
残差上位何%を除去するのかは目的によって変わる。


#Mini-project
やること1：トレーニングデータに外れ値が入っている時と入っていない時で、どの程度影響が出るか見る。(～Video13)
やること2：外れ値について、それを除外するべきか、どの程度の重みづけをするべきか議論する。(Enron Data～)

##Video10-11

```python:outlier_removal_regression.py
### fill in a regression here!  Name the regression object reg so that
### the plotting code below works, and you can see what your regression looks like

from sklearn import linear_model
reg = linear_model.LinearRegression()
reg.fit(ages_train,net_worths_train)
print("slope:"+str(reg.coef_))
score = reg.score(ages_test,net_worths_test)
print("score:"+str(score))

try:
    plt.plot(ages, reg.predict(ages), color="blue")
except NameError:
    pass
plt.scatter(ages, net_worths)
plt.show()

```
＜出力＞
slope:[[5.07793064]]
score:0.8782624703664673
![regression_minipjt.png](https://qiita-image-store.s3.amazonaws.com/0/333380/744cd476-3810-05df-d5c3-78abcabef068.png)


##Video11-13
上記グラフの残差の大きい上位10%（外れ値）を除去する。
outlier_cleaner.pyを編集して、outlier_removal_regression.pyから呼び出す。

```python:outlier_cleaner.py
def outlierCleaner(predictions, ages, net_worths):
    """
        Clean away the 10% of points that have the largest
        residual errors (difference between the prediction
        and the actual net worth).

        Return a list of tuples named cleaned_data where 
        each tuple is of the form (age, net_worth, error).
    """
        
    cleaned_data = []
    ### your code goes here
 
    import numpy as np
    errors = (net_worths-predictions)**2
    print("b4 errors:"+str(errors))
    print("b4 length of errors:"+str(len(errors)))
    count = 0
    for a in range(len(errors)/10):#残差:errorsを計算してその上位10%をargmaxをつかって除去する
        print(a)
        delete_index = np.argmax(errors)#除去するindex取得
        print("del_index:"+str(delete_index))
        errors = np.delete(errors,delete_index)#errorsを除去
        net_worths = np.delete(net_worths,delete_index)#net_worthを除去
        ages = np.delete(ages,delete_index)#agesを除去
        count += 1
        print(a)
    print("after_age:"+str(len(ages)))
    print("after_net:"+str(len(net_worths)))
    print("after_e_return:"+str(len(errors)))
    print("#of counts:"+str(count))

    cleaned_data = zip(ages, net_worths, errors)#zip関数でreturn
    print(cleaned_data)
    
    return cleaned_data
```
＜出力＞
b4 errors:[[5.49314075e+02]
 [8.74018243e+02]
 [2.10985975e+03]
 [8.20719288e+02]
 [3.33955079e+02]
 [1.03855290e+03]
 [2.42056324e+03]
 [2.84417265e+02]
 [1.80003368e+01]
 [5.46937428e+03]
 [2.46759668e+03]
 [5.89142376e+01]
 [4.04019187e+02]
 [5.83557496e+01]
 [3.67475079e-01]
 [2.65277735e+02]
 [2.55094862e+03]
 [1.03095731e+05]
 [1.05157556e+02]
 [1.97114858e+03]
 [2.51995487e+02]
 [1.11814280e+03]
 [6.90257229e-03]
 [9.46007033e+02]
 [7.89176767e+03]
 [8.92980331e+02]
 [7.24517048e+01]
 [4.40611769e+03]
 [9.71086760e+00]
 [1.06184024e+02]
 [3.34024158e+02]
 [1.19767510e+03]
 [2.92282388e+02]
 [3.86459243e+04]
 [1.08860079e+02]
 [9.00882723e+02]
 [3.77853311e+02]
 [3.25111830e+02]
 [2.92291612e+01]
 [8.69401498e+02]
 [1.51435876e+03]
 [1.63318279e+03]
 [1.30770361e+03]
 [3.02007032e+02]
 [1.23458669e+02]
 [1.99696845e+02]
 [6.06380940e+03]
 [3.48342150e+03]
 [2.20404645e+03]
 [1.17524093e+02]
 [1.07091439e+02]
 [3.82978411e+04]
 [2.79221011e+03]
 [5.54457948e+01]
 [6.07042084e+02]
 [3.79240294e+02]
 [3.96035155e+01]
 [4.78430626e+02]
 [8.12717319e+02]
 [3.40671191e+04]
 [8.19519430e+02]
 [1.60229676e+01]
 [1.70592848e+02]
 [1.52832072e+03]
 [5.17541424e+04]
 [9.44050625e+02]
 [1.27925426e+03]
 [6.80744750e+02]
 [7.96091762e+03]
 [1.41883658e+03]
 [1.13239036e+02]
 [9.97765068e+00]
 [6.63262819e+01]
 [5.78083315e+02]
 [1.13020427e+01]
 [1.54316554e+00]
 [9.74816967e+02]
 [6.37560916e+02]
 [4.86604451e+02]
 [1.00895061e+03]
 [3.53625603e+04]
 [3.61055322e+02]
 [4.80006882e+03]
 [3.39458297e+03]
 [5.02835675e+03]
 [1.71321504e+03]
 [6.30294346e+03]
 [5.18463667e+00]
 [3.55164608e+04]
 [1.16282618e+00]]
b4 length of errors:90
0
del_index:17
0
1
del_index:63
1
2
del_index:32
2
3
del_index:49
3
4
del_index:84
4
5
del_index:76
5
6
del_index:56
6
7
del_index:63
7
8
del_index:23
8
after_age:81
after_net:81
after_e_return:81
"#of counts:9
[(57, 338.08951848957577, 549.3140750577214), (57, 344.21586775541203, 874.0182428677101), (34, 243.79287311628013, 2109.8597527631737), (64, 378.84578142088196, 820.7192884721906), (25, 133.88384977349668, 333.9550790312232), (53, 326.56693250553326, 1038.552900488104), (62, 389.24094136476344, 2420.5632413103494), (41, 250.26985299068266, 284.4172650141791), (21, 136.08924543132068, 18.000336751168476), (53, 368.29556369017877, 5469.3742756707215), (48, 318.62560238232396, 2467.596678695175), (24, 139.40479592465923, 58.914237640173425), (24, 126.98012845155395, 404.0191872254793), (30, 185.1870343522352, 58.35574955356093), (32, 187.0976049078975, 0.36747507902456666), (45, 270.00424941518054, 265.27773467025895), (52, 339.7693315999082, 2550.948623972569), (21, 142.1012008717193, 105.15755567445083), (60, 374.28347921780096, 1971.1485787074182), (43, 259.43540492859483, 251.9954868544548), (32, 221.1424445243783, 1118.1428047963063), (30, 177.46485912830926, 0.006902572286934023), (58, 350.4872262016239, 946.0070334262596), (51, 314.0672608610164, 892.9803313446159), (30, 186.05979756839332, 72.45170481414503), (55, 370.8748005714165, 4406.117693168428), (45, 256.83312700057957, 9.710867600126003), (30, 167.24337759331152, 106.18402388726031), (36, 189.73919689274533, 334.0241584649352), (37, 247.70089824206377, 1197.6751033249789), (23, 159.09869458305818, 292.2823876757843), (24, 157.51396036691344, 108.86007946425934), (20, 96.75392597329417, 900.8827228070461), (32, 207.14225143023492, 377.85331097527086), (35, 220.96845179419734, 325.1118304128782), (45, 259.1233005198794, 29.229161170114622), (29, 142.98439510684403, 869.4014977403845), (64, 389.1123455094963, 1514.3587625103983), (30, 217.96059712816052, 1633.182787550219), (47, 300.0349466519055, 1307.703614179883), (59, 342.1862790220314, 302.0070318105972), (39, 234.36051228130097, 123.45866858358036), (32, 201.83521553071006, 199.69684453999048), (61, 412.83425631014785, 6063.809399469276), (58, 378.75051644788397, 3483.421495731356), (58, 366.6772719000951, 2204.046445415731), (22, 147.76534866226712, 117.52409292084414), (27, 151.96564953554724, 107.09143947944409), (49, 326.86999044991904, 2792.210106134813), (33, 200.22792608821422, 55.44579479573997), (37, 237.7316793878959, 607.0420835974011), (21, 151.32065795784047, 379.2402942762437), (24, 140.78722458080426, 39.603515497039254), (34, 219.7327205009527, 478.4306259526763), (49, 302.53682046672367, 812.7173185706121), (31, 153.99862170798215, 819.5194302127975), (22, 140.9273656237994, 16.022967616278564), (27, 175.37526862807127, 170.5928480671572), (49, 313.12236579136925, 1528.3207196355636), (53, 325.0657524829757, 944.0506251766491), (38, 253.93805008750448, 1279.2542602083643), (44, 274.7300555052413, 680.7447503034904), (37, 250.76090254244718, 1418.8365777754177), (51, 294.8258675183866, 113.23903550041787), (38, 221.33012794213624, 9.977650679971), (48, 260.8065975224045, 66.32628187643527), (47, 287.91612500413447, 578.0833147752858), (21, 128.4847139765763, 11.302042692870963), (32, 186.46155999388083, 1.5431655389711973), (52, 320.4844739902693, 974.8169673090335), (32, 212.95377076201385, 637.5609156286199), (27, 140.25503627573087, 486.60445058737963), (43, 275.3250221931937, 1008.9506147300231), (50, 260.1050979182051, 361.05532249820845), (65, 424.5580423540415, 4800.068822528119), (46, 317.0578808948303, 3394.582966280816), (57, 385.5629753784311, 5028.356753786213), (65, 396.6665152430942, 1713.2150361203603), (63, 424.51073132931936, 6302.943463995066), (49, 271.75164334973937, 5.184636665643252), (29, 173.54835439167368, 1.1628261840632008)]
n_slope:[[6.36859481]]
n_score:0.983189455395532
![regression_minipjt_.png](https://qiita-image-store.s3.amazonaws.com/0/333380/18fa3e09-c69f-b8ac-0320-cdd9ed80e562.png)

##Enron Data

```python:enron_outliers.py
import pickle
import sys
import matplotlib.pyplot
sys.path.append("../tools/")
from feature_format import featureFormat, targetFeatureSplit


### read in data dictionary, convert to numpy array
data_dict = pickle.load( open("../final_project/final_project_dataset.pkl", "r") )
features = ["salary", "bonus"]
print(len(data_dict))#自分で加えた


data = featureFormat(data_dict, features)


### your code below
print(data)#自分で加えた
max_sal = 0
secnd_to_max_sal = 0
for point in data:
    salary = point[0]
    bonus = point[1]
    if max_sal <= salary:#自分で加えた
    	secnd_to_max_sal = max_sal#自分で加えた
    	max_sal = salary#自分で加えた
    matplotlib.pyplot.scatter( salary, bonus )
    
matplotlib.pyplot.xlabel("salary")
matplotlib.pyplot.ylabel("bonus")
matplotlib.pyplot.show()

print("1st:"+str(max_sal))#自分で加えた
print("2nd:"+str(secnd_to_max_sal))#自分で加えた

```
＜出力＞
146
[[3.6578800e+05 6.0000000e+05]
 [2.6710200e+05 1.2000000e+06]
 [1.7094100e+05 3.5000000e+05]
 [2.4329300e+05 1.5000000e+06]
 [2.6709300e+05 3.2500000e+05]
 [3.7044800e+05 2.6000000e+06]
 [1.9709100e+05 4.0000000e+05]
 [1.3072400e+05 0.0000000e+00]
 [2.8858900e+05 7.8875000e+05]
 [2.4854600e+05 8.5000000e+05]
 [2.5748600e+05 7.0000000e+05]
 [2.8854200e+05 1.2000000e+06]
 [2.5165400e+05 1.1000000e+06]
 [2.8855800e+05 2.5000000e+05]
 [6.3744000e+04 0.0000000e+00]
 [3.5709100e+05 8.5000000e+05]
 [2.7144200e+05 3.1000000e+06]
 [3.0411000e+05 2.0000000e+06]
 [1.8792200e+05 2.5000000e+05]
 [2.1362500e+05 1.0000000e+06]
 [2.4920100e+05 7.0000000e+05]
 [2.3133000e+05 7.0000000e+05]
 [1.8224500e+05 2.0000000e+05]
 [2.1178800e+05 1.7000000e+06]
 [2.2430500e+05 8.0000000e+05]
 [2.7374600e+05 1.0000000e+06]
 [3.3928800e+05 8.0000000e+06]
 [2.1658200e+05 0.0000000e+00]
 [2.1050000e+05 4.2500000e+05]
 [2.7288000e+05 7.5000000e+05]
 [4.7700000e+02 0.0000000e+00]
 [2.6907600e+05 6.5000000e+05]
 [4.2878000e+05 1.5000000e+06]
 [2.1184400e+05 2.0000000e+05]
 [2.0612100e+05 6.0000000e+05]
 [1.7424600e+05 0.0000000e+00]
 [5.1036400e+05 3.0000000e+06]
 [3.6503800e+05 1.1000000e+06]
 [3.6516300e+05 3.0000000e+06]
 [1.6277900e+05 1.0000000e+05]
 [2.3645700e+05 2.0000000e+05]
 [1.0723210e+06 7.0000000e+06]
 [2.6151600e+05 7.5000000e+05]
 [3.2907800e+05 7.5000000e+05]
 [1.8489900e+05 3.2500000e+05]
 [1.9200800e+05 5.0987000e+05]
 [2.6341300e+05 9.0000000e+05]
 [2.6266300e+05 7.0000000e+05]
 [3.7412500e+05 1.1500000e+06]
 [2.7860100e+05 1.3500000e+06]
 [1.9915700e+05 3.5000000e+05]
 [9.6840000e+04 0.0000000e+00]
 [8.0818000e+04 0.0000000e+00]
 [2.1399900e+05 5.2499990e+06]
 [2.1069200e+05 7.5000000e+05]
 [2.2209300e+05 0.0000000e+00]
 [4.4069800e+05 1.3000000e+06]
 [2.4018900e+05 1.2500000e+06]
 [4.2063600e+05 1.7500000e+06]
 [2.7510100e+05 4.0000000e+05]
 [3.1428800e+05 8.0000000e+05]
 [9.4941000e+04 0.0000000e+00]
 [2.3950200e+05 5.0000000e+05]
 [1.1112580e+06 5.6000000e+06]
 [6.6150000e+03 0.0000000e+00]
 [6.5503700e+05 3.0000000e+05]
 [4.0433800e+05 1.0000000e+06]
 [2.6704229e+07 9.7343619e+07]
 [2.5999600e+05 3.2500000e+05]
 [3.1754300e+05 4.5000000e+05]
 [2.0195500e+05 4.1750000e+06]
 [2.4814600e+05 6.0000000e+05]
 [7.6399000e+04 1.0000000e+05]
 [2.6278800e+05 1.0000000e+06]
 [2.6180900e+05 3.0000000e+05]
 [2.4801700e+05 5.0000000e+05]
 [2.2928400e+05 4.0000000e+05]
 [2.3194600e+05 8.5000000e+05]
 [2.2100300e+05 7.0000000e+04]
 [1.5840300e+05 0.0000000e+00]
 [2.5010000e+05 6.0000000e+05]
 [4.9237500e+05 8.0000000e+05]
 [1.0609320e+06 2.0000000e+06]
 [2.6187900e+05 1.0000000e+06]
 [2.3967100e+05 4.0000000e+05]
 [3.0458800e+05 2.5000000e+06]
 [3.0994600e+05 7.0000000e+05]
 [8.5274000e+04 0.0000000e+00]
 [2.4733800e+05 3.0000000e+05]
 [3.4948700e+05 0.0000000e+00]
 [3.3054600e+05 9.0000000e+05]
 [4.1518900e+05 1.0000000e+06]
 [2.6521400e+05 6.0000000e+05]
 [2.7860100e+05 8.0000000e+05]
 [2.7497500e+05 6.0000000e+05]]
1st:26704229.0
2nd:1111258.0
![outlier_Enron.png](https://qiita-image-store.s3.amazonaws.com/0/333380/cd97d4fd-9a96-0c7e-2b68-b86b6a18ef4c.png)

なんか明らかに外れ値が1匹右上に。。。
max_salとsecnd_to_max_salは、一番目と二番目に大きいsalaryの値を入れて、表示するようにした。一桁違う。Video15ではこの一番大きい値でfinal_project/enron61702insiderpay.pdf を検索して、この外れ値が誰のデータかを回答する。答えは「TOTAL」。合計値が入っていたんですねー。

次は「TOTAL」の除去。popを使う。pop加えただけ。

```python:enron_outliers.py
#!/usr/bin/python

import pickle
import sys
import matplotlib.pyplot
sys.path.append("../tools/")
from feature_format import featureFormat, targetFeatureSplit

### read in data dictionary, convert to numpy array
data_dict = pickle.load( open("../final_project/final_project_dataset.pkl", "r") )
features = ["salary", "bonus"]
#my code in quiz17 :Any More outliers?
print(len(data_dict))#自分で加えた
data_dict.pop("TOTAL")#自分で加えた
print(len(data_dict))#自分で加えた

data = featureFormat(data_dict, features)

### your code below
print(data)#自分で加えた
max_sal = 0
secnd_to_max_sal = 0
for point in data:
    salary = point[0]   
    bonus = point[1]
    if max_sal <= salary:#自分で加えた
    	secnd_to_max_sal = max_sal#自分で加えた
    	max_sal = salary#自分で加えた
    matplotlib.pyplot.scatter( salary, bonus )
    
matplotlib.pyplot.xlabel("salary")
matplotlib.pyplot.ylabel("bonus")
matplotlib.pyplot.show()

print("1st:"+str(max_sal))#自分で加えた
print("2nd:"+str(secnd_to_max_sal))#自分で加えた

```
＜出力＞
146
145
[[3.657880e+05 6.000000e+05]
 [2.671020e+05 1.200000e+06]
 [1.709410e+05 3.500000e+05]
 [2.432930e+05 1.500000e+06]
 [2.670930e+05 3.250000e+05]
 [3.704480e+05 2.600000e+06]
 [1.970910e+05 4.000000e+05]
 [1.307240e+05 0.000000e+00]
 [2.885890e+05 7.887500e+05]
 [2.485460e+05 8.500000e+05]
 [2.574860e+05 7.000000e+05]
 [2.885420e+05 1.200000e+06]
 [2.516540e+05 1.100000e+06]
 [2.885580e+05 2.500000e+05]
 [6.374400e+04 0.000000e+00]
 [3.570910e+05 8.500000e+05]
 [2.714420e+05 3.100000e+06]
 [3.041100e+05 2.000000e+06]
 [1.879220e+05 2.500000e+05]
 [2.136250e+05 1.000000e+06]
 [2.492010e+05 7.000000e+05]
 [2.313300e+05 7.000000e+05]
 [1.822450e+05 2.000000e+05]
 [2.117880e+05 1.700000e+06]
 [2.243050e+05 8.000000e+05]
 [2.737460e+05 1.000000e+06]
 [3.392880e+05 8.000000e+06]
 [2.165820e+05 0.000000e+00]
 [2.105000e+05 4.250000e+05]
 [2.728800e+05 7.500000e+05]
 [4.770000e+02 0.000000e+00]
 [2.690760e+05 6.500000e+05]
 [4.287800e+05 1.500000e+06]
 [2.118440e+05 2.000000e+05]
 [2.061210e+05 6.000000e+05]
 [1.742460e+05 0.000000e+00]
 [5.103640e+05 3.000000e+06]
 [3.650380e+05 1.100000e+06]
 [3.651630e+05 3.000000e+06]
 [1.627790e+05 1.000000e+05]
 [2.364570e+05 2.000000e+05]
 [1.072321e+06 7.000000e+06]
 [2.615160e+05 7.500000e+05]
 [3.290780e+05 7.500000e+05]
 [1.848990e+05 3.250000e+05]
 [1.920080e+05 5.098700e+05]
 [2.634130e+05 9.000000e+05]
 [2.626630e+05 7.000000e+05]
 [3.741250e+05 1.150000e+06]
 [2.786010e+05 1.350000e+06]
 [1.991570e+05 3.500000e+05]
 [9.684000e+04 0.000000e+00]
 [8.081800e+04 0.000000e+00]
 [2.139990e+05 5.249999e+06]
 [2.106920e+05 7.500000e+05]
 [2.220930e+05 0.000000e+00]
 [4.406980e+05 1.300000e+06]
 [2.401890e+05 1.250000e+06]
 [4.206360e+05 1.750000e+06]
 [2.751010e+05 4.000000e+05]
 [3.142880e+05 8.000000e+05]
 [9.494100e+04 0.000000e+00]
 [2.395020e+05 5.000000e+05]
 [1.111258e+06 5.600000e+06]
 [6.615000e+03 0.000000e+00]
 [6.550370e+05 3.000000e+05]
 [4.043380e+05 1.000000e+06]
 [2.599960e+05 3.250000e+05]
 [3.175430e+05 4.500000e+05]
 [2.019550e+05 4.175000e+06]
 [2.481460e+05 6.000000e+05]
 [7.639900e+04 1.000000e+05]
 [2.627880e+05 1.000000e+06]
 [2.618090e+05 3.000000e+05]
 [2.480170e+05 5.000000e+05]
 [2.292840e+05 4.000000e+05]
 [2.319460e+05 8.500000e+05]
 [2.210030e+05 7.000000e+04]
 [1.584030e+05 0.000000e+00]
 [2.501000e+05 6.000000e+05]
 [4.923750e+05 8.000000e+05]
 [1.060932e+06 2.000000e+06]
 [2.618790e+05 1.000000e+06]
 [2.396710e+05 4.000000e+05]
 [3.045880e+05 2.500000e+06]
 [3.099460e+05 7.000000e+05]
 [8.527400e+04 0.000000e+00]
 [2.473380e+05 3.000000e+05]
 [3.494870e+05 0.000000e+00]
 [3.305460e+05 9.000000e+05]
 [4.151890e+05 1.000000e+06]
 [2.652140e+05 6.000000e+05]
 [2.786010e+05 8.000000e+05]
 [2.749750e+05 6.000000e+05]]
![outlier_Enron_2.png](https://qiita-image-store.s3.amazonaws.com/0/333380/576b3d68-1cfc-b454-d749-7e937368aed9.png)￼
1st:1111258.0
2nd:1072321.0

１つ大きなのを除去後は、今までは外れ値でなかったものが外れ値になってますね。4つ？かな。これがVideo17の答え。

##Video18
"Two people made bonuses of at least 5 million dollars, and a salary of over 1 million dollars; in other words, they made out like bandits. What are the names associated with those points?"
と聞いてくるので、さっきの散布図の右上に近い2つのデータ値(1st, 2nd)を上記PDFから検索して、名前を記入。答えは
SKILLING JEFFREY K
LAY KENNETH L


#感想
外れ値除去の基本は散布図等で見える化してから、人の目で外れ値を見つける→除去が王道と言っていた。人はパターンを認識しやすいからね。


#調べた英単語
residual・・・adj, 残りの、残余の、剰余の、説明のつかない、除去できない
ex)residual error:残差

explicitly・・・adv,(言葉に出して)はっきりと、明白に
plausible・・・adj, もっともらしい、まことしやかな、口先のうまい
consistent・・・adj, 首尾一貫した、矛盾のない、(…と)矛盾しないで、調和して、言行一致した、信念のある
correlation・・・n, 相関させること、相関性、相互関係
ex)They have a correlation.それらは相関関係がある。

acquaint・・・v, …を)知らせる、告げる、
ex)I acquainted him with my intentions. 私の考えを彼に伝えた.

quirk・・・n, 癖、奇癖、偶然、運命のいたずら
ex)a quirk of fate. 運命の巡り合わせ