I am working on the file "Capstone+Narrative+analytics+and+experimentation.ipynb".
The first task is:
Data sets description
The purpose of this report is to analyze the evolution of the household electricity market and GDP per continent in the period 2000-2014. This will show how different economic cycles have affected both variables and the potential correlation between them. Furthermore, in a per continent basis, the both variables will be scrutinized to depict the similarities and differences between continents that will show the relationship between them. Three data sets have been imported from United Nations (from now onwards UN) database. The data sets contain global information for the time span 2000-2014 regarding:

Electricity consumption for the household market (GhW). In this dataset, the UN has estimated values for some countries (ex-colonies of the UK) based on their April-March consumption. This estimation has been done up until 2004 yearly electricity consumption was standardized following the natural year and not the fiscal year. Electricity consumption in Iraq in 2014 has a null value, as it was not reported due to the war that started in 2015.

GDP per country in current USD. From all the data sets available, measuring GDP the one measuring it in current USD has been chosen to avoid the impact of the different base years across countries when calculating the base of the deflation year and to avoid the use of different exchange rates across countries during the time span under analysis.

Population. In this case, the population has been converted into (Thousands) as a unit to make the standardized GDP and electricity consumption significant. This variable shows the net population at the end of the year considering births, deaths and declared migration.

The three of them are significant because, although different data sets containing this information exist, only this institution (UN) gathers them. They are consistent in terms of the methods used to gather this information and credible due to the institution that is providing them. The three variables can be compared against each other...
Can you write Python code for it?

#Cleanse Data Frame Electricity Consumption:
#Delete rows without countries, rename columns and take out those that are not used
#Zeros are identified to see their potential impact on the data set and further results
dfa= df[:3206]
dfb= dfa.rename(columns={'Country or Area': 'Country', 'Quantity': 'Household Consumption (GWh)'})
dfc = dfb.drop(["Quantity Footnotes", "Unit", "Commodity - Transaction"], axis=1)
dfc['Year'] = dfc['Year'].astype(int)
#dfc.loc[dfc['Household Consumption (GWh)'] == 0]

#Cleanse Data Frame GDP Current Prices (USD):
#Cleanse Data Frame Electricity Consumption:
#Delete rows without countries, rename columns and take out those that are not used
#Zeros are identified to see their potential impact on the data set and further results
dfd= df1[:3322]
dfe= dfd.rename(columns={'Country or Area': 'Country', 'Value': 'GDP Current Prices (USD)'})
dfg = dfe.drop('Value Footnotes', axis=1)
dfg['Year'] = dfg['Year'].astype(int)
#dfg.loc[dfg['GDP Current Prices (USD)'] == 0]

#Cleanse Data Frame Population:
#Take out rows without countries
#Rename columns
#Clean columns taking out those that are not used
dfh= df2[:3522]
dfi= dfh.rename(columns={'Country or Area': 'Country', 'Value': 'Population'})
dfj = dfi.drop('Value Footnotes', axis=1)
dfj['Year'] = dfj['Year'].astype(int)
#dfj.loc[dfj['Population'] == 0]

#Merge data into a single dataset
#Cleanse new dataset
result = dfc.merge(dfg, left_on=["Country","Year"], right_on=["Country","Year"], how='outer')
result = result.merge(dfj, left_on=["Country","Year"], right_on=["Country","Year"], how='outer')
result = result.dropna()

#Rescale & rename variables in the new data set to GDP Current Prices (Million USD) & Population (Thousands)
result['GDP Current Prices (Million USD)']=result['GDP Current Prices (USD)']/1000000
result['Population (Thousands)']=result['Population']/1000

#Drop redundant columns after merging the data sets
result = result.drop('GDP Current Prices (USD)',1)
result = result.drop('Population',1)
result= result.rename(columns={'GDP...


I am working on the file "Capstone.ipynb".
The first task is:
Analysis
In the case of the modified tiger problem it is too complicated to get more than 50% reliability on random chance.
Both algorithms fail without learning.
Can you write Python code for it?

# Load the 'qtiger_sim_no-learning' file from the default Q-Learning simulation
vs.plot_trials('qtiger_sim_default-learning.csv')

# Load the 'qtiger_sim_no-learning' file from the default Q-Learning simulation
vs.plot_trials('tiger_sim_default-learning.csv')


I am working on the file "Capstone.ipynb".
The first task is:
Analysis
So it looks like the number of states is so small that moving randomly is a valid strategy.
Can you write Python code for it?

# Load the 'qsim_default-learning' file from the default Q-Learning simulation
vs.plot_trials('qsim_default-learning.csv')

# Load the 'qsim_default-learning' file from the modified q-learning algorithm.
vs.plot_trials('sim_default-learning.csv')


I am working on the file "Capstone.ipynb".
The first task is:
Analysis
When learning is turned on the POMDPagent performs better than the vanilla q-learner.
The Q-learner learns a policy that does worse than with learning turned off.
Can you write Python code for it?

# Load the 'qtiger_sim_no-learning' file from the default Q-Learning simulation
vs.plot_trials('qtiger_sim_no-learning.csv')

# Load the 'tiger_sim_no-learning' file from the default Q-Learning simulation
vs.plot_trials('tiger_sim_no-learning.csv')


I am working on the file "Capstone.ipynb".
The first task is:
Basic POMDPAgent 4x1Simulation Results with no learning
Can you write Python code for it?

# Load the 'sim_no-learning' log file from the initial simulation results
vs.plot_trials('sim_no-learning.csv')


I am working on the file "Capstone.ipynb".
The first task is:
Basic q-learning agent 4x1Simulation with no learning
Can you write Python code for it?

# Load the 'qsim_no-learning' file from the default Q-Learning simulation
vs.plot_trials('qsim_no-learning.csv')


I am working on the file "Capstone/TrumpHillary.ipynb".
The first task is:
3. Finding the most representative terms for each candidate
This model cares more about how rare a term is, not just frequency.
Many terms used by a candidate will also be used by his/her rival, not giving much insight into the tweet's author. For example, the term "and" is used frequently in the dataset, but passing it through the model will result in a prediction probability of about 50/50 (not helpful at all); however, there are terms that are used much less frequently, but almost exclusively by certain candidates.  Those are the words (or groups of words) this model looks for in order to make an accurate prediction.
Extract terms with highest tf-idf scores
Can you write Python code for it?

top_h = {}
top_t = {}

for w in (bow_transformer.get_feature_names()[:len(bow_transformer.get_feature_names())]):
    p = nb_detector.predict_proba([w])[0][0]
    if len(w) > 3:
        if p > 0.5:
            top_h[w] = p
        elif p < 0.5:
            top_t[w] = p
    else:
        pass

top_t_10 = sorted(top_t, key=top_t.get, reverse=False)[:10]
top_h_10 = sorted(top_h, key=top_h.get, reverse=True)[:10]

dic = {}
for l in [top_t_10, top_h_10]:
    for key, values in (top_t.viewitems() | top_h.viewitems()):
        if key in l:
            dic[key] = values

top_df = pd.DataFrame(list(dic.viewitems()), columns=['word', 'hillary_prob'])
top_df['trump_prob'] = (1 - top_df['hillary_prob'])
top_df_t = top_df[:int((len(dic)/2))]
top_df_t = top_df_t[['word','trump_prob','hillary_prob']].sort_values(['trump_prob'], ascending=[True])
top_df_h = top_df[int((len(dic)/2)):].sort_values(['hillary_prob'], ascending=[True])


I am working on the file "Capstone/TrumpHillary.ipynb".
The first task is:
5. Enter your tweet!

Write your tweet in the cell below
Run the cell (or the notebook, if you haven't yet)
Check out the results!
Can you write Python code for it?

your_tweet = "ENTER YOUR TWEET HERE, INSIDE THE QUOTES"

# ---- this part stays the same----
if your_tweet == "ENTER YOUR TWEET HERE, INSIDE THE QUOTES":
    pass
else:
    print("Tweet #1:", "'",your_tweet, "'", ' \n \n', "I'm about %.0f%%" % (100 * max(nb_detector.predict_proba([your_tweet])[0])), "sure this was tweeted by", nb_detector.predict([your_tweet])[0])


I am working on the file "Capstone/TrumpHillary.ipynb".
The first task is:
Create new dataframe for prediction model with candidate name (as label) and tweets (as message):
Can you write Python code for it?

messages = df[['label','message']]

print(messages[:5])

def split_into_tokens(message):
    message = message  # convert bytes into proper unicode
    return TextBlob(message).words

messages.message.head()

messages.message.head().apply(split_into_tokens)

def split_into_lemmas(message):
    message = message.lower()
    words = TextBlob(message).words
    # for each word, take its "base form" = lemma
    return [word.lemma for word in words]

messages.message.head().apply(split_into_lemmas)

bow_transformer = CountVectorizer(analyzer=split_into_lemmas).fit(messages['message'])
print(len(bow_transformer.vocabulary_))
print(bow_transformer.get_feature_names()[:5])

messages_bow = bow_transformer.transform(messages['message'])
print('sparse matrix shape:', messages_bow.shape)
print('number of non-zeros:', messages_bow.nnz)
print('sparsity: %.2f%%' % (100.0 * messages_bow.nnz / (messages_bow.shape[0] * messages_bow.shape[1])))

tfidf_transformer = TfidfTransformer().fit(messages_bow)

print (tfidf_transformer.idf_[bow_transformer.vocabulary_['the']])
print (tfidf_transformer.idf_[bow_transformer.vocabulary_['hannity']])

messages_tfidf = tfidf_transformer.transform(messages_bow)
print(messages_tfidf.shape)

%time spam_detector = MultinomialNB().fit(messages_tfidf, messages['label'])

all_predictions = spam_detector.predict(messages_tfidf)

tr_acc = accuracy_score(messages['label'], all_predictions)
print("Accuracy on training set:  %.2f%%" % (100 * tr_acc))

fig, ax = plt.subplots(figsize=(3.5,2.5))
sns.heatmap(confusion_matrix(messages['label'], all_predictions), annot=True, linewidths=.5, ax=ax, cmap="Blues", fmt="d").set(xlabel='Predicted Value', ylabel='Expected Value')
sns.plt.title('Training Set Confusion Matrix')

print(classification_report(messages['label'], all_predictions))

msg_train, msg_test, label_train, label_test = \
    train_test_split(messages['message'], messages['label'], test_size=0.2, random_state=1)

print(len(msg_train), len(msg_test), len(msg_train) + len(msg_test))


I am working on the file "Capstone/TrumpHillary.ipynb".
The first task is:
Extract mentions from tweets
Can you write Python code for it?

def all_mentions(tw):
    test2 = re.findall('(\@[A-Za-z_]+)', tw)
    if test2:
        return test2
    else:
        return ""

df['top_mentions'] = df['text'].apply(lambda x: all_mentions(x))

mention_list_trump = []
mention_list_clinton = []
for n in range(len(df['top_mentions'])):
    if df['handle'][n] == 'realDonaldTrump':
        mention_list_trump += df['top_mentions'][n]
    elif df['handle'][n] == 'HillaryClinton':
        mention_list_clinton += df['top_mentions'][n]


I am working on the file "Capstone/TrumpHillary.ipynb".
The first task is:
It seems Trump was much more fond of hashtags than Hillary, using his favorite hashtag (#Trump) almost 9x more than she used her favorite (#DemsInPhilly).
Can you write Python code for it?

df['length_no_url'] = df['text']
df['length_no_url'] = df['length_no_url'].apply(lambda x: len(x.lower().split('http')[0]))
df['message'] = df['text'].apply(lambda x: x.lower().split('http')[0])

def candidate_code(x):
    if x == 'HillaryClinton':
        return 'Hillary'
    elif x == 'realDonaldTrump':
        return 'Trump'
    else:
        return ''

df['label'] = df['handle'].apply(lambda x: candidate_code(x))


I am working on the file "Capstone/TrumpHillary.ipynb".
The first task is:
Seems like one thing both candidates had in common was their frequent mention of Trump. Surprinsingly, while Trump was Hillary's 2nd most mentioned account, she was not on Trump's list.
Extract hashtags from tweets
Can you write Python code for it?

def get_hashtags(tw):
    test3 = re.findall('(\#[A-Za-z_]+)', tw)
    if test3:
        return test3
    else:
        return ""

df['top_hashtags'] = df['text'].apply(lambda x: get_hashtags(x))

hashtags_list_trump = []
hashtags_list_clinton = []
for n in range(len(df['top_hashtags'])):
    if df['handle'][n] == 'realDonaldTrump':
        hashtags_list_trump += df['top_hashtags'][n]
    elif df['handle'][n] == 'HillaryClinton':
        hashtags_list_clinton += df['top_hashtags'][n]


I am working on the file "Capstone/TrumpHillary.ipynb".
The first task is:
Top 10 terms with highest probability of indicating a Hillary tweet
Can you write Python code for it?

sns.set_context({"figure.figsize": (11, 7)})
top_df_h.plot(kind='barh', stacked=True, color=["#08306B","#E91D0E"]).legend(loc='center left', bbox_to_anchor=(1, 0.5))
plt.yticks(range(len(top_df_h['word'])), list(top_df_h['word']))
plt.title('Top 10 terms with highest probability of indicating a Hillary tweet')
plt.xlabel('Probability')


I am working on the file "Capstone/TrumpHillary.ipynb".
The first task is:
Training on Benchmark Model
Can you write Python code for it?

pipeline = Pipeline([
    ('bow', CountVectorizer(analyzer=split_into_lemmas)),  # strings to token integer counts
    ('tfidf', TfidfTransformer()),# integer counts to weighted TF-IDF scores
    ('to_dense', DenseTransformer()),
    ('classifier',GaussianNB()),  # train on TF-IDF vectors w/ Naive Bayes classifier
])

scores = cross_val_score(pipeline,  # steps to convert raw messages into models
                         msg_train,  # training data
                         label_train,  # training labels
                         cv=10,  # split data randomly into 10 parts: 9 for training, 1 for scoring
                         scoring='accuracy',  # which scoring metric?
                         n_jobs=-1,  # -1 = use all cores = faster
                         )
print(scores)

print('Mean score:', scores.mean(), '\n')
print('Stdev:', scores.std())

params = {
    'tfidf__use_idf': (True, False),
    'bow__analyzer': (split_into_lemmas, split_into_tokens),
}

grid = GridSearchCV(
    pipeline,  # pipeline from above
    params,  # parameters to tune via cross validation
    refit=True,  # fit using all available data at the end, on the best found param combination
    n_jobs=-1,  # number of cores to use for parallelization; -1 for "all cores"
    scoring='accuracy',  # what score are we optimizing?
    cv=StratifiedKFold(label_train, n_folds=5),  # what type of cross validation to use
)

%time nb_detector = grid.fit(msg_train, label_train)
print(nb_detector.grid_scores_)

predictions = nb_detector.predict(msg_test)
print(classification_report(label_test, predictions))

fig, ax = plt.subplots(figsize=(3.5,2.5))
sns.heatmap(confusion_matrix(label_test, predictions), annot=True, linewidths=.5, ax=ax, cmap="Blues", fmt="d").set(xlabel='Predicted Value', ylabel='Expected Value')
sns.plt.title('Test Set Confusion Matrix')

print("Accuracy on test set:  %.2f%%" % (100 * (nb_detector.grid_scores_[0][1])))


I am working on the file "CapstonePoject/capstone_report.ipynb".
The first task is:
2.2算法与方法
先从特征开始讲起把：
从上述部分可以看出，仅通过使用客户数量数据或仅仅是促销是不能能预测销售的。销售受到每个属性的影响。为了对销售进行预测，首先我们将摆脱“客户”功能和“销售”功能中的异常值，因为数字过高或过低可能会出现异常情况。
这样的数据会影响模型的精度。处理异常值后，我们可以开始预处理数据。这包括摆脱Null值，encoding一些特征，如StoreType，StateHoliday等。
处理Missing Data是根据特征的现实意义进行填补和丢弃的，比如除每个星期天外，我都默认商店是处于“open”状态的，这比较符合我们的生活常识
如前所述，日期的转换对于预测而言是非常重要的，这也是视觉化中证明的，因为日期，月份和日期的销售变化很大。
我们也可以处理竞争的细节，因为它肯定会影响到真正影响销售的客户数量。促销可能有助于让客户回来，这就是为什么我们需要对所有Promo列进行编码的原因。一旦我们对数据进行了预处理，我们就可以使用cross_validation.train_test_split方法进行拆分。
该方法随机洗牌数据并返回两套训练和测试。可以定义测试的大小。然后，训练集用于训练几个模型 -
模型上用到的相关方法：

我实验了大概好几种，分开来看，分别是 DecisionTree回归 ，GradientBoost回归，KNeighborsClassifier我进行了分别的测验，然后根据情况又进行了Ensemable，但最后还是别人造的轮子 xgboost更好用

-1. DecisionTree回归 - 该模型的目标是创建一个通过学习从数据特征推断的简单决策规则来预测目标变量的值的模型。 DecisionTree采用特征，并使用ifnd-else决策规则来获取目标变量的方法。
-2. Kneighbour回归 - 最近邻方法的原理是找到与新点最近距离最近的预定义数量的训练样本，并从它们预测标签。 K邻居回归基于每个查询点的k个最近邻居实现学习，其中k是用户指定的整数值。 Kneighbour的回归实例 -
-3. GradientBoost回归 - 渐变增强回归是推广到任意可微分损失函数的推广。
梯度增强产生一个预测模型，它以弱预测模型（通常是决策树）的形式组成。它以像其他增强方法一样的阶段性方式构建模型，并且通过允许优化任意不同的损失函数来推广它们。
由于以下属性，这些模型已被选择用于此问题 - •考虑到影响销售的功能，数据可以轻松地分解为使用if-then-else决策规则对输入特征进行决策。这需要可以轻松完成的数据准备，所以我们不需要担心这里的主要缺点。
虽然决策树可能不稳定，但处理分类数据和数值也是非常好的。
因此，我们的数据集中的多种类型的功能将不会成为问题。我们可以通过查看模型的测试分数来确定是否对我们的数据不稳定，然后决定是否可以使用。
•我们的数据集有大量的数据点。 Kneighbours使用强力执行快速计算，这有助于我们降低模型的成本。另一方面，仅在某些输入特征不是连续值的情况下才是有效的。我们可以通过测试分数找出影响模型准确性的程度。
•GradientBoost是一个缓慢的模型，因此模型的成本将会很高。另一方面，它通过优化任意不可分解的损失函数使用不同的方法。如果我们无法通过任何其他模型获得良好的成绩，我们可以通过这种方法获得良好的成绩。
2.3基准
考虑到我们将要尝试3个以上的模型，我期望一个模型具有良好的RMSE分数，注意在RMSE的情况下越低越好，0是完美的分数。
对销售有一个好主意，商店经理需要对模型的准确性有一定的信念。
可以接受一定的错误率，但如果误差几乎达到五分之一，那么该模型是不错的。 有经验的店经理将能够预测那么多错误率。 如果模型没有正确地预测至少五分之一的数据，则数据需要更多的处理，并且所选择的模型需要被优化或改变。 因此，RMSE的基准应为0.20
三、方法
首先我们首先处理异常值。 检测数据中的异常值在任何数据预处理步骤中都非常重要的分析。
异常值的存在常常会使结果不好
考虑到这些数据点。 有很多“规则“是什么构成数据集中的异常值？
使用Tukey's方法来识别异常值：离群值是计算为四分位数范围（IQR）的1.5倍。 数据点具有超出IQR之外的异常值的功能该功能被认为是异常的。
因此，我写了一个代码，用可视化的方法辅助，发现客户和中的异常值
销售，然后看到哪些异常值在这两个特征中是常见的。然后常见的异常值被丢弃。
如前所述，数据需要适量的处理。直接使用一些具有数值的功能 -...
Can you write Python code for it?

display(Image('C:/Users/Administrator/Desktop/report/' + "19.png", width=1000))


I am working on the file "CapstonePoject/capstone_report.ipynb".
The first task is:
5.2思考
项目的分析最初是项目的一个有意义的部分，因为它能够告诉我们哪个功能会影响销售，几乎和DecisionTree回归函数的feature_importance属性一样。
由于数据未进行预处理，因此在数据可视化方面存在困难。
数据中有很多NaN值在几乎每个阶段都在降低产出的质量。
我预计最终的模型要花费更少的时间，但训练时间不是以小时计算的话根本不重要。
优化模型是一个挑战，因为处理的索引错误（如代码中所述）。 现在这个模型可以用于预测销售，即使有更多的商店或不同的业务来了，如果有类似的功能，那么这个模型可以用于适当的预测
如何将模型应用在商业上
理想的方法是（需要检查与业务，看看我们是否需要包括他们的任何一个实施以下方法的制约因素）

与业务核实我们需要多长时间才能从生产中刷新该模型状态。
开发端到端管道，采用来自所有1115家商店的综合销售数据做数据预处理，特征工程，然后训练模型（使用CV验证方法），并根据刷新频率输出预测
管道应能够持续整合新数据（每天/每周）和帮助预测在包括新数据在内的训练模型时，尽可能准确。
应将报告发送给每个店主，以了解他未来6周的具体店铺预测

5.3优化
项目的分析最初是项目的一个有意义的部分，因为它能够告诉我们哪个功能会影响销售，几乎和DecisionTree回归函数的feature_importance属性一样。
优化模型是一个挑战，因为处理的索引错误（如代码中所述）。 现在这个模型可以用于预测销售，即使有更多的商店或不同的业务来了，如果有类似的功能，那么这个模型可以用于适当的预测
可以参见我用xgboost做出的结果，因为主要是去调参和之前需要实验的一些特征工程，所以此处不会详细分析，具体内容可以去看看代码。此处我会贴一些最后的截图。第一次不怎么会调参，第二次看了一些博客后有了显著的提升。下面是最终结果
Can you write Python code for it?

display(Image('C:/Users/Administrator/Desktop/report/' + "21.png", width=1000))

display(Image('C:/Users/Administrator/Desktop/report/' + "22.png", width=1000))


I am working on the file "CapstonePoject/capstone_report.ipynb".
The first task is:
如图所见，商店，DayOfWeek和Date是最重要的feature，似乎在销售上有最大的差异。 这个预测是正确的。 另一方面，假期和促销也似乎有很大的差异，但这些特征已经被降低了
这里有我后来用xgboost实现了低于0.1的error的时候进行的特征的分析，和上面的结果有较显著的区别，如下图所示。尤其是排名3-10的特征，在xgboost运算后明显有了更大的权重，也就是说，xgboost更具有发现特征对于预测结果的能力，不像单模型那样过于简单，靠常识也能知道过于依赖某个特征可信度会有比较大风险。所以最后结果也自然有了区别
Can you write Python code for it?

display(Image('C:/Users/Administrator/Desktop/report/' + "20.png", width=1000))


I am working on the file "CapstonePoject/capstone_report.ipynb".
The first task is:
接下来看看学校放假对于销售额和用户量的影响
Can you write Python code for it?

display(Image('C:/Users/Administrator/Desktop/report/' + "11.png", width=1000))


I am working on the file "CapstonePoject/capstone_report.ipynb".
The first task is:
然后再看一个比较关键的特征，一般竞争者们之间的距离和销售额的关系。形状比较像正态分布
Can you write Python code for it?

display(Image('C:/Users/Administrator/Desktop/report/' + "16.png", width=1000))


I am working on the file "CapstonePoject/capstone_report.ipynb".
The first task is:
然后可以再分析下客户量和销售额的关系，下图基本能大致说清楚客户数量与客单价的联系了
Can you write Python code for it?

display(Image('C:/Users/Administrator/Desktop/report/' + "12.png", width=1000))


I am working on the file "CapstonePoject/capstone_report.ipynb".
The first task is:
由上图能非常明显的看出，我们的推理得到了验证：“销售额的变化基本是和用户数量的变化是一致的，百分比变化几乎完全一样”
现在来看看促销对销售额与顾客数量是否明显的影响
Can you write Python code for it?

display(Image('C:/Users/Administrator/Desktop/report/' + "8.png", width=1000))


I am working on the file "CapstonePoject/capstone_report.ipynb".
The first task is:
通过将时间数据进行清理，再探索一下销售额与时间（月）的关系，如下图所示展示了平均销售额与月份的关系以及百分比改变状况。
Can you write Python code for it?

display(Image('C:/Users/Administrator/Desktop/report/' + "4.png", width=1000))


I am working on the file "CapstonePoject/capstone_report.ipynb".
The first task is:
长期促销销售额对于用户数量的影响，如下图
Can you write Python code for it?

display(Image('C:/Users/Administrator/Desktop/report/' + "15.png", width=1000))


I am working on the file "CapstoneSEER/SEER Data Analysis Phase 1- Ingestion.ipynb".
The first task is:
The LoadSeerData class opens a connection to the database
Can you write Python code for it?

class LoadSeerData(MasterSeer):

    def __init__(self, path=r'./data', reload=True, testMode=False, verbose=True, batch=10000):

        # user supplied parameters
        self.reload = reload        # deletes and recreates db before start of loading data.
        self.testMode = testMode    # import one file, 100 records and return
        self.verbose = verbose      # prints status messages
        self.batchSize = batch      # number of rows to commit to db in one transation

        if type(path) != str:
            raise TypeError('path must be a string')

        if path[-1] != '/':
            path += '/'            # if path does not end with a backslash, add one

        self.path = path

        # open connection to the database
        super().__init__(path, reload, testMode, verbose)
        self.db_conn, self.db_cur = super().init_database(self.reload)

    def __del__(self):
        self.db_conn.close()


I am working on the file "CapstoneSEER/SEER Data Analysis Phase 1- Ingestion.ipynb".
The first task is:
The create_table function creates and stores a table using the fields read from the data dictionary
Can you write Python code for it?

    def create_table(self, tblName):
        ''' Create the table from the fields read from data dictionary and stored in self.dataDictInfo
            Make list comma delimited
        '''

        fieldList = self.dfDataDict.FIELD_NAME
        delimList = ','.join(map(str, fieldList))

        # create the table
        # SECURITY - Not subject to code injection even if Data Dictionary was
        # hacked since create table is the command specified.
        #            Not running any SELECT statements to hack.  Buffer
        #            overflow problems mitigated with checks importing
        #            dictionary.
        self.db_conn.execute('CREATE TABLE {0:s}(SOURCE,'.format(tblName) + delimList + ')')

    def __str__(self, **kwargs):
        pass


I am working on the file "CapstoneSEER/SEER Data Analysis Phase 1- Ingestion.ipynb".
The first task is:
The load_data function loads the SEER raw data into an sqlite3 database. The fname parameter is the path to the SEER data which supports a specific file or a wildcard filename to import all the data in one call.
Can you write Python code for it?

    def load_data(self, fname=r'incidence\yr1973_2012.seer9\breast.txt'):

        try:
            self.dfDataDict = super().load_data_dictionary()
        except Exception as e:
            print('ERROR loading data dictionary.')
            raise(e)

        if len(self.dfDataDict) == 0:
            raise('Bad Data Dictionary Data')

        timeStart = time.perf_counter()

        totRows = 0
        for fileName in glob.glob(self.path + fname):
            totRows += self.load_one_file(fileName)

        if self.verbose:
            print('Loading Data completed.\n Rows Imported: {0:d} in {1:.1f} seconds.\n Loaded {2:.1f} per sec.'.format
                  (totRows, time.perf_counter() - timeStart, (totRows / (time.perf_counter() - timeStart))))


I am working on the file "CapstoneSEER/SEER Data Analysis Phase 1- Ingestion.ipynb".
The first task is:
The load_one_file function performs the sql inserts in the database. The fname parameter is the name of the individual SEER dataflie to import. The function then returns the number of rows inserted. The function also assigns column names.
Can you write Python code for it?

    def load_one_file(self, fname):

        if self.verbose:
            print('\nStart Loading Data: {0}'.format(fname))

        # Need to get the name of the SEER text file so we can store it into
        # the SOURCE field.
        fileSource = os.path.basename(fname)
        fileSource = os.path.splitext(fileSource)[0]

        try:
            self.db_conn.execute('DROP TABLE {0}'.format(fileSource))
        except:
            pass

        colInfo = []  # hold start, stop byte offset for each field, used by read_fwf
        for off, len in zip(self.dfDataDict.OFFSET, self.dfDataDict.LENGTH):
            colInfo.append((off-1, off-1+len))

        if self.verbose:
            print('Starting read of raw data.')

        dfData = pd.read_fwf(fname, colspecs = colInfo, header=None) #, nrows=100000)
        #, nrows = self.batchSize, skiprows=totRows)

        # assign column names
        dfData.columns = self.dfDataDict.FIELD_NAME

        if self.verbose:
            print('Starting load of data to database.')

        sql.to_sql(dfData, name=fileSource, con=self.db_conn, index=False, if_exists='append',
                   chunksize=self.batchSize)

        if self.verbose:
            print('\n - Loading completed. Rows Imported: {0:d}'.format(dfData.shape[0]))

        return dfData.shape[0] # number of rows


I am working on the file "CapstoneSEER/SEER Data Analysis Phase 2- Wrangling.ipynb".
The first task is:
Last, we use the one-hot-encode procedure to return a new dataframe with vectorized data. We use this function only when we want to call standard distance metrics (such as k-nearest neighbors) for catagorical variables (e.g. race). See this link to learn more: http://stackoverflow.com/questions/17469835/one-hot-encoding-for-machine-learning
Can you write Python code for it?

     def one_hot_data(self, data, cols):

        # check to only encode columns that are in the data
        col_to_process = [c for c in cols if c in data]
        return pd.get_dummies(data, columns = col_to_process,  prefix = col_to_process)


I am working on the file "CapstoneSEER/SEER Data Analysis Phase 2- Wrangling.ipynb".
The first task is:
The initi_databse function creates a database connection and cursor to sqlite3. If the reload parameter is true, the function deletes all the data from and creates a new empty database.
Can you write Python code for it?

    def init_database(self, reload):
        try:
            if reload:
                os.remove(self.path + self.DB_NAME)
        except:
            pass

        try:
            #initialize database
            self.db_conn = sqlite3.connect(self.path + self.DB_NAME)
            self.db_cur = self.db_conn.cursor()

            if self.verbose:
                print('Database initialized')

            return self.db_conn, self.db_cur

        except Exception as e:
            print('ERROR connecting to the database: ')
            return None, None


I am working on the file "CapstoneSEER/SEER Data Analysis Phase 2- Wrangling.ipynb".
The first task is:
The next function loads the data dictionary describing the raw SEEER data and returns a dataframe of the data dictionary.
Can you write Python code for it?

    def load_data_dictionary(self, fname = r'SeerDataDict.txt'):

        REGEX_DD = '\$char([0-9]+).'

        t0 = time.perf_counter()

        if self.verbose:
            print('\nStart Load of Data Dictionary')

        # read our custom tab delimited data dictionary
        df = pd.read_csv(self.path + fname, delimiter='\t')

        # drop all rows where IMPORT_0_1 is a zero.
        df = df[df.IMPORT_0_1 > 0]

        # pre-compile regex to improve performance in loop
        reCompiled = re.compile(REGEX_DD)
        flen = []       # list to hold parsed field lengths

        # add length column
        for row in df.TYPE:
            fields = reCompiled.match(row)
            if fields:
                x = int(fields.groups()[0])
                flen.append(x)

        # check to make sure we read the correct amount of field lengths
        if len(flen) != len(df):
            print('ERROR reading field lengths')
            return None

        # add length column to dataframe
        df['LENGTH'] = flen

        if self.verbose:
            print('Data Dictionary loaded in {0:5.4f} sec.'.format(time.perf_counter() - t0),
                  flush=True)

        return df


I am working on the file "CapstoneSEER/SEER Data Analysis Phase 3- Data Exploration.ipynb".
The first task is:
Next we moved to creating survival charts using a larger sample size. We created a class with a "plot_survival" function. For the graph we picked variables that the scientific literature finds significant-- Stage, ER status, PR status, age, and radiation treatment. The second plot compares the frequency of survival for censored and non-censored patients.
Can you write Python code for it?

class ExploreSeer(MasterSeer):

    def __init__(self, path=r'./data/', testMode=False, verbose=True, sample_size=5000):

        # user supplied parameters
        self.testMode = testMode        # import one file, 500 records and return
        self.verbose = verbose          # prints status messages
        self.sample_size = sample_size  # number of rows to pull for testing

        if type(path) != str:
            raise TypeError('path must be a string')

        if path[-1] != '/':
            path += '/'            # if path does not end with a backslash, add one

        self.path = path

        # open connection to the database
        super().__init__(path, False, verbose=verbose)
        self.db_conn, self.db_cur = super().init_database(False)

    def __del__(self):
        super().__del__()

    def plot_survival(self):

        df = super().load_data(col  = ['YR_BRTH','AGE_DX','LATERAL','RADIATN','HISTREC','ERSTATUS',
                                       'PRSTATUS','BEHANAL','HST_STGA','NUMPRIMS', 'SRV_TIME_MON',
                                       'SRV_TIME_MON_PA', 'DTH_CLASS', 'O_DTH_CLASS', 'STAT_REC'],
                cond = 'SRV_TIME_MON < 1000 AND HST_STGA < 8 AND DTH_CLASS < 9 AND ERSTATUS < 4 AND PRSTATUS < 4',
                               sample_size = 100000)

        kmf = KaplanMeierFitter()

        try:
            df.RADIATN = df.RADIATN.replace(7, 0)
            df = df[df.RADIATN < 7]
        except Exception as err:
            pass

        # 0-negative, 1-borderline,, 2-positive
        df = df[df.ERSTATUS != 4]
        df = df[df.ERSTATUS != 9]
        df.ERSTATUS = df.ERSTATUS.replace(2, 0)
        df.ERSTATUS = df.ERSTATUS.replace(1, 2)
        df.ERSTATUS = df.ERSTATUS.replace(3, 1)

        # 0-negative, 1-borderline,, 2-positive
        df = df[df.PRSTATUS != 4]
        df = df[df.PRSTATUS != 9]
        df.PRSTATUS = df.PRSTATUS.replace(2, 0)
        df.PRSTATUS = df.PRSTATUS.replace(1, 2)
        df.PRSTATUS = df.PRSTATUS.replace(3, 1)

        rad =...


I am working on the file "CapstoneSEER/SEER Data Analysis Phase 3- Data Exploration.ipynb".
The first task is:
To begin exploring the data we took a sample of the SEER data, defined the features and dependent variable, printed the top few lines to ensure a successful data ingest, and ran descriptive statistics
Can you write Python code for it?

FEATURES  = [
    "Birth Year",
    "Age at Diagnosis",
    "Race",
    "Origin",
    "laterality",
    "Radiation",
    "Histrec",
    "ER Status",
    "PR Status",
    "Behanal",
    "Stage",
    "Numprimes",
    "Survival Time",
    "Bucket"
]

LABEL_MAP = {
    0: "< 60 Months",
    1: "60 < months > 120",
    2: "> 120 months",
}

# Read the data into a DataFrame
df = pd.read_csv("clean1.csv", sep=',' , header=0, names=FEATURES)

# Convert class labels into text
for k,v in LABEL_MAP.items():
    df.ix[df.Bucket == k, 'Bucket'] = v

print(df.head(n=5))

df.describe()


I am working on the file "CapstoneSEER/SEER Data Analysis Phase 3- Data Exploration.ipynb".
The first task is:
We used a histogram to see the distribution of survival time in months
Can you write Python code for it?

fig = plt.figure()
ax = fig.add_subplot(111)
ax.hist(df['Survival Time'], bins = 10, range = (df['Survival Time'].min(),df['Survival Time'].max()))
plt.title('Survival Time Distribution')
plt.xlabel('Survival Time')
plt.ylabel('Months')
plt.show()


I am working on the file "CarND-Advanced-Lane-Lines/src/.ipynb_checkpoints/camera_calibration-checkpoint.ipynb".
The first task is:
Camera Calibration with OpenCV
Run the code in the cell below to extract object points and image points for camera calibration.
Can you write Python code for it?

import numpy as np
import cv2
import glob
import matplotlib.pyplot as plt
%matplotlib qt

# prepare object points, like (0,0,0), (1,0,0), (2,0,0) ....,(6,5,0)
objp = np.zeros((6*8,3), np.float32)
objp[:,:2] = np.mgrid[0:8, 0:6].T.reshape(-1,2)

# Arrays to store object points and image points from all the images.
objpoints = [] # 3d points in real world space
imgpoints = [] # 2d points in image plane.

# Make a list of calibration images
images = glob.glob('../camera_cal/calibration*.jpg')

# Step through the list and search for chessboard corners
for idx, fname in enumerate(images):
    img = cv2.imread(fname)
    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

    # Find the chessboard corners
    ret, corners = cv2.findChessboardCorners(gray, (8,6), None)

    # If found, add object points, image points
    if ret == True:
        objpoints.append(objp)
        imgpoints.append(corners)

        # Draw and display the corners
        cv2.drawChessboardCorners(img, (8,6), corners, ret)
        #write_name = 'corners_found'+str(idx)+'.jpg'
        #cv2.imwrite(write_name, img)
        cv2.imshow('img', img)
        cv2.waitKey(500)

cv2.destroyAllWindows()


I am working on the file "CarND-Advanced-Lane-Lines/src/.ipynb_checkpoints/camera_calibration-checkpoint.ipynb".
The first task is:
If the above cell ran sucessfully, you should now have objpoints and imgpoints needed for camera calibration.  Run the cell below to calibrate, calculate distortion coefficients, and test undistortion on an image!
Can you write Python code for it?

import pickle
%matplotlib inline

# Test undistortion on an image
img = cv2.imread('calibration_wide/test_image.jpg')
img_size = (img.shape[1], img.shape[0])

# Do camera calibration given object points and image points
ret, mtx, dist, rvecs, tvecs = cv2.calibrateCamera(objpoints, imgpoints, img_size,None,None)

dst = cv2.undistort(img, mtx, dist, None, mtx)
cv2.imwrite('calibration_wide/test_undist.jpg',dst)

# Save the camera calibration result for later use (we won't worry about rvecs / tvecs)
dist_pickle = {}
dist_pickle["mtx"] = mtx
dist_pickle["dist"] = dist
pickle.dump( dist_pickle, open( "calibration_wide/wide_dist_pickle.p", "wb" ) )
#dst = cv2.cvtColor(dst, cv2.COLOR_BGR2RGB)
# Visualize undistortion
f, (ax1, ax2) = plt.subplots(1, 2, figsize=(20,10))
ax1.imshow(img)
ax1.set_title('Original Image', fontsize=30)
ax2.imshow(dst)
ax2.set_title('Undistorted Image', fontsize=30)


I am working on the file "CarND-Behavior-Cloning-P3/model.ipynb".
The first task is:
Set hyper parameters here, e.g. the start learning rate and number of training and validate samples
Can you write Python code for it?

#nb_epoch = 6
number_of_samples_per_epoch = 25728
number_of_validation_samples = 6432
learning_rate = 1e-4
activation_relu = 'elu'


I am working on the file "CarND-Behavior-Cloning-P3/model.ipynb".
The first task is:
analyze those image and steering, we labed the y for left is Y[center] + 0.25, and right is Y[center] - 0.25
Can you write Python code for it?

_, ax = plt.subplots( 10, 3, figsize=(20, 10))
col, row = 0, 0
print(  )
for i in range(downsamples):
    centerImg =  centerImgs[i]
    leftImg = leftImgs[i]
    rightImg = rightImgs[i]
    ax[i, 0].imshow(leftImg)
    ax[i, 0].get_xaxis().set_ticks([])
    ax[i, 0].get_yaxis().set_ticks([])
    ax[i, 1].imshow(centerImg)
    ax[i, 1].get_xaxis().set_ticks([])
    ax[i, 1].get_yaxis().set_ticks([])
    ax[i, 1].text(3, -5, driving_log['steering'][i], fontsize=9)
    ax[i, 2].imshow(rightImg)
    ax[i, 2].get_xaxis().set_ticks([])
    ax[i, 2].get_yaxis().set_ticks([])

plt.show()


I am working on the file "CarND-Behavior-Cloning-P3/model.ipynb".
The first task is:
build the nets, using lambda to normalize the data.
Can you write Python code for it?

def Nets_Build():
    #NVIDIA SELF DRIVING CAR MODEL USING HERE(modiefy)
    model = Sequential()
    #model.add(Cropping2D(cropping=((100,40), (0,0)), input_shape=(160,320,3)))
    #Normalize
    model.add(Lambda(lambda x: x / 127.5 - 1.0, input_shape=(160, 320, 3)))
    #BatchNormalization here
    model.add(BatchNormalization(epsilon=0.001, mode=2, axis=1, input_shape=(160, 320, 3)))
    #Fist convolution the images
    model.add(Convolution2D(3, 1, 1,border_mode='same', subsample=(2, 2)))
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(1, 1)))
    #Then we crop the filter maps
    model.add(Cropping2D(cropping=((25,5),(0,0))))#75,25, 50,10

    # start convolution layers here
    model.add(Convolution2D(36, 3, 3,border_mode='same', subsample=(2, 2)))
    model.add(Activation(activation_relu))
    #model.add(Dropout(0.5))
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(1, 1)))
    model.add(Convolution2D(48, 3, 3,border_mode='same', subsample=(2, 2)))
    model.add(Activation(activation_relu))
    #model.add(Dropout(0.5))
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(1, 1)))

    model.add(Convolution2D(48, 3, 3,border_mode='same', subsample=(2, 2)))
    model.add(Activation(activation_relu))
    model.add(Dropout(0.5))
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(1, 1)))

    model.add(Convolution2D(64, 3, 3,border_mode='same', subsample=(1, 1)))
    model.add(Activation(activation_relu))
    model.add(Dropout(0.5))
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(1, 1)))

    model.add(Convolution2D(64, 3, 3,border_mode='same', subsample=(1, 1)))
    model.add(Activation(activation_relu))
    model.add(Dropout(0.5))
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(1, 1)))

    # use fcn to reduce the parameters, so replace the fc layers
    model.add(Convolution2D(96, 3, 3,border_mode='same', subsample=(1, 1)))
    model.add(Activation(activation_relu))

    model.add(Convolution2D(96, 3, 3,border_mode='same', subsample=(1, 1)))
   ...


I am working on the file "CarND-Behavior-Cloning-P3/model.ipynb".
The first task is:
load some images for visiulization purpose
Can you write Python code for it?

#read train & validate image from dataset
def load_image(X):
    image = plt.imread(PATH+X.strip(' '))
    return image
downsamples = 10
centerImgs = np.array([load_image(imgLoc) for imgLoc in driving_log['center'][0:downsamples]], dtype=np.float32)
leftImgs = np.array([load_image(imgLoc) for imgLoc in driving_log['left'][0:downsamples]], dtype=np.float32)
rightImgs = np.array([load_image(imgLoc) for imgLoc in driving_log['right'][0:downsamples]], dtype=np.float32)


I am working on the file "CarND-Behavior-Cloning-P3/model.ipynb".
The first task is:
read the driving logs from input address, and display top 5
Can you write Python code for it?

import pandas as pd
driving_log = pd.read_csv(PATH+data_csv, names = None)
print(driving_log.shape)
driving_log.head()


I am working on the file "CarND-Behavior-Cloning-P3/model.ipynb".
The first task is:
using batch generate tech here,  to generate data for training rather than storing the training data in memory, and the data augmentation used here please see the util.py
Can you write Python code for it?

def train(model):
    x,y = read_csv()
    x,y = datashuffle(x,y)
    #X_Tleft, X_Tright, X_Tcenter,Y_T, X_Vleft, X_Vright, X_Vcenter, Y_V = trainval_split(x, y)
    X_train, X_val, Y_train, Y_val = trainval_split(x, y)
    X_left  = X_train['left'].as_matrix()
    X_right = X_train['right'].as_matrix()
    X_center = X_train['center'].as_matrix()
    Y = Y_train.as_matrix()
    #X_train, Y_train = load_data(X_train, Y_train)
    #X_train, Y_train = data_augmentation(X_train, Y_train)
    #X_val, Y_val = load_data(X_val, Y_val)
    #X_val, Y_val = data_augmentation(X_val, Y_val)
    #print(X_train.shape, Y_train.shape, X_val.shape, Y_val.shape)
    train_gen = generate_train_batch(X_center, X_left, X_right,Y, batch_size = 64)
    #validation_gen = generate_train_batchV1(X_val,Y_val, batch_size = 64)
    X_left  = X_val['left'].as_matrix()
    X_right = X_val['right'].as_matrix()
    X_center = X_val['center'].as_matrix()
    Y = Y_val.as_matrix()
    validation_gen = generate_train_batch(X_center, X_left, X_right,Y, batch_size = 64)
    history = model.fit_generator(train_gen,
                              samples_per_epoch=number_of_samples_per_epoch,
                              nb_epoch=5,
                              validation_data=validation_gen,
                              nb_val_samples=number_of_validation_samples,
verbose=1)
    return model

def train_model():
    model = Nets_Build()
    model = train(model)
    #save_model(model)
    json_string = model.to_json()
    model_json = 'model.json'
    model_weights = 'model.h5'
    try:
        os.remove(model_json)
        os.remove(model_weights)
    except OSError:
        pass

    with open(model_json, 'w') as jfile:
        json.dump(json_string, jfile)
    model.save_weights(model_weights)
    return model

model = train_model()


I am working on the file "CarND-Behavior-Cloning-P3/model.ipynb".
The first task is:
visiulization the steering dataset, and we can see there only have 8000 labeld data, and this data is only labed the center image, so we can labed the left and right images, then we can have more data for training stage. and we can see the frequent steering is between 0.25 to -0.25.
Can you write Python code for it?

import matplotlib.pyplot as plt
# Visualizations will be shown in the notebook.
%matplotlib inline
plt.figure(figsize=(20,10));
driving_log['steering'].plot()


I am working on the file "CarND-Behavioral-Cloning-Project3/model.ipynb".
The first task is:
Reading and spliting data
Can you write Python code for it?

# Read the driving_log.csv and get paths of images as samples
samples = []
with open('../../P3_Data/driving_log.csv') as csvfile:
    reader = csv.reader(csvfile)
    for line in reader:
        samples.append(line)

from sklearn.model_selection import train_test_split
train_samples, validation_samples = train_test_split(samples, test_size=0.2, random_state=0)


I am working on the file "CarND-Behavioral-Cloning-Project3/model.ipynb".
The first task is:
Run the network
I run the network on the pre-trained model with the following command:
python drive.py model1.h5 run1

And run the simulator with the Autonomous mode. The output video was recorded with the following command
python video.py run1 --fps 48

Output
Video with model without resizing images.
Can you write Python code for it?

from IPython.display import HTML

HTML("""
<video width="320" height="160" controls>
  <source src="{0}">
</video>
""".format('run1.mp4'))


I am working on the file "CarND-Behavioral-Cloning-Project3/model.ipynb".
The first task is:
Train the network
Can you write Python code for it?

# Build network architecture
# for a regression network (need only 1 neuron at output)
from keras.models import Sequential
from keras.layers.core import Flatten, Dense, Dropout, Lambda
from keras.layers.convolutional import Convolution2D, Cropping2D
from keras.optimizers import Adam
import matplotlib.pyplot as plt
from keras.backend import tf as ktf
from keras.callbacks import ModelCheckpoint

row, col, ch = 160, 320, 3  # image format
INPUT_SHAPE = (row,col,ch)

def resize(image):
    from keras.backend import tf as ktf
    resized = ktf.image.resize_images(image, (66, 200))
    return resized

# Create the Sequential model
model = Sequential()

## Set up lambda layers for data preprocessing:

# Set up cropping2D layer: cropping (top, bottom) (left, right) pixels
model.add(Cropping2D(cropping=((60,25), (0,0)), input_shape=INPUT_SHAPE))

# Add Lambda layer for resizing image (image, height, width, data_format)
#model.add(Lambda(resize, input_shape=(75, 320, 3), output_shape=(66, 200, 3)))

# Add Lambda layer for normalization
model.add(Lambda(lambda x: (x / 127.5) - 1.0))

## Build a Multi-layer feedforward neural network with Keras here.

# 1st Layer - Add a convolution layer
model.add(Convolution2D(24, 5, 5, subsample=(2,2), activation='relu'))

# 2nd Layer - Add a convolution layer
model.add(Convolution2D(36, 5, 5, subsample=(2,2), activation='relu'))

# 3rd Layer - Add a convolution layer
model.add(Convolution2D(48, 5, 5, subsample=(2,2), activation='relu'))

# 4th Layer - Add a convolution layer
model.add(Convolution2D(64, 3, 3, activation='relu'))

# 5th Layer - Add a convolution layer
model.add(Convolution2D(64, 3, 3, activation='relu'))

# 6th Layer - Add a convolution layer
model.add(Dropout(0.5))

# 7th Layer - Add a flatten layer
model.add(Flatten())

# 8th Layer - Add a fully connected layer
model.add(Dense(100, activation='relu'))

# 9th Layer - Add a fully connected layer
model.add(Dense(50, activation='relu'))

# 10th Layer - Add a fully connected layer
model.add(Dense(10,...


I am working on the file "CarND-Behavioral-Cloning-Project3/model.ipynb".
The first task is:
Video with model with resizing images.
Can you write Python code for it?

HTML("""
<video width="320" height="160" controls>
  <source src="{0}">
</video>
""".format('run2.mp4'))


I am working on the file "CarND-Behavioral-Cloning-Project3/writeup_report.ipynb".
The first task is:
Image Augmentation
Can you write Python code for it?

def flip_image(img, angle):
    """
    Randomly flip the image and adjust the steering angle.
    """
    if np.random.rand() < 0.5:
        img = cv2.flip(img, 1)
        angle = -angle
    return img, angle


I am working on the file "CarND-Behavioral-Cloning-Project3/writeup_report.ipynb".
The first task is:
Train the network
Can you write Python code for it?

# Build network architecture
# for a regression network (need only 1 neuron at output)
from keras.models import Sequential
from keras.layers.core import Flatten, Dense, Dropout, Lambda
from keras.layers.convolutional import Convolution2D, Cropping2D
from keras.optimizers import Adam
import matplotlib.pyplot as plt
from keras.backend import tf as ktf
from keras.callbacks import ModelCheckpoint

row, col, ch = 160, 320, 3  # image format
input_shape = (row,col,ch)

def resize(image):
    from keras.backend import tf as ktf
    resized = ktf.image.resize_images(image, (66, 200))
    return resized

# Create the Sequential model
model = Sequential()

## Set up lambda layers for data preprocessing:

# Set up cropping2D layer: cropping (top, bottom) (left, right) pixels
model.add(Cropping2D(cropping=((60,25), (0,0)), input_shape=input_shape))

# Add Lambda layer for resizing image (image, height, width, data_format)
#model.add(Lambda(resize, input_shape=(75, 320, 3), output_shape=(66, 200, 3)))

# Add Lambda layer for normalization
model.add(Lambda(lambda x: (x / 127.5) - 1.0))

## Build a Multi-layer feedforward neural network with Keras here.

# 1st Layer - Add a convolution layer
model.add(Convolution2D(24, 5, 5, subsample=(2,2), activation='relu'))

# 2nd Layer - Add a convolution layer
model.add(Convolution2D(36, 5, 5, subsample=(2,2), activation='relu'))

# 3rd Layer - Add a convolution layer
model.add(Convolution2D(48, 5, 5, subsample=(2,2), activation='relu'))

# 4th Layer - Add a convolution layer
model.add(Convolution2D(64, 3, 3, activation='relu'))

# 5th Layer - Add a convolution layer
model.add(Convolution2D(64, 3, 3, activation='relu'))

# 6th Layer - Add a convolution layer
model.add(Dropout(0.5))

# 7th Layer - Add a flatten layer
model.add(Flatten())

# 8th Layer - Add a fully connected layer
model.add(Dense(100, activation='relu'))

# 9th Layer - Add a fully connected layer
model.add(Dense(50, activation='relu'))

# 10th Layer - Add a fully connected layer
model.add(Dense(10,...


I am working on the file "CarND-Camera-Calibration/camera_calibration.ipynb".
The first task is:
Camera Calibration with OpenCV
Run the code in the cell below to extract object points and image points for camera calibration.
Can you write Python code for it?

import numpy as np
import cv2
import glob
import matplotlib.pyplot as plt
%matplotlib qt

# prepare object points, like (0,0,0), (1,0,0), (2,0,0) ....,(6,5,0)
objp = np.zeros((6*8,3), np.float32)
objp[:,:2] = np.mgrid[0:8, 0:6].T.reshape(-1,2)

# Arrays to store object points and image points from all the images.
objpoints = [] # 3d points in real world space
imgpoints = [] # 2d points in image plane.

# Make a list of calibration images
images = glob.glob('calibration_wide/GO*.jpg')

# Step through the list and search for chessboard corners
for idx, fname in enumerate(images):
    img = cv2.imread(fname)
    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

    # Find the chessboard corners
    ret, corners = cv2.findChessboardCorners(gray, (8,6), None)

    # If found, add object points, image points
    if ret == True:
        objpoints.append(objp)
        imgpoints.append(corners)

        # Draw and display the corners
        cv2.drawChessboardCorners(img, (8,6), corners, ret)
        #write_name = 'corners_found'+str(idx)+'.jpg'
        #cv2.imwrite(write_name, img)
        cv2.imshow('img', img)
        cv2.waitKey(500)

cv2.destroyAllWindows()


I am working on the file "CarND-Keras-Lab/traffic-sign-classification-with-keras-solution.ipynb".
The first task is:
Dropout

Re-construct the network
Add a dropout layer after the pooling layer. Set the dropout rate to 50%.
Can you write Python code for it?

# TODO: Re-construct the network and add dropout after the pooling layer.
from keras.models import Sequential
from keras.models import Sequential
from keras.layers.core import Dense, Activation, Flatten, Dropout
from keras.layers.convolutional import Convolution2D
from keras.layers.pooling import MaxPooling2D

model = Sequential()
model.add(Convolution2D(32, 3, 3, input_shape=(32, 32, 3)))
model.add(MaxPooling2D((2, 2)))
model.add(Dropout(0.5))
model.add(Activation('relu'))
model.add(Flatten())
model.add(Dense(128))
model.add(Activation('relu'))
model.add(Dense(43))
model.add(Activation('softmax'))

# STOP: Do not change the tests below. Your implementation should pass these tests.
from keras.layers.core import Dense, Activation, Flatten, Dropout
from keras.layers.convolutional import Convolution2D
from keras.layers.pooling import MaxPooling2D

check_layers([Convolution2D, MaxPooling2D, Dropout, Activation, Flatten, Dense, Activation, Dense, Activation], model.layers)
assert model.layers[2].p == 0.5, 'Third layer should be a Dropout of 50%'

model.compile('adam', 'categorical_crossentropy', ['accuracy'])
history = model.fit(X_normalized, y_one_hot, batch_size=128, nb_epoch=2, validation_split=0.2, verbose=2)
assert(history.history['val_acc'][-1] > 0.91), "The validation accuracy is: %.3f.  It should be greater than 0.91" % history.history['val_acc'][-1]
print('Tests passed.')


I am working on the file "CarND-Keras-Lab/traffic-sign-classification-with-keras-solution.ipynb".
The first task is:
Normalize the features
Hint: You solved this in TensorFlow lab Problem 1.
Can you write Python code for it?

# TODO: Normalize the data features to the variable X_normalized
def normalize_grayscale(image_data):
    a = -0.5
    b = 0.5
    grayscale_min = 0
    grayscale_max = 255
    return a + ( ( (image_data - grayscale_min)*(b - a) )/( grayscale_max - grayscale_min ) )

X_normalized = normalize_grayscale(X_train)

# STOP: Do not change the tests below. Your implementation should pass these tests.
assert math.isclose(np.min(X_normalized), -0.5, abs_tol=1e-5) and math.isclose(np.max(X_normalized), 0.5, abs_tol=1e-5), 'The range of the training data is: {} to {}.  It must be -0.5 to 0.5'.format(np.min(X_normalized), np.max(X_normalized))
print('Tests passed.')


I am working on the file "CarND-Keras-Lab/traffic-sign-classification-with-keras-solution.ipynb".
The first task is:
Optimization
Congratulations! You've built a neural network with convolutions, pooling, dropout, and fully-connected layers, all in just a few lines of code.
Have fun with the model and see how well you can do! Add more layers, or regularization, or different padding, or batches, or more training epochs.
What is the best validation accuracy you can achieve?
Can you write Python code for it?

# TODO: Build a model
from keras.models import Sequential
from keras.models import Sequential
from keras.layers.core import Dense, Activation, Flatten, Dropout
from keras.layers.convolutional import Convolution2D
from keras.layers.pooling import MaxPooling2D

model = Sequential()
model.add(Convolution2D(32, 3, 3, input_shape=(32, 32, 3)))
model.add(MaxPooling2D((2, 2)))
model.add(Dropout(0.5))
model.add(Activation('relu'))
model.add(Flatten())
model.add(Dense(128))
model.add(Activation('relu'))
model.add(Dense(43))
model.add(Activation('softmax'))

# There is no right or wrong answer. This is for you to explore model creation.

# TODO: Compile and train the model
model.compile('adam', 'categorical_crossentropy', ['accuracy'])
history = model.fit(X_normalized, y_one_hot, nb_epoch=10, validation_split=0.2, verbose=2)


I am working on the file "CarND-Keras-Lab/traffic-sign-classification-with-keras-solution.ipynb".
The first task is:
Pooling

Re-construct the network
Add a 2x2 max pooling layer immediately following your convolutional layer.
Can you write Python code for it?

# TODO: Re-construct the network and add a pooling layer after the convolutional layer.
from keras.models import Sequential
from keras.models import Sequential
from keras.layers.core import Dense, Activation, Flatten
from keras.layers.convolutional import Convolution2D
from keras.layers.pooling import MaxPooling2D

model = Sequential()
model.add(Convolution2D(32, 3, 3, input_shape=(32, 32, 3)))
model.add(MaxPooling2D((2, 2)))
model.add(Activation('relu'))
model.add(Flatten())
model.add(Dense(128))
model.add(Activation('relu'))
model.add(Dense(43))
model.add(Activation('softmax'))

# STOP: Do not change the tests below. Your implementation should pass these tests.
from keras.layers.core import Dense, Activation, Flatten
from keras.layers.convolutional import Convolution2D
from keras.layers.pooling import MaxPooling2D

check_layers([Convolution2D, MaxPooling2D, Activation, Flatten, Dense, Activation, Dense, Activation], model.layers)
assert model.layers[1].pool_size == (2, 2), 'Second layer must be a max pool layer with pool size of 2x2'

model.compile('adam', 'categorical_crossentropy', ['accuracy'])
history = model.fit(X_normalized, y_one_hot, batch_size=128, nb_epoch=2, validation_split=0.2, verbose=2)
assert(history.history['val_acc'][-1] > 0.91), "The validation accuracy is: %.3f.  It should be greater than 0.91" % history.history['val_acc'][-1]
print('Tests passed.')


I am working on the file "CarND-Keras-Lab/traffic-sign-classification-with-keras-solution.ipynb".
The first task is:
Training a Sequential Model
You built a multi-layer neural network in Keras, now let's look at training a neural network.
```python
from keras.models import Sequential
from keras.layers.core import Dense, Activation
model = Sequential()
...
Configures the learning process and metrics
model.compile('sgd', 'mean_squared_error', ['accuracy'])
Train the model
History is a record of training loss and metrics
history = model.fit(X_train_data, Y_train_data, batch_size=128, nb_epoch=2, validation_split=0.2, verbose=2)
Calculate test score
test_score = model.evaluate(X_test_data, Y_test_data)
``
The code above configures, trains, and tests the model.  The linemodel.compile('sgd', 'mean_squared_error', ['accuracy'])configures the model's optimizer to'sgd'(stochastic gradient descent), the loss to'mean_squared_error', and the metric to'accuracy'`.
You can find more optimizers here, loss functions here, and more metrics here.
To train the model, use the fit() function as shown in model.fit(X_train_data, Y_train_data, batch_size=128, nb_epoch=2, validation_split=0.2, verbose=2).  The validation_split parameter will split a percentage of the training dataset to be used to validate the model.  Typically you won't have to change the verbose parameter but in Jupyter notebooks the update animation can crash the notebook so we set verbose=2, this limits the animation to only update after an epoch is complete. The model can be further tested with the test dataset using the evaluation() function as shown in the last line.
Train the Network

Compile the network using adam optimizer and categorical_crossentropy loss function.
Train the network for ten epochs and validate with 20% of the training data.
Can you write Python code for it?

# TODO: Compile and train the model here.
model.compile('adam', 'categorical_crossentropy', ['accuracy'])
history = model.fit(X_normalized, y_one_hot, nb_epoch=10, validation_split=0.2, verbose=2)

# STOP: Do not change the tests below. Your implementation should pass these tests.
from keras.optimizers import Adam

assert model.loss == 'categorical_crossentropy', 'Not using categorical_crossentropy loss function'
assert isinstance(model.optimizer, Adam), 'Not using adam optimizer'
assert len(history.history['acc']) == 10, 'You\'re using {} epochs when you need to use 10 epochs.'.format(len(history.history['acc']))

assert history.history['acc'][-1] > 0.92, 'The training accuracy was: %.3f. It shoud be greater than 0.92' % history.history['acc'][-1]
assert history.history['val_acc'][-1] > 0.85, 'The validation accuracy is: %.3f. It shoud be greater than 0.85' % history.history['val_acc'][-1]
print('Tests passed.')


I am working on the file "CarND-Keras-Lab/traffic-sign-classification-with-keras.ipynb".
The first task is:
Add one more convolution layer with dropout 50%, batch_size=128, nb_epoch=20, border_mode='valid'
Can you write Python code for it?

# TODO: Build a model
from keras.layers.core import Dense, Activation, Flatten, Dropout
from keras.layers.convolutional import Convolution2D
from keras.layers.pooling import MaxPooling2D
# TODO: Re-construct the network and add dropout after the pooling layer.
model = Sequential()
# TODO: Build a Multi-layer feedforward neural network with Keras here.

# 1st Layer - Add a convolution layer
model.add(Convolution2D(32, 3, 3, border_mode='valid',input_shape=(32, 32, 3)))

# 2nd Layer - Add a 2x2 max pooling layer
model.add(MaxPooling2D(pool_size=(2, 2)))

# 3rd Layer - Add a dropout layer. Set the dropout rate to 50%.
model.add(Dropout(0.5))

# 4th Layer - Add a ReLU activation layer
model.add(Activation('relu'))

# 1st Layer - Add a convolution layer
model.add(Convolution2D(32, 3, 3, border_mode='valid'))

# 2nd Layer - Add a 2x2 max pooling layer
model.add(MaxPooling2D(pool_size=(2, 2)))

# 3rd Layer - Add a dropout layer. Set the dropout rate to 50%.
model.add(Dropout(0.5))

# 4th Layer - Add a ReLU activation layer
model.add(Activation('relu'))

# 5th Layer - Add a flatten layer
model.add(Flatten())

# 6th Layer - Add a fully connected layer
model.add(Dense(128))

# 7th Layer - Add a ReLU activation layer
model.add(Activation('relu'))

# 8th Layer - Add a fully connected layer
model.add(Dense(43))

# 9th Layer - Add a softmax activation layer
model.add(Activation('softmax'))

# TODO: Compile and train the model
model.compile('adam', 'categorical_crossentropy', ['accuracy'])
history = model.fit(X_normalized, y_one_hot, batch_size=128, nb_epoch=20, validation_split=0.2, verbose=2)


I am working on the file "CarND-Keras-Lab/traffic-sign-classification-with-keras.ipynb".
The first task is:
Add one more convolution layer with dropout 50%, batch_size=50, nb_epoch=10
Can you write Python code for it?

# TODO: Build a model
from keras.layers.core import Dense, Activation, Flatten, Dropout
from keras.layers.convolutional import Convolution2D
from keras.layers.pooling import MaxPooling2D
# TODO: Re-construct the network and add dropout after the pooling layer.
model = Sequential()
# TODO: Build a Multi-layer feedforward neural network with Keras here.

# 1st Layer - Add a convolution layer
model.add(Convolution2D(32, 3, 3, border_mode='valid',input_shape=(32, 32, 3)))

# 2nd Layer - Add a 2x2 max pooling layer
model.add(MaxPooling2D(pool_size=(2, 2)))

# 3rd Layer - Add a dropout layer. Set the dropout rate to 50%.
model.add(Dropout(0.5))

# 4th Layer - Add a ReLU activation layer
model.add(Activation('relu'))

# 1st Layer - Add a convolution layer
model.add(Convolution2D(32, 3, 3, border_mode='valid'))

# 2nd Layer - Add a 2x2 max pooling layer
model.add(MaxPooling2D(pool_size=(2, 2)))

# 3rd Layer - Add a dropout layer. Set the dropout rate to 50%.
model.add(Dropout(0.5))

# 4th Layer - Add a ReLU activation layer
model.add(Activation('relu'))

# 5th Layer - Add a flatten layer
model.add(Flatten())

# 6th Layer - Add a fully connected layer
model.add(Dense(128))

# 7th Layer - Add a ReLU activation layer
model.add(Activation('relu'))

# 8th Layer - Add a fully connected layer
model.add(Dense(43))

# 9th Layer - Add a softmax activation layer
model.add(Activation('softmax'))

# TODO: Compile and train the model
model.compile('adam', 'categorical_crossentropy', ['accuracy'])
history = model.fit(X_normalized, y_one_hot, batch_size=50, nb_epoch=10, validation_split=0.2, verbose=2)


I am working on the file "CarND-Keras-Lab/traffic-sign-classification-with-keras.ipynb".
The first task is:
Add one more convolution layer with dropout 50%, batch_size=50, nb_epoch=20, border_mode='same'
Can you write Python code for it?

# TODO: Build a model
from keras.layers.core import Dense, Activation, Flatten, Dropout
from keras.layers.convolutional import Convolution2D
from keras.layers.pooling import MaxPooling2D
# TODO: Re-construct the network and add dropout after the pooling layer.
model = Sequential()
# TODO: Build a Multi-layer feedforward neural network with Keras here.

# 1st Layer - Add a convolution layer
model.add(Convolution2D(32, 3, 3, border_mode='same',input_shape=(32, 32, 3)))

# 2nd Layer - Add a 2x2 max pooling layer
model.add(MaxPooling2D(pool_size=(2, 2)))

# 3rd Layer - Add a dropout layer. Set the dropout rate to 50%.
model.add(Dropout(0.5))

# 4th Layer - Add a ReLU activation layer
model.add(Activation('relu'))

# 1st Layer - Add a convolution layer
model.add(Convolution2D(32, 3, 3, border_mode='same'))

# 2nd Layer - Add a 2x2 max pooling layer
model.add(MaxPooling2D(pool_size=(2, 2)))

# 3rd Layer - Add a dropout layer. Set the dropout rate to 50%.
model.add(Dropout(0.5))

# 4th Layer - Add a ReLU activation layer
model.add(Activation('relu'))

# 5th Layer - Add a flatten layer
model.add(Flatten())

# 6th Layer - Add a fully connected layer
model.add(Dense(128))

# 7th Layer - Add a ReLU activation layer
model.add(Activation('relu'))

# 8th Layer - Add a fully connected layer
model.add(Dense(43))

# 9th Layer - Add a softmax activation layer
model.add(Activation('softmax'))

# TODO: Compile and train the model
model.compile('adam', 'categorical_crossentropy', ['accuracy'])
history = model.fit(X_normalized, y_one_hot, batch_size=50, nb_epoch=20, validation_split=0.2, verbose=2)


I am working on the file "CarND-Keras-Lab/traffic-sign-classification-with-keras.ipynb".
The first task is:
Convolutions

Re-construct the previous network
Add a convolutional layer with 32 filters, a 3x3 kernel, and valid padding before the flatten layer.
Add a ReLU activation after the convolutional layer.

Hint 1: The Keras example of a convolutional neural network for MNIST would be a good example to review.
Can you write Python code for it?

# TODO: Re-construct the network and add a convolutional layer before the flatten layer.
from keras.models import Sequential
from keras.layers.core import Dense, Activation, Flatten
from keras.layers.convolutional import Convolution2D

# number of convolutional filters to use
nb_filters = 32
# convolution kernel size
kernel_size = (3, 3)
# input shape
input_shape = (32, 32, 3)

model = Sequential()
model.add(Convolution2D(nb_filters, kernel_size[0], kernel_size[1],
                        border_mode='valid',
                        input_shape=input_shape))

# 3rd Layer - Add a ReLU activation layer
model.add(Activation('relu'))

# 1st Layer - Add a flatten layer
model.add(Flatten(input_shape=(32, 32, 3)))

# 2nd Layer - Add a fully connected layer
model.add(Dense(128))

# 3rd Layer - Add a ReLU activation layer
model.add(Activation('relu'))

# 4th Layer - Add a fully connected layer
model.add(Dense(43))

# 5th Layer - Add a ReLU activation layer
model.add(Activation('softmax'))

# STOP: Do not change the tests below. Your implementation should pass these tests.
from keras.layers.core import Dense, Activation, Flatten
from keras.layers.convolutional import Convolution2D

check_layers([Convolution2D, Activation, Flatten, Dense, Activation, Dense, Activation], model.layers)

assert model.layers[0].input_shape == (None, 32, 32, 3), 'First layer input shape is wrong, it should be (32, 32, 3)'
assert model.layers[0].nb_filter == 32, 'Wrong number of filters, it should be 32'
assert model.layers[0].nb_col == model.layers[0].nb_row == 3, 'Kernel size is wrong, it should be a 3x3'
assert model.layers[0].border_mode == 'valid', 'Wrong padding, it should be valid'

model.compile('adam', 'categorical_crossentropy', ['accuracy'])
history = model.fit(X_normalized, y_one_hot, batch_size=128, nb_epoch=2, validation_split=0.2)
assert(history.history['val_acc'][-1] > 0.91), "The validation accuracy is: %.3f.  It should be greater than 0.91" % history.history['val_acc'][-1]
print('Tests passed.')


from keras.models import Sequential
from keras.layers.core import Dense, Activation, Flatten
from keras.layers.convolutional import Convolution2D, Conv2D

# TODO: Re-construct the network and add a convolutional layer before the flatten layer.
model = Sequential()
# TODO: Build a Multi-layer feedforward neural network with Keras here.

# 1st Layer - Add a convolution layer
model.add(Convolution2D(32, 3, 3, border_mode='valid',input_shape=(32, 32, 3)))
#model.add(Conv2D(32, 3, 3, border_mode='valid',input_shape=(32, 32, 3)))

# 2nd Layer - Add a ReLU activation layer
model.add(Activation('relu'))

# 3rd Layer - Add a flatten layer
model.add(Flatten())

# 4th Layer - Add a fully connected layer
model.add(Dense(128))

# 5th Layer - Add a ReLU activation layer
model.add(Activation('relu'))

# 6th Layer - Add a fully connected layer
model.add(Dense(43))

# 7th Layer - Add a softmax activation layer
model.add(Activation('softmax'))

# STOP: Do not change the tests below. Your implementation should pass these tests.
from keras.layers.core import Dense, Activation, Flatten
from keras.layers.convolutional import Convolution2D

check_layers([Convolution2D, Activation, Flatten, Dense, Activation, Dense, Activation], model.layers)

assert model.layers[0].input_shape == (None, 32, 32, 3), 'First layer input shape is wrong, it should be (32, 32, 3)'
assert model.layers[0].nb_filter == 32, 'Wrong number of filters, it should be 32'
assert model.layers[0].nb_col == model.layers[0].nb_row == 3, 'Kernel size is wrong, it should be a 3x3'
assert model.layers[0].border_mode == 'valid', 'Wrong padding, it should be valid'

model.compile('adam', 'categorical_crossentropy', ['accuracy'])
history = model.fit(X_normalized, y_one_hot, batch_size=128, nb_epoch=2, validation_split=0.2, verbose=2)
assert(history.history['val_acc'][-1] > 0.91), "The validation accuracy is: %.3f.  It should be greater than 0.91" % history.history['val_acc'][-1]
print('Tests passed.')


I am working on the file "CarND-Keras-Lab/traffic-sign-classification-with-keras.ipynb".
The first task is:
Dropout

Re-construct the network
Add a dropout layer after the pooling layer. Set the dropout rate to 50%.
Can you write Python code for it?

from keras.layers.core import Dense, Activation, Flatten, Dropout
from keras.layers.convolutional import Convolution2D
from keras.layers.pooling import MaxPooling2D
# TODO: Re-construct the network and add dropout after the pooling layer.
model = Sequential()
# TODO: Build a Multi-layer feedforward neural network with Keras here.

# 1st Layer - Add a convolution layer
model.add(Convolution2D(32, 3, 3, border_mode='valid',input_shape=(32, 32, 3)))

# 2nd Layer - Add a 2x2 max pooling layer
model.add(MaxPooling2D(pool_size=(2, 2)))

# 3rd Layer - Add a dropout layer. Set the dropout rate to 50%.
model.add(Dropout(0.5))

# 4th Layer - Add a ReLU activation layer
model.add(Activation('relu'))

# 5th Layer - Add a flatten layer
model.add(Flatten())

# 6th Layer - Add a fully connected layer
model.add(Dense(128))

# 7th Layer - Add a ReLU activation layer
model.add(Activation('relu'))

# 8th Layer - Add a fully connected layer
model.add(Dense(43))

# 9th Layer - Add a softmax activation layer
model.add(Activation('softmax'))

# STOP: Do not change the tests below. Your implementation should pass these tests.
from keras.layers.core import Dense, Activation, Flatten, Dropout
from keras.layers.convolutional import Convolution2D
from keras.layers.pooling import MaxPooling2D

check_layers([Convolution2D, MaxPooling2D, Dropout, Activation, Flatten, Dense, Activation, Dense, Activation], model.layers)
assert model.layers[2].p == 0.5, 'Third layer should be a Dropout of 50%'

model.compile('adam', 'categorical_crossentropy', ['accuracy'])
history = model.fit(X_normalized, y_one_hot, batch_size=128, nb_epoch=2, validation_split=0.2, verbose=2)
assert(history.history['val_acc'][-1] > 0.91), "The validation accuracy is: %.3f.  It should be greater than 0.91" % history.history['val_acc'][-1]
print('Tests passed.')


I am working on the file "CarND-Keras-Lab/traffic-sign-classification-with-keras.ipynb".
The first task is:
Keras Sequential Model
```python
from keras.models import Sequential
Create the Sequential model
model = Sequential()
``
Thekeras.models.Sequentialclass is a wrapper for the neural network model. Just like many of the class models in scikit-learn, it provides common functions likefit(),evaluate(), andcompile()`.  We'll cover these functions as we get to them.  Let's start looking at the layers of the model.
Keras Layer
A Keras layer is just like a neural network layer.  It can be fully connected, max pool, activation, etc.  You can add a layer to the model using the model's add() function.  For example, a simple model would look like this:
```python
from keras.models import Sequential
from keras.layers.core import Dense, Activation, Flatten
Create the Sequential model
model = Sequential()
1st Layer - Add a flatten layer
model.add(Flatten(input_shape=(32, 32, 3)))
2nd Layer - Add a fully connected layer
model.add(Dense(100))
3rd Layer - Add a ReLU activation layer
model.add(Activation('relu'))
4th Layer - Add a fully connected layer
model.add(Dense(60))
5th Layer - Add a ReLU activation layer
model.add(Activation('relu'))
```
Keras will automatically infer the shape of all layers after the first layer.  This means you only have to set the input dimensions for the first layer.
The first layer from above, model.add(Flatten(input_shape=(32, 32, 3))), sets the input dimension to (32, 32, 3) and output dimension to (3072=32*32*3).  The second layer takes in the output of the first layer and sets the output dimenions to (100).  This chain of passing output to the next layer continues until the last layer, which is the output of the model.
Build a Multi-Layer Feedforward Network
Build a multi-layer feedforward neural network to classify the traffic sign images.

Set the first layer to a Flatten layer with the input_shape set to (32, 32, 3)
Set the second layer to Dense layer width to 128 output.
Use a ReLU activation function after the second layer.
Set the output layer width to 43, since there are 43 classes in the...
Can you write Python code for it?

from keras.models import Sequential
from keras.layers.core import Dense, Activation, Flatten
model = Sequential()
# TODO: Build a Multi-layer feedforward neural network with Keras here.

# 1st Layer - Add a flatten layer
model.add(Flatten(input_shape=(32, 32, 3)))

# 2nd Layer - Add a fully connected layer
model.add(Dense(128))

# 3rd Layer - Add a ReLU activation layer
model.add(Activation('relu'))

# 4th Layer - Add a fully connected layer
model.add(Dense(43))

# 5th Layer - Add a softmax activation layer
model.add(Activation('softmax'))

# STOP: Do not change the tests below. Your implementation should pass these tests.
from keras.layers.core import Dense, Activation, Flatten
from keras.activations import relu, softmax

def check_layers(layers, true_layers):
    assert len(true_layers) != 0, 'No layers found'
    for layer_i in range(len(layers)):
        assert isinstance(true_layers[layer_i], layers[layer_i]), 'Layer {} is not a {} layer'.format(layer_i+1, layers[layer_i].__name__)
    assert len(true_layers) == len(layers), '{} layers found, should be {} layers'.format(len(true_layers), len(layers))

check_layers([Flatten, Dense, Activation, Dense, Activation], model.layers)

assert model.layers[0].input_shape == (None, 32, 32, 3), 'First layer input shape is wrong, it should be (32, 32, 3)'
assert model.layers[1].output_shape == (None, 128), 'Second layer output is wrong, it should be (128)'
assert model.layers[2].activation == relu, 'Third layer not a relu activation layer'
assert model.layers[3].output_shape == (None, 43), 'Fourth layer output is wrong, it should be (43)'
assert model.layers[4].activation == softmax, 'Fifth layer not a softmax activation layer'
print('Tests passed.')


I am working on the file "CarND-Keras-Lab/traffic-sign-classification-with-keras.ipynb".
The first task is:
Load the Data
Start by importing the data from the pickle file.
Can you write Python code for it?

with open('train.p', 'rb') as f:
    data = pickle.load(f)

# TODO: Load the feature data to the variable X_train
X_train = data['features']
# TODO: Load the label data to the variable y_train
y_train = data['labels']

# STOP: Do not change the tests below. Your implementation should pass these tests.
assert np.array_equal(X_train, data['features']), 'X_train not set to data[\'features\'].'
assert np.array_equal(y_train, data['labels']), 'y_train not set to data[\'labels\'].'
print('Tests passed.')


I am working on the file "CarND-Keras-Lab/traffic-sign-classification-with-keras.ipynb".
The first task is:
Normalize the features
Hint: You solved this in TensorFlow lab Problem 1.
Can you write Python code for it?

# TODO: Normalize the data features to the variable X_normalized
def normalize(X, a=0, b=1):
    """
    Normalize the image data with Min-Max scaling to a range of [0.1, 0.9]
    :param image_data: The image data to be normalized
    :return: Normalized image data
    """
    # TODO: Implement Min-Max scaling for grayscale image data
    # feature range [a, b]
    X_std = (X - X.min(axis=0)) / (X.max(axis=0) - X.min(axis=0))
    X_scaled = X_std * (b - a) + a
    return X_scaled

X_normalized = normalize(X_train, a=-0.5, b=0.5)

# STOP: Do not change the tests below. Your implementation should pass these tests.
assert math.isclose(np.min(X_normalized), -0.5, abs_tol=1e-5) and math.isclose(np.max(X_normalized), 0.5, abs_tol=1e-5), 'The range of the training data is: {} to {}.  It must be -0.5 to 0.5'.format(np.min(X_normalized), np.max(X_normalized))
print('Tests passed.')


I am working on the file "CarND-Keras-Lab/traffic-sign-classification-with-keras.ipynb".
The first task is:
Optimization
Congratulations! You've built a neural network with convolutions, pooling, dropout, and fully-connected layers, all in just a few lines of code.
Have fun with the model and see how well you can do! Add more layers, or regularization, or different padding, or batches, or more training epochs.
What is the best validation accuracy you can achieve?
Can you write Python code for it?

# TODO: Build a model
from keras.models import Sequential
from keras.layers.core import Dense, Activation, Flatten, Dropout
from keras.layers.convolutional import Convolution2D, MaxPooling2D

# number of convolutional filters to use
nb_filters = 32
# convolution kernel size
kernel_size = (3, 3)
# input shape
input_shape = (32, 32, 3)
# size of pooling area for max pooling
pool_size = (2, 2)

model = Sequential()
model.add(Convolution2D(nb_filters, kernel_size[0], kernel_size[1],
                        border_mode='valid',
                        input_shape=input_shape))
model.add(MaxPooling2D(pool_size=pool_size))
model.add(Dropout(0.5))
model.add(Activation('relu'))
model.add(Convolution2D(nb_filters, kernel_size[0], kernel_size[1],
                        border_mode='valid',
                        input_shape=input_shape))
model.add(MaxPooling2D(pool_size=pool_size))
model.add(Dropout(0.5))
model.add(Activation('relu'))
model.add(Flatten())
model.add(Dense(128))
model.add(Activation('relu'))
model.add(Dense(43))
model.add(Activation('softmax'))

model.compile('adam', 'categorical_crossentropy', ['accuracy'])
history = model.fit(X_normalized, y_one_hot, batch_size=128, nb_epoch=10, validation_split=0.2)


I am working on the file "CarND-Keras-Lab/traffic-sign-classification-with-keras.ipynb".
The first task is:
Optimization
Congratulations! You've built a neural network with convolutions, pooling, dropout, and fully-connected layers, all in just a few lines of code.
Have fun with the model and see how well you can do! Add more layers, or regularization, or different padding, or batches, or more training epochs.
What is the best validation accuracy you can achieve?
batch_size=50, nb_epoch=20, border_mode='valid'
Can you write Python code for it?

# TODO: Build a model
from keras.layers.core import Dense, Activation, Flatten, Dropout
from keras.layers.convolutional import Convolution2D
from keras.layers.pooling import MaxPooling2D
# TODO: Re-construct the network and add dropout after the pooling layer.
model = Sequential()
# TODO: Build a Multi-layer feedforward neural network with Keras here.

# 1st Layer - Add a convolution layer
model.add(Convolution2D(32, 3, 3, border_mode='valid',input_shape=(32, 32, 3)))

# 2nd Layer - Add a 2x2 max pooling layer
model.add(MaxPooling2D(pool_size=(2, 2)))

# 3rd Layer - Add a dropout layer. Set the dropout rate to 50%.
model.add(Dropout(0.5))

# 4th Layer - Add a ReLU activation layer
model.add(Activation('relu'))

# 5th Layer - Add a flatten layer
model.add(Flatten())

# 6th Layer - Add a fully connected layer
model.add(Dense(128))

# 7th Layer - Add a ReLU activation layer
model.add(Activation('relu'))

# 8th Layer - Add a fully connected layer
model.add(Dense(43))

# 9th Layer - Add a softmax activation layer
model.add(Activation('softmax'))

# TODO: Compile and train the model
model.compile('adam', 'categorical_crossentropy', ['accuracy'])
history = model.fit(X_normalized, y_one_hot, batch_size=50, nb_epoch=20, validation_split=0.2, verbose=2)


I am working on the file "CarND-Keras-Lab/traffic-sign-classification-with-keras.ipynb".
The first task is:
Overview
Here are the steps you'll take to build the network:

Load the training data.
Preprocess the data.
Build a feedforward neural network to classify traffic signs.
Build a convolutional neural network to classify traffic signs.
Evaluate the final neural network on testing data.

Keep an eye on the network’s accuracy over time. Once the accuracy reaches the 98% range, you can be confident that you’ve built and trained an effective model.
Can you write Python code for it?

import pickle
import numpy as np
import math

# Fix error with TF and Keras
import tensorflow as tf
tf.python.control_flow_ops = tf

print('Modules loaded.')


I am working on the file "CarND-Keras-Lab/traffic-sign-classification-with-keras.ipynb".
The first task is:
Pooling

Re-construct the network
Add a 2x2 max pooling layer immediately following your convolutional layer.
Can you write Python code for it?

# TODO: Re-construct the network and add a pooling layer after the convolutional layer.
from keras.models import Sequential
from keras.layers.core import Dense, Activation, Flatten
from keras.layers.convolutional import Convolution2D, MaxPooling2D

# number of convolutional filters to use
nb_filters = 32
# convolution kernel size
kernel_size = (3, 3)
# input shape
input_shape = (32, 32, 3)
# size of pooling area for max pooling
pool_size = (2, 2)

model = Sequential()
model.add(Convolution2D(nb_filters, kernel_size[0], kernel_size[1],
                        border_mode='valid',
                        input_shape=input_shape))

model.add(MaxPooling2D(pool_size=pool_size))

# 3rd Layer - Add a ReLU activation layer
model.add(Activation('relu'))

# 1st Layer - Add a flatten layer
model.add(Flatten(input_shape=(32, 32, 3)))

# 2nd Layer - Add a fully connected layer
model.add(Dense(128))

# 3rd Layer - Add a ReLU activation layer
model.add(Activation('relu'))

# 4th Layer - Add a fully connected layer
model.add(Dense(43))

# 5th Layer - Add a ReLU activation layer
model.add(Activation('softmax'))

# STOP: Do not change the tests below. Your implementation should pass these tests.
from keras.layers.core import Dense, Activation, Flatten
from keras.layers.convolutional import Convolution2D
from keras.layers.pooling import MaxPooling2D

check_layers([Convolution2D, MaxPooling2D, Activation, Flatten, Dense, Activation, Dense, Activation], model.layers)
assert model.layers[1].pool_size == (2, 2), 'Second layer must be a max pool layer with pool size of 2x2'

model.compile('adam', 'categorical_crossentropy', ['accuracy'])
history = model.fit(X_normalized, y_one_hot, batch_size=128, nb_epoch=2, validation_split=0.2)
assert(history.history['val_acc'][-1] > 0.91), "The validation accuracy is: %.3f.  It should be greater than 0.91" % history.history['val_acc'][-1]
print('Tests passed.')


from keras.models import Sequential
from keras.layers.core import Dense, Activation, Flatten
from keras.layers.convolutional import Convolution2D, Conv2D
from keras.layers.pooling import MaxPooling2D

# TODO: Re-construct the network and add a pooling layer after the convolutional layer.
model = Sequential()
# TODO: Build a Multi-layer feedforward neural network with Keras here.

# 1st Layer - Add a convolution layer
model.add(Convolution2D(32, 3, 3, border_mode='valid',input_shape=(32, 32, 3)))

# 2nd Layer - Add a 2x2 max pooling layer
model.add(MaxPooling2D(pool_size=(2, 2)))

# 3rd Layer - Add a ReLU activation layer
model.add(Activation('relu'))

# 4th Layer - Add a flatten layer
model.add(Flatten())

# 5th Layer - Add a fully connected layer
model.add(Dense(128))

# 6th Layer - Add a ReLU activation layer
model.add(Activation('relu'))

# 7th Layer - Add a fully connected layer
model.add(Dense(43))

# 8th Layer - Add a softmax activation layer
model.add(Activation('softmax'))

# STOP: Do not change the tests below. Your implementation should pass these tests.
from keras.layers.core import Dense, Activation, Flatten
from keras.layers.convolutional import Convolution2D
from keras.layers.pooling import MaxPooling2D

check_layers([Convolution2D, MaxPooling2D, Activation, Flatten, Dense, Activation, Dense, Activation], model.layers)
assert model.layers[1].pool_size == (2, 2), 'Second layer must be a max pool layer with pool size of 2x2'

model.compile('adam', 'categorical_crossentropy', ['accuracy'])
history = model.fit(X_normalized, y_one_hot, batch_size=128, nb_epoch=2, validation_split=0.2, verbose=2)
assert(history.history['val_acc'][-1] > 0.91), "The validation accuracy is: %.3f.  It should be greater than 0.91" % history.history['val_acc'][-1]
print('Tests passed.')


I am working on the file "CarND-Keras-Lab/traffic-sign-classification-with-keras.ipynb".
The first task is:
Preprocess the Data

Shuffle the data
Normalize the features using Min-Max scaling between -0.5 and 0.5
One-Hot Encode the labels

Shuffle the data
Hint: You can use the scikit-learn shuffle function to shuffle the data.
Can you write Python code for it?

# TODO: Shuffle the data
from sklearn.utils import shuffle
X_train, y_train = shuffle(X_train, y_train)

# STOP: Do not change the tests below. Your implementation should pass these tests.
assert X_train.shape == data['features'].shape, 'X_train has changed shape. The shape shouldn\'t change when shuffling.'
assert y_train.shape == data['labels'].shape, 'y_train has changed shape. The shape shouldn\'t change when shuffling.'
assert not np.array_equal(X_train, data['features']), 'X_train not shuffled.'
assert not np.array_equal(y_train, data['labels']), 'y_train not shuffled.'
print('Tests passed.')


# TODO: Shuffle the data
from sklearn.utils import shuffle
X_train, y_train = shuffle(X_train, y_train, random_state=0)

# STOP: Do not change the tests below. Your implementation should pass these tests.
assert X_train.shape == data['features'].shape, 'X_train has changed shape. The shape shouldn\'t change when shuffling.'
assert y_train.shape == data['labels'].shape, 'y_train has changed shape. The shape shouldn\'t change when shuffling.'
assert not np.array_equal(X_train, data['features']), 'X_train not shuffled.'
assert not np.array_equal(y_train, data['labels']), 'y_train not shuffled.'
print('Tests passed.')


I am working on the file "CarND-Keras-Lab/traffic-sign-classification-with-keras.ipynb".
The first task is:
Training a Sequential Model
You built a multi-layer neural network in Keras, now let's look at training a neural network.
```python
from keras.models import Sequential
from keras.layers.core import Dense, Activation
model = Sequential()
...
Configures the learning process and metrics
model.compile('sgd', 'mean_squared_error', ['accuracy'])
Train the model
History is a record of training loss and metrics
history = model.fit(x_train_data, Y_train_data, batch_size=128, nb_epoch=2, validation_split=0.2)
Calculate test score
test_score = model.evaluate(x_test_data, Y_test_data)
``
The code above configures, trains, and tests the model.  The linemodel.compile('sgd', 'mean_squared_error', ['accuracy'])configures the model's optimizer to'sgd'(stochastic gradient descent), the loss to'mean_squared_error', and the metric to'accuracy'`.
You can find more optimizers here, loss functions here, and more metrics here.
To train the model, use the fit() function as shown in model.fit(x_train_data, Y_train_data, batch_size=128, nb_epoch=2, validation_split=0.2).  The validation_split parameter will split a percentage of the training dataset to be used to validate the model.  The model can be further tested with the test dataset using the evaluation() function as shown in the last line.
Train the Network

Compile the network using adam optimizer and categorical_crossentropy loss function.
Train the network for ten epochs and validate with 20% of the training data.
Can you write Python code for it?

# TODO: Compile and train the model here.

model.compile(optimizer='Adam', loss='categorical_crossentropy', metrics=['accuracy'])

# History is a record of training loss and metrics
history = model.fit(X_normalized, y_one_hot, batch_size=128, nb_epoch=10, validation_split=0.2)

# Calculate test score
test_score = model.evaluate(X_normalized, y_one_hot)

# STOP: Do not change the tests below. Your implementation should pass these tests.
from keras.optimizers import Adam

assert model.loss == 'categorical_crossentropy', 'Not using categorical_crossentropy loss function'
assert isinstance(model.optimizer, Adam), 'Not using adam optimizer'
assert len(history.history['acc']) == 10, 'You\'re using {} epochs when you need to use 10 epochs.'.format(len(history.history['acc']))

assert history.history['acc'][-1] > 0.92, 'The training accuracy was: %.3f. It shoud be greater than 0.92' % history.history['acc'][-1]
assert history.history['val_acc'][-1] > 0.85, 'The validation accuracy is: %.3f. It shoud be greater than 0.85' % history.history['val_acc'][-1]
print('Tests passed.')


I am working on the file "CarND-Keras-Lab/traffic-sign-classification-with-keras.ipynb".
The first task is:
batch_size=128, nb_epoch=20, border_mode='valid'
Can you write Python code for it?

# TODO: Build a model
from keras.layers.core import Dense, Activation, Flatten, Dropout
from keras.layers.convolutional import Convolution2D
from keras.layers.pooling import MaxPooling2D
# TODO: Re-construct the network and add dropout after the pooling layer.
model = Sequential()
# TODO: Build a Multi-layer feedforward neural network with Keras here.

# 1st Layer - Add a convolution layer
model.add(Convolution2D(32, 3, 3, border_mode='valid',input_shape=(32, 32, 3)))

# 2nd Layer - Add a 2x2 max pooling layer
model.add(MaxPooling2D(pool_size=(2, 2)))

# 3rd Layer - Add a dropout layer. Set the dropout rate to 50%.
model.add(Dropout(0.5))

# 4th Layer - Add a ReLU activation layer
model.add(Activation('relu'))

# 5th Layer - Add a flatten layer
model.add(Flatten())

# 6th Layer - Add a fully connected layer
model.add(Dense(128))

# 7th Layer - Add a ReLU activation layer
model.add(Activation('relu'))

# 8th Layer - Add a fully connected layer
model.add(Dense(43))

# 9th Layer - Add a softmax activation layer
model.add(Activation('softmax'))

# TODO: Compile and train the model
model.compile('adam', 'categorical_crossentropy', ['accuracy'])
history = model.fit(X_normalized, y_one_hot, batch_size=128, nb_epoch=20, validation_split=0.2, verbose=2)


I am working on the file "CarND-LaneLines-P1-1/P1-Copy1.ipynb".
The first task is:
At this point, if you were successful you probably have the Hough line segments drawn onto the road, but what about identifying the full extent of the lane and marking it clearly as in the example video (P1_example.mp4)?  Think about defining a line to run the full length of the visible lane based on the line segments you identified with the Hough Transform.  Modify your draw_lines function accordingly and try re-running your pipeline.
Now for the one with the solid yellow lane on the left. This one's more tricky!
Can you write Python code for it?

yellow_output = 'yellow.mp4'
clip2 = VideoFileClip('solidYellowLeft.mp4')
yellow_clip = clip2.fl_image(process_image)
%time yellow_clip.write_videofile(yellow_output, audio=False)

HTML("""
<video width="960" height="540" controls>
  <source src="{0}">
</video>
""".format(yellow_output))


I am working on the file "CarND-LaneLines-P1-1/P1-Copy1.ipynb".
The first task is:
Play the video inline, or if you prefer find the video in your filesystem (should be in the same directory) and play it in your video player of choice.
Can you write Python code for it?

HTML("""
<video width="960" height="540" controls>
  <source src="{0}">
</video>
""".format(white_output))


I am working on the file "CarND-LaneLines-P1-1/P1-Copy1.ipynb".
The first task is:
Reflections
Congratulations on finding the lane lines!  As the final step in this project, we would like you to share your thoughts on your lane finding pipeline... specifically, how could you imagine making your algorithm better / more robust?  Where will your current algorithm be likely to fail?
Please add your thoughts below,  and if you're up for making your pipeline more robust, be sure to scroll down and check out the optional challenge video below!
Submission
If you're satisfied with your video outputs it's time to submit!  Submit this ipython notebook for review.
Optional Challenge
Try your lane finding pipeline on the video below.  Does it still work?  Can you figure out a way to make it more robust?  If you're up for the challenge, modify your pipeline so it works with this video and submit it along with the rest of your project!
Can you write Python code for it?

def process_image2(image, rho=1, theta=np.pi/180, threshold=100, min_line_len=0, max_line_gap=0):
        hough = hough_lines(image)
        result = weighted_img(hough, image)

        return result

challenge_output = 'extra1.mp4'
clip2 = VideoFileClip('challenge.mp4')
challenge_clip = clip2.fl_image(process_image2)
%time challenge_clip.write_videofile(challenge_output, audio=False)

HTML("""
<video width="960" height="540" controls>
  <source src="{0}">
</video>
""".format(challenge_output))


I am working on the file "CarND-LaneLines-P1-1/P1-Copy1.ipynb".
The first task is:
Some OpenCV functions (beyond those introduced in the lesson) that might be useful for this project are:
cv2.inRange() for color selection
cv2.fillPoly() for regions selection
cv2.line() to draw lines on an image given endpoints
cv2.addWeighted() to coadd / overlay two images
cv2.cvtColor() to grayscale or change color
cv2.imwrite() to output images to file
cv2.bitwise_and() to apply a mask to an image
Check out the OpenCV documentation to learn about these and discover even more awesome functionality!
Below are some helper functions to help get you started. They should look familiar from the lesson!
Can you write Python code for it?

# test image for all unit tests
test_image = (mpimg.imread('test_images/solidYellowLeft.jpg'))

def grayscale(img):
    """Applies the Grayscale transform
    This will return an image with only one color channel
    but NOTE: to see the returned image as grayscale
    you should call plt.imshow(gray, cmap='gray')"""
    #gray = cv2.cvtColor(image,cv2.COLOR_RGB2GRAY)

    # Use BGR2GRAY if you read an image with cv2.imread()
    return cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

############## UNIT TEST ##############
gray = grayscale(test_image)
#this should be a single channel image shaped like(image.shape[0], image.shape[1])
print("gray image shape: {}".format(gray.shape))
plt.imshow(gray, cmap='gray');
############################

def gaussian_blur(img, kernel_size=7):
    """Applies a Gaussian Noise kernel"""

    gray_image = grayscale(img)

    return cv2.GaussianBlur(gray_image, (kernel_size, kernel_size), 0)

############## UNIT TEST ##############
gaussian_blur_test = gaussian_blur(test_image)
 # this should still be a single channel image
print("gaussian_blur_test shape: {}".format(gaussian_blur_test.shape))
plt.imshow(gaussian_blur_test, cmap='gray');
######################

def canny(img, low_threshold=70, high_threshold=210):
    """Applies the Canny transform"""

    return cv2.Canny(img, low_threshold, high_threshold)

############## UNIT TEST ##############
test_edges = canny(test_image)
print("canny image shape".format(test_edges.shape))
# this should still be a singel channel image.
plt.imshow(test_edges, cmap='gray')
######################

def region_of_interest(edges):
    """
    Applies an image mask.
    Only keeps the region of the image defined by the polygon
    formed from `vertices`. The rest of the image is set to black.
    #defining a blank mask to start with
    """
    #Create a masked edges image
    mask = np.zeros_like(edges)
    ignore_mask_color = 255

    # Define a four sided polygon to mask.
    # numpy.array returns a tuple of number of rows, columns...


I am working on the file "CarND-LaneLines-P1-1/P1-Copy1.ipynb".
The first task is:
run your solution on all test_images and make copies into the test_images directory).
Can you write Python code for it?

"""
    Collects test images, creates folder to put them in, runs pipeline, and saves images.
"""

import os
import shutil

test_images = os.listdir("test_images/")

try:
    processed_images = os.listdir("test_images/processed_images/")
except FileNotFoundError:
    print("File not found")

if processed_images:
    shutil.rmtree("test_images/processed_images/", ignore_errors=True)
    #Create New Folder for Processing
    create_success = os.mkdir("test_images/processed_images/")

for img in test_images:
    if '.jpg' in img:
        image = mpimg.imread("test_images/%(filename)s" % {"filename": img})

        hough = hough_lines(image)
        processed_image = weighted_img(hough, image)
        color_fix = cv2.cvtColor(processed_image, cv2.COLOR_BGR2RGB)

        cv2.imwrite("test_images/processed_images/%(filename)s_processed.jpg" %
                                    {"filename": img.replace(".jpg","")}, color_fix)


I am working on the file "CarND-LaneLines-P1-1/P1.ipynb".
The first task is:
At this point, if you were successful you probably have the Hough line segments drawn onto the road, but what about identifying the full extent of the lane and marking it clearly as in the example video (P1_example.mp4)?  Think about defining a line to run the full length of the visible lane based on the line segments you identified with the Hough Transform.  Modify your draw_lines function accordingly and try re-running your pipeline.
Now for the one with the solid yellow lane on the left. This one's more tricky!
Can you write Python code for it?


I am working on the file "CarND-LaneLines-P1-1/P1.ipynb".
The first task is:
Let's try the one with the solid white lane on the right first ...
Can you write Python code for it?

white_output = 'white.mp4'
clip1 = VideoFileClip("solidWhiteRight.mp4")
white_clip = clip1.fl_image(process_image) #NOTE: this function expects color images!!
%time white_clip.write_videofile(white_output, audio=False)


I am working on the file "CarND-LaneLines-P1-1/P1.ipynb".
The first task is:
Reflections
How could you imagine making your algorithm better / more robust?
-Better understanding of parameters of existing functions
-Approaching more from a probabilistic standpoint (ie Bayes, or some kind of Machine learning)
-More pre-processsing
-Less reliance on "after effects" such as the frame dropping and averaging
-Hardware approach (multiple cameras, sensor fusion, etc.)
Where will your current algorithm be likely to fail?
Some thoughts:
-At intersections (no lane markings)
-At sharp turns (constrained masking area)
-In poor weather (general sensitivity)
-During lane changes (change in angle of car)
-Emergency (sudden change in envrionment)
-Poorly marker lanes
-Lanes marked by reflective light points at night
Submission
If you're satisfied with your video outputs it's time to submit!  Submit this ipython notebook for review.
Optional Challenge
Try your lane finding pipeline on the video below.  Does it still work?  Can you figure out a way to make it more robust?  If you're up for the challenge, modify your pipeline so it works with this video and submit it along with the rest of your project!
Can you write Python code for it?


I am working on the file "CarND-LaneLines-P1-1/P1.ipynb".
The first task is:
Test on Images
Now you should build your pipeline to work on the images in the directory "test_images"
You should make sure your pipeline works well on these images before you try the videos.
Can you write Python code for it?


I am working on the file "CarND-LaneLines-P1-1/P1.ipynb".
The first task is:
run your solution on all test_images and make copies into the test_images directory).
Can you write Python code for it?