I am working on the file ".ipynb_checkpoints/data_preparation-checkpoint.ipynb".
The first task is:
5. N-gram generation
Can you write Python code for it?

# helper functions

def remove_zero(inlist):
    for ele in inlist:
        if len(ele) == 0:
            inlist.remove(ele)

def gram_to_string(segments, sentence_text):
    """
    return the surface givan n_gram
    """
    start_idx = segments[0][1]['CharacterOffsetBegin']
    end_idx = segments[-1][1]['CharacterOffsetEnd']
    text = sentence_text[int(start_idx): int(end_idx)].encode('UTF-8')
    return (text, start_idx, end_idx)

def n_grams_helper(parsed, n, sentence_text):
    """
    wrapper for previous function
    """
    ret = []

    text = parsed['text']
    words = parsed['words']
    length = len(words)

    if length<n:
        return []

    for idx in range(0, length-n+1):
        segments = words[idx : idx+n]
        ret.append(gram_to_string(segments,sentence_text))
    return ret

def n_grams_generator(stanford_core, text, max_n, parsed):
    grams = {}

    # parsed = stanford_parse(stanford_core, text)
    sentences = parsed['sentences']

    for n in range(1, max_n):
        grams[n] = []
        for sen in sentences:
            grams[n].extend(n_grams_helper(sen, n, text))
        remove_zero(grams[n])
    return grams

bar = pyprind.ProgBar(len(tweet_corpus), width = 70)

for tweet in tweet_corpus.values():
    bar.update()
    tweet['ngrams'] = n_grams_generator(corenlp, tweet['tweet_info']['text'],\
                                                    100, tweet['stanford_parsed'])

# n_grams_reprocess:

for tweet in tweet_corpus.values():
    for n in tweet['ngrams'].keys():
        for onegram in tweet['ngrams'][n]:
            if onegram.count("/") == 1:
                if "http" in onegram:
                    continue
                segments = onegram.split("/")
                remove_zero(segments)
                tweet['ngrams'][n].extend(segments)

            if onegram.count("-") == 1:
                segments = onegram.split("-")
                remove_zero(segments)
                tweet['ngrams'][n].extend(segments)

            if...


I am working on the file ".ipynb_checkpoints/data_preparation-checkpoint.ipynb".
The first task is:
Get a set of unique tweet ids
Can you write Python code for it?

tweet_ids = set()
for file_name in file_list:
    with open ("./data/v1-NEELOnly/" + file_name, "r") as f:
        for line in f:
            segs = line.split("\t")
            tweet_id = segs[0]
            tweet_ids.add(tweet_id)


I am working on the file ".ipynb_checkpoints/data_preparation-checkpoint.ipynb".
The first task is:
Iterate throught the tweets
Can you write Python code for it?

for tweet in tweet_corpus.values():
    tweet['hashtag_mapping'] = make_hashtag_mapping(tweet)
    tweet['usermention_mapping'] = make_usermention_mapping(tweet)
    tweet['cashtag_mapping'] = make_cashtag_mapping(tweet)
    tweet['url_mapping'] = make_url_mapping(tweet)

with open(data_folder+"Tweet/NEEL_tweets(with_tw).pickle", "wb") as f:
    pickle.dump(tweet_corpus, f)


I am working on the file ".ipynb_checkpoints/data_preparation-checkpoint.ipynb".
The first task is:
b. Load File
label-trainDev.tsv -- train split of NEEL [2] dataset
label-train.tsv -- train split of label-trainDev.tsv used in this paper (used for parameter tuning)
label-dev.tsv -- dev split of label-trainDev.tsv used in this paper (used for parameter tuning)
label-test.tsv -- test split of NEEL [2] dataset
Can you write Python code for it?

file_list = ["label-trainDev.tsv", "label-train.tsv", "label-dev.tsv", "label-test.tsv"]


I am working on the file ".ipynb_checkpoints/data_preparation-checkpoint.ipynb".
The first task is:
c. Cashtag
Can you write Python code for it?

print tweet_corpus['100982604374880256']['tweet_info']['entities']['symbols']

def make_cashtag_mapping(tweet):
    ret = {}
    for tag in tweet['tweet_info']['entities']['symbols']:
        ret[tag['text']] = tag
    return ret


I am working on the file ".ipynb_checkpoints/data_preparation-checkpoint.ipynb".
The first task is:
c. Retrieve Tweets using API
Define a function to retrieve tweets
Can you write Python code for it?

def fill_tweet_set(id_list):
    # set up progress bar
    bar = pyprind.ProgBar(len(id_list))

    tweet_set = {}
    failed = {}

    for id in id_list:
        bar.update()

        try:
            tweet_set[id] = api.get_status(id,)._json
        except:
            failed[id] = sys.exc_info()[0]

    return (tweet_set, failed)

result = fill_tweet_set(tweet_ids)


I am working on the file ".ipynb_checkpoints/data_preparation-checkpoint.ipynb".
The first task is:
d. URL
Can you write Python code for it?

print tweet_corpus['100982604374880256']['tweet_info']['entities']['urls']

def make_url_mapping(tweet):
    ret = {}
    for url in tweet['tweet_info']['entities']['urls']:
        ret[url['url']] = url
    return ret


I am working on the file ".ipynb_checkpoints/data_science_lib_demo-checkpoint.ipynb".
The first task is:
Basic models w/ no pre-processing
KNN
Here we do a simple K-nearest neighbors (KNN) classification with stratified 10-fold (default) cross-validation with a grid search over the default of 1 to 30 nearest neighbors and the use of either "uniform" or "distance" weights:
Can you write Python code for it?

%%time

reload(dsl)

estimator = 'knn'

# Set pipeline keyword arguments
optimized_pipeline_kwargs = {
    'feature_selection_type': None,
    'scale_type': None,
    'feature_interactions': False,
    'transform_type': None
    }

# Initialize pipeline
optimized_pipeline = ppl.OptimizedPipeline(estimator,**optimized_pipeline_kwargs)

# Set pipeline fitting parameters
fit_kwargs = {
    'cv': 10,
    'num_parameter_combos': None,
    'n_jobs': -1,
    'random_state': 6,
    'suppress_output': True,
    'use_default_param_dist': True,
    'param_dist': None,
    'test_size': 0.2 # 20% saved as test set
}

# Fit data
optimized_pipeline.fit(X,y,**fit_kwargs)

# Save
optimized_pipelines[estimator] = optimized_pipeline


I am working on the file ".ipynb_checkpoints/data_science_lib_demo-checkpoint.ipynb".
The first task is:
Note how the estimator__weights parameter isn't set for the KNN estimator.
Other models
This code currently supports K-nearest neighbors, logistic regression, support vector machines, multilayer perceptrons, random forest, and adaboost.
We can loop through and pick the best model like this:
Can you write Python code for it?

%%time

reload(dsl)

classifiers = ['knn','logistic_regression','svm',
               'multilayer_perceptron','random_forest','adaboost']

for estimator in classifiers:
    # Set pipeline keyword arguments
    optimized_pipeline_kwargs = {
        'feature_selection_type': None,
        'scale_type': None,
        'feature_interactions': False,
        'transform_type': None
        }

    # Initialize pipeline
    optimized_pipeline = dsl.OptimizedPipeline(estimator,**optimized_pipeline_kwargs)

    # Set pipeline fitting parameters
    fit_kwargs = {
        'cv': 10,
        'num_parameter_combos': None,
        'n_jobs': -1,
        'random_state': 6,
        'suppress_output': True,
        'use_default_param_dist': True,
        'param_dist': None,
        'test_size': 0.2
    }

    # Fit data
    optimized_pipeline.fit(X,y,**fit_kwargs)

    # Save
    optimized_pipelines[estimator] = optimized_pipeline

format_str = '{0:<22} {1:<15} {2:<15}'

print format_str.format(*['model','train score','test score'])
print format_str.format(*['','',''])
for x in [[key,value.train_score_,value.test_score_] for key,value in optimized_pipelines.iteritems()]:
    print format_str.format(*x)


I am working on the file ".ipynb_checkpoints/data_science_lib_demo-checkpoint.ipynb".
The first task is:
Now we have 17 features.
Split into input/output data
Can you write Python code for it?

# Set output feature
output_feature = 'Survived_1'

# Get all column names
column_names = list(simulation_df.columns)

# Get input features
input_features = [x for x in column_names if x != output_feature]

# Split into features and responses
X = simulation_df[input_features].copy()
y = simulation_df[output_feature].copy()


I am working on the file ".ipynb_checkpoints/data_science_lib_demo-checkpoint.ipynb".
The first task is:
Null model
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/data_science_lib_demo-checkpoint.ipynb".
The first task is:
Random forest without scaling still appears to have the best test score. Though that with scaling had closer test and train scores.
All models with SelectKBest feature selection
Setting the feature_selection_type keyword argument will use SelectKBest with f_classif for feature selection:
Can you write Python code for it?

%%time

reload(dsl)

classifiers = ['knn','logistic_regression','svm',
               'multilayer_perceptron','random_forest','adaboost']

prefix = 'select'

for estimator in classifiers:
    # Set pipeline keyword arguments
    optimized_pipeline_kwargs = {
        'feature_selection_type': 'select_k_best',
        'scale_type': None,
        'feature_interactions': False,
        'transform_type': None
        }

    # Initialize pipeline
    optimized_pipeline = dsl.OptimizedPipeline(estimator,**optimized_pipeline_kwargs)

    # Set pipeline fitting parameters
    fit_kwargs = {
        'cv': 10,
        'num_parameter_combos': None,
        'n_jobs': -1,
        'random_state': 6,
        'suppress_output': True,
        'use_default_param_dist': True,
        'param_dist': None,
        'test_size': 0.2
    }

    # Fit data
    optimized_pipeline.fit(X,y,**fit_kwargs)

    # Form name used to save optimized pipeline
    pipeline_name = '%s_%s'%(prefix,estimator)

    # Save
    optimized_pipelines[pipeline_name] = optimized_pipeline

format_str = '{0:<30} {1:<15} {2:<15} {3:<15}'

print format_str.format(*['model','train score','test score','train-test'])
print format_str.format(*['','','',''])
for x in [[key,value.train_score_,value.test_score_,value.train_score_-value.test_score_] for key,value in optimized_pipelines.iteritems()]:
    print format_str.format(*x)


I am working on the file ".ipynb_checkpoints/data_science_lib_demo-checkpoint.ipynb".
The first task is:
So the best model was random forest.
Here's the report for the model:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/data_science_lib_demo-checkpoint.ipynb".
The first task is:
The best training score was acheived by the random forest classifier.
Can you write Python code for it?

score_df['test-train'].sort_values().plot(kind='barh',figsize=(10,20))

ax = score_df.plot(x=['test_score'],y='test-train',style='o',legend=None)

ax.set_xlabel('test score')
ax.set_ylabel('test-train')


I am working on the file ".ipynb_checkpoints/data_science_lib_demo-checkpoint.ipynb".
The first task is:
The optimal parameter was 96 for the n_estimators parameter for the RandomizedForestClassifier.
All models with standard scaling
We can set the scaling type using the scale_type keyword argument:
Can you write Python code for it?

%%time

reload(dsl)

classifiers = ['knn','logistic_regression','svm',
               'multilayer_perceptron','random_forest','adaboost']

prefix = 'scale'

for estimator in classifiers:
    # Set pipeline keyword arguments
    optimized_pipeline_kwargs = {
        'feature_selection_type': None,
        'scale_type': 'standard',
        'feature_interactions': False,
        'transform_type': None
        }

    # Initialize pipeline
    optimized_pipeline = dsl.OptimizedPipeline(estimator,**optimized_pipeline_kwargs)

    # Set pipeline fitting parameters
    fit_kwargs = {
        'cv': 10,
        'num_parameter_combos': None,
        'n_jobs': -1,
        'random_state': 6,
        'suppress_output': True,
        'use_default_param_dist': True,
        'param_dist': None,
        'test_size': 0.2
    }

    # Fit data
    optimized_pipeline.fit(X,y,**fit_kwargs)

    # Form name used to save optimized pipeline
    pipeline_name = '%s_%s'%(prefix,estimator)

    # Save
    optimized_pipelines[pipeline_name] = optimized_pipeline

format_str = '{0:<30} {1:<15} {2:<15}'

print format_str.format(*['model','train score','test score'])
print format_str.format(*['','',''])
for x in [[key,value.train_score_,value.test_score_] for key,value in optimized_pipelines.iteritems()]:
    print format_str.format(*x)


I am working on the file ".ipynb_checkpoints/data_science_lib_demo-checkpoint.ipynb".
The first task is:
The output of the train_model() method is an instance of my custom OptimizedPipeline class  containing all of the data associated with the nested stratified k-folds cross-validation.
This includes the data, its test/train splits (based on the test_size percentage keyword argument), the GridSearchCV or RandomizedGridSearchCV object, the Pipeline object that has been retrained using all of the data with the best parameters, test/train scores, and validation metrics/reports.
A report can be printed immediately after the fit by setting the suppress_output keyword argument to True.
It lists the steps in the pipeline, their optimized settings, the test/training accuracy (or L2 regression score), the grid search parameters, and the best parameters.
If the estimator used is a classifier it also includes the confusion matrix, normalized confusion matrix, and a classification report containing precision/recall/f1-score for each class.
This same report is also accessible by printing the OptimizedPipeline class instance:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/data_science_lib_demo-checkpoint.ipynb".
The first task is:
These default parameters can be ignored by setting the use_default_param_dist keyword argument to False.
The param_dist keyword argument can be used to keep default parameters (if use_default_param_dist set to True) or to be used as the sole source of parameters (if use_default_param_dist set to False).
Here is a demonstration of generation of default parameters with those in param_dist being overridden:
Can you write Python code for it?

%%time

reload(dsl)

model_name = 'custom_override_%s'%(estimator_name)

# Set custom parameters
param_dist = {
    'estimator__n_neighbors': range(30,500)
}

estimator = 'knn'

# Set pipeline keyword arguments
optimized_pipeline_kwargs = {
    'feature_selection_type': None,
    'scale_type': None,
    'feature_interactions': False,
    'transform_type': None
    }

# Initialize pipeline
optimized_pipeline = dsl.OptimizedPipeline(estimator,**optimized_pipeline_kwargs)

# Set pipeline fitting parameters
fit_kwargs = {
    'cv': 10,
    'num_parameter_combos': None,
    'n_jobs': -1,
    'random_state': 6,
    'suppress_output': False,
    'use_default_param_dist': True,
    'param_dist': param_dist,
    'test_size': 0.2 # 20% saved as test set
}

# Fit data
optimized_pipeline.fit(X,y,**fit_kwargs)

# Save
optimized_pipelines[model_name] = optimized_pipeline


I am working on the file ".ipynb_checkpoints/data_science_lib_demo-checkpoint.ipynb".
The first task is:
This doesn't appear to result in many gains in this case.
All models with transformed data
Setting the transform_type to 'pca' or 't-sne' will apply Principal Component Analysis or t-distributed stochastic neighbor embedding, respectively, to the data before applying the estimator:
Can you write Python code for it?

%%time

reload(dsl)

classifiers = ['knn','logistic_regression','svm',
               'multilayer_perceptron','random_forest','adaboost']

prefix = 'pca'

for estimator in classifiers:
    # Set pipeline keyword arguments
    optimized_pipeline_kwargs = {
        'feature_selection_type': None,
        'scale_type': None,
        'feature_interactions': None,
        'transform_type': 'pca'
        }

    # Initialize pipeline
    optimized_pipeline = dsl.OptimizedPipeline(estimator,**optimized_pipeline_kwargs)

    # Set pipeline fitting parameters
    fit_kwargs = {
        'cv': 10,
        'num_parameter_combos': None,
        'n_jobs': -1,
        'random_state': 6,
        'suppress_output': True,
        'use_default_param_dist': True,
        'param_dist': None,
        'test_size': 0.2
    }

    # Fit data
    optimized_pipeline.fit(X,y,**fit_kwargs)

    # Form name used to save optimized pipeline
    pipeline_name = '%s_%s'%(prefix,estimator)

    # Save
    optimized_pipelines[pipeline_name] = optimized_pipeline

format_str = '{0:<30} {1:<15} {2:<15} {3:<15}'

print format_str.format(*['model','train score','test score','train-test'])
print format_str.format(*['','','',''])
for x in [[key,value.train_score_,value.test_score_,value.train_score_-value.test_score_] for key,value in optimized_pipelines.iteritems()]:
    print format_str.format(*x)


I am working on the file ".ipynb_checkpoints/data_science_lib_demo-checkpoint.ipynb".
The first task is:
Turns out that the best settings are 12 neighbors and the use of the 'uniform' weight.
Note how I've set the random_state keyword agument to 6 so that the models can be compared using the same test/train split.
The default parameters to grid-search over for k-nearest neighbors is 1 to 30 neighbors and either the 'uniform' or 'distance' weight.
The defaults for the pre-processing steps, classifiers, and regressors can be viewed by using the get_default_pipeline_step_parameters() method with the number of features as the input:
Can you write Python code for it?

pre_processing_grid_parameters,classifier_grid_parameters,regression_grid_parameters = \
optimized_pipeline.get_default_pipeline_step_parameters(X.shape[0])

classifier_grid_parameters['knn']


I am working on the file ".ipynb_checkpoints/data_science_lib_demo-checkpoint.ipynb".
The first task is:
With 48 different pre-processing/transformation/classification combinations, this has become rather unwieldy.
Here I make a quick dataframe of the test/train scores and visualize:
Can you write Python code for it?

%matplotlib inline

model_indices = optimized_pipelines.keys()
train_scores = [value.train_score_ for key,value in optimized_pipelines.iteritems()]
test_scores = [value.test_score_ for key,value in optimized_pipelines.iteritems()]

score_df = pd.DataFrame({'training_score':train_scores,'test_score':test_scores},
                        index=model_indices)

score_df['test-train'] = score_df['test_score']-score_df['training_score']

score_df['test_score'].sort_values().plot(kind='barh',figsize=(10,20))


I am working on the file ".ipynb_checkpoints/data_science_lib_demo-checkpoint.ipynb".
The first task is:
Wow, that took forever.
We can get a better idea on how long this will take by setting the num_parameter_combos keyword argument. Setting this will only allow that number of grid combinations to be used for each run:
Can you write Python code for it?

%%time

reload(dsl)

classifiers = ['knn','logistic_regression','svm',
               'multilayer_perceptron','random_forest','adaboost']

prefix = 't_sne_less_combo'

for estimator in classifiers:
    # Set pipeline keyword arguments
    optimized_pipeline_kwargs = {
        'feature_selection_type': None,
        'scale_type': None,
        'feature_interactions': None,
        'transform_type': 't-sne'
        }

    # Initialize pipeline
    optimized_pipeline = dsl.OptimizedPipeline(estimator,**optimized_pipeline_kwargs)

    # Set pipeline fitting parameters
    fit_kwargs = {
        'cv': 10,
        'num_parameter_combos': 1,
        'n_jobs': -1,
        'random_state': 6,
        'suppress_output': True,
        'use_default_param_dist': True,
        'param_dist': None,
        'test_size': 0.2
    }

    # Fit data
    optimized_pipeline.fit(X,y,**fit_kwargs)

    # Form name used to save optimized pipeline
    pipeline_name = '%s_%s'%(prefix,estimator)

    # Save
    optimized_pipelines[pipeline_name] = optimized_pipeline


I am working on the file ".ipynb_checkpoints/data_structure_and_algorithm_py3_4-checkpoint.ipynb".
The first task is:
Counter instance 是一个鲜为人知的特性 可与数学运算操作结合
Can you write Python code for it?

a = Counter(words)
b = Counter(morewords)

a

b

# combine counts
# 合并
c = a + b
print(c)

# Subtract counts
# 减取
d = a -b
print(d)


I am working on the file ".ipynb_checkpoints/data_structure_and_algorithm_py3_4-checkpoint.ipynb".
The first task is:
上述方法仅仅在sequence 中element 属于hashable (不能改变其顺序 同时其存储结构亦不可改变)
Can you write Python code for it?

def dedupe2(items,key=None):
    seen = set()
    for item in items:
        val = item if key is None else key(item)
        if val not in seen:
            yield item
            seen.add(val)


I am working on the file ".ipynb_checkpoints/data_structure_and_algorithm_py3_4-checkpoint.ipynb".
The first task is:
上述示例中morewords还有eyes单词 所以 利用for循环 可利用Counter对象再次计算 还可以使用update method
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/data_structure_and_algorithm_py3_4-checkpoint.ipynb".
The first task is:
不过这样即会使元素本身顺序改变 生成的结果中element 位置被打乱<br>使用generator 可让函数更加通用  以下是对文件本身进行消除重复行
Can you write Python code for it?

with open('somefile.txt','r') as f:
    for line in dedupe2(f):
        print(line)


I am working on the file ".ipynb_checkpoints/data_structure_and_algorithm_py3_4-checkpoint.ipynb".
The first task is:
为寻找两字典的相同点 可通过简单的在两字典keys()  or  items() method 中Return 结果 进行set 操作
Can you write Python code for it?

# Find keys in common
kc = a.keys() & b.keys()
print('a 和 b 共有的键',kc)
# Find keys in a that are not in b
knb = a.keys() - b.keys()
print('a 有的键 而 b 没有的键',knb)
# Find (key value) pair in common
kv = a.items() & b.items()
print('a 和 b 共有的元素',kv)


I am working on the file ".ipynb_checkpoints/data_structure_and_algorithm_py3_4-checkpoint.ipynb".
The first task is:
作为输入 Counter对象可接受任意的hashable sequence 对象 在底层实现上，一个Counter 对象即为一个dict 将element 映射到他出现的次数上
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/data_structure_and_algorithm_py3_4-checkpoint.ipynb".
The first task is:
本身somefile 中有10行zlxs 结果显示一行<br>key 函数 模仿了 sorted min max 函数内置函数的类似功能 key 多了这个以lambda 匿名函数的大力帮助
1.11 命名切片
你的程序已经出现一大堆无法直视的硬编码切片下标
Can you write Python code for it?

ns = ''
for nn in range(10):
    ns += str(nn)
ns = ns*3


I am working on the file ".ipynb_checkpoints/data_structure_and_algorithm_py3_4-checkpoint.ipynb".
The first task is:
这里key args 指定一个函数 将sequence element 转换成 hashable 类型
Can you write Python code for it?

b = [{'x':1,'y':2},{'x':1,'y':3},{'x':1,'y':2},{'x':1,'y':4}]
bd = list(dedupe2(b,key=lambda d: (d['x'],d['y'])))
bd2 = list(dedupe2(b,key=lambda d: (d['x'])))

print('删除满足重复的d[\'x\'],d[\'y\']格式的元素')
bd
# 因为以上格式而又不重复的即为此3 其中{'x':1,'y':2}重复了两次 则被删除

print('删除满足重复的d[\'x\']格式的元素')
bd2
# 因为以上格式而又不重复的即为此2 b中所有元素 都是以{'x': 1开始的 所以返回第一个element


I am working on the file ".ipynb_checkpoints/dataframe_visualization_lib_demo-checkpoint.ipynb".
The first task is:
Finally, we perform a polynomial regression using degrees 1 to 5 and use GridSearchCV
Can you write Python code for it?

%%time

# Scalers
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import MinMaxScaler
from sklearn.preprocessing import Normalizer
from sklearn.preprocessing import Binarizer

# Feature selection tools
from sklearn.feature_selection import SelectKBest, f_classif

# Classifiers
from sklearn.naive_bayes import MultinomialNB
from sklearn.neighbors import KNeighborsClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC, LinearSVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.neural_network import MLPClassifier

# Regression tools
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression

# Cross validation tools
from sklearn.cross_validation import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.grid_search import GridSearchCV

# Classification metrics
from sklearn.metrics import confusion_matrix
from sklearn.metrics import classification_report

# PolynomialFeature used with LinearRegression is a Polynomial regression
pipeline = Pipeline([('poly',PolynomialFeatures()),('regressor', LinearRegression())])

print pipeline

# Initialize grid search with pipeline
grid_search = GridSearchCV(pipeline,
                           {'poly__degree': range(1,5)},
                           cv=10,
                           scoring='neg_mean_squared_error',
                           n_jobs=-1)

# Print grid_search parameters
print grid_search

# Split data into train and test sets
X_train, X_test, y_train, y_test = \
    train_test_split(X, y, test_size=0.2)

# Perform grid search using train data
grid_search.fit(X_train, y_train)

# Make prediction based on model
y_pred = grid_search.best_estimator_.predict(X_test)

# Print scores
print '\nTraining L2 norm score: ',-grid_search.best_score_
print '\nTest L2 norm score: ', np.sqrt(np.square(y_test-y_pred).sum(axis=0))

print '\nBest parameters:\n'
print grid_search.best_params_

# Fit pipeline with best parameters obtained from...


I am working on the file ".ipynb_checkpoints/dataframe_visualization_lib_demo-checkpoint.ipynb".
The first task is:
Graph pairplot
Can you write Python code for it?

%%time
reload(dataframe_visualization_lib)
dataframe_visualization_lib.compare_data(trimmed_df,scatter_plot_filter='Survived',
                                         data_types=data_types,fig_size=100)


I am working on the file ".ipynb_checkpoints/dataframe_visualization_lib_demo-checkpoint.ipynb".
The first task is:
Infer categories of features and correct
Can you write Python code for it?

import dataframe_visualization_lib
reload(dataframe_visualization_lib)

data_types = dataframe_visualization_lib.infer_feature_types(df,suppress_report=False)


I am working on the file ".ipynb_checkpoints/dataframe_visualization_lib_demo-checkpoint.ipynb".
The first task is:
Interpret results
Here's how to read the graph:

Start with the diagonal entries, which correspond to either the histogram for a numerical feature or a horizontal bar chart for a categorical feature.
For categorical features, the color of each bar corresponds to the label to the far-right of the figure.
Ex: Pclass (value,color) pairs are (3,grey), (1,cyan), (2,orange).

For numerical variables, the diagonal is simply the total histogram for the variable

Cells above and below a categorical variable's diagonal are colored by its value.

Look at the diagonal entries and their label on the left side. The diagonals define the color used for each value of that particular feature

If the diagonal represents a

I had a problem.
I wanted to visualize both numerical AND categorical features in a pairplot for exploratory analysis of datasets.
Unfortunately, the closest I could get was to encode categories with numbers and visualize scatter plots.
Can you write Python code for it?

%%time
reload(dataframe_visualization_lib)
dataframe_visualization_lib.compare_data(trimmed_df,data_types=data_types,fig_size=100,
                                         plot_vars=['Age','Embarked'])


I am working on the file ".ipynb_checkpoints/dataframe_visualization_lib_demo-checkpoint.ipynb".
The first task is:
Let's look at the 'Age' data:
Can you write Python code for it?

# Obtain median Age
median_age = df['Age'].median()

# Plot a quick histogram of it:
df['Age'].plot.hist(bins=15,color='r')

# Set title
plt.title('Age distribution (median=%.1f)'%(median_age),size=15)

# Set labels
plt.xlabel('Age',size=12)
plt.ylabel('Number of passengers',size=12)

# Add vertical median line
plt.axvline(median_age,color='orange')


I am working on the file ".ipynb_checkpoints/dataframe_visualization_lib_demo-checkpoint.ipynb".
The first task is:
Replace missing values
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/dataframe_visualization_lib_demo-checkpoint.ipynb".
The first task is:
Replace null values with model prediction:
Can you write Python code for it?

new_data = []
for index, row in simulation_df.iterrows():
    if pd.isnull(row[output_feature]):
        X = row[input_features]
        new_data.append(pipeline.predict(X.values.reshape(1,-1))[0])
    else:
        new_data.append(row[output_feature])

simulation_df[output_feature] = new_data

simulation_df.head(10)

print output_feature
simulation_df[output_feature]

simulation_df[pd.isnull(simulation_df['Age'])]


I am working on the file ".ipynb_checkpoints/dataframe_visualization_lib_demo-checkpoint.ipynb".
The first task is:
There is missing Age, Cabin, and Embarked data.
There is so much missing Cabin data it should probably just be discarded.
Let's look at the distribution for Embarked:
Can you write Python code for it?

print df['Embarked'].value_counts()
print df['Embarked'].value_counts()/df['Embarked'].value_counts().sum()


I am working on the file ".ipynb_checkpoints/dataframe_visualization_lib_demo-checkpoint.ipynb".
The first task is:
There is such a high percentage of S cabins and only 3 entries don't have this data filling the blanks with S (highest frequency) is probably rather good:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/dataframe_visualization_lib_demo-checkpoint.ipynb".
The first task is:
Trim Name, PassengerId, and Ticket
Can you write Python code for it?

del trimmed_df['Name']
del trimmed_df['Ticket']
del trimmed_df['PassengerId']

trimmed_df.head()


I am working on the file ".ipynb_checkpoints/dataframe_visualization_lib_demo-checkpoint.ipynb".
The first task is:
Visualize to get a feel for data
Add ages from simulation to original data
Can you write Python code for it?

df['Age'] = simulation_df['Age']
df.info()


I am working on the file ".ipynb_checkpoints/dipole-checkpoint.ipynb".
The first task is:
Compare these numerical results to the approximate formula of a dipole field (only valid far away from charges), note how it really only starts to compare to the real dipole when far away from the dipole
Can you write Python code for it?

px=q*(x1-x2) #x component of dipole moment [Cm]
py=q*(y1-y2) #y component of dipole moment [Cm]
vdip=pref*np.divide(px*x2d+py*y2d,np.power(r,3))
exdip=-pref*(px*np.power(r,-3)-3*np.multiply(np.multiply(px*x2d+py*y2d,x2d),np.power(r,-5)))
eydip=-pref*(py*np.power(r,-3)-3*np.multiply(np.multiply(px*x2d+py*y2d,y2d),np.power(r,-5)))
edip=np.sqrt(np.square(exdip)+np.square(eydip))


I am working on the file ".ipynb_checkpoints/dipole-checkpoint.ipynb".
The first task is:
Note how different the field strength and the fieldlines are for the approximated dipole vs the actual dipole. Only far away from the dipole are the fields approaching. Remember this if you ever want to apply the convenient dipole formula but might be too close to the actual dipole source.
Can you write Python code for it?

plt.imshow(v,extent=[-xmax, xmax, -xmax, xmax])
plt.title('electrostatic potential V')
plt.xlabel('x')
plt.ylabel('y')
plt.axis('square')


I am working on the file ".ipynb_checkpoints/dipole-checkpoint.ipynb".
The first task is:
Note how the field emanates from the positive charge sinks into the negative charge
Can you write Python code for it?

plt.imshow(e,extent=[-xmax, xmax, -xmax, xmax])
plt.title('electric field and fieldlines')
plt.xlabel('x');
plt.ylabel('y');
plt.streamplot(x2d,y2d,ex,ey)
plt.axis('square')
plt.colorbar
plt.show()


I am working on the file ".ipynb_checkpoints/dipole-checkpoint.ipynb".
The first task is:
Note the positive potential on the positive charge and the negative potential on the negative charge. The potential inbetween the charges is a smooth function with no minima or maxima and the potential inside the conducting spheres is constant.
Can you write Python code for it?

nlines=50;
plt.contour(x2d,y2d,v,nlines)
plt.title('equipotential surfaces')
plt.xlabel('x')
plt.ylabel('y')
plt.axis('square')
plt.colorbar


I am working on the file ".ipynb_checkpoints/dipole-checkpoint.ipynb".
The first task is:
Now apply superposition of both fields (but take care of special cases inside the spheres where there is no field as they are conducting spheres)
Can you write Python code for it?

ex=np.multiply(e1x+e2x,np.multiply(out1,out2)) #E field only if outside both spheres (multiplying with out1 and out2 results in nonzero only outside spheres), inside sphere E=0 because conductor [V/m]
ey=np.multiply(e1y+e2y,np.multiply(out1,out2))
e=np.sqrt(np.square(ex)+np.square(ey)) #and size of E
v=np.multiply(v1+v2,np.multiply(out1,out2))+in1*pref*q1/radius+in2*pref*q2/radius #potential, inside spheres is constant (conductors)


I am working on the file ".ipynb_checkpoints/dipole-checkpoint.ipynb".
The first task is:
create grid to plot (choose 2D plane for visualisation cutting through charge centers , but calculation is correct for 3D)
Can you write Python code for it?

xpoints=256 #nr of grid points in 1 direction
xmax=1 #extension of grid [m]
pref=9e9 # 1/(4pi eps0)
x=np.linspace(-xmax,xmax,xpoints)
y=x
[x2d,y2d]=np.meshgrid(x,y) #2D matrices holding x or y coordinate for each point on the grid
r=np.sqrt(np.square(x2d)+np.square(y2d))


I am working on the file ".ipynb_checkpoints/dipole-checkpoint.ipynb".
The first task is:
define 2 conducting spheres with opposite charge
Can you write Python code for it?

#sphere 1
q=1e-10 #charge on spheres [C]
radius=0.2*xmax #radius of spheres [m]
q1=q #charge on sphere 1 [C]
x1=-0.5*xmax #x position of sphere 1 [m]
y1=0 #y position of sphere 1[m]

r1=np.sqrt(np.square(x2d-x1)+np.square(y2d-y1)) #distance to charge 1
in1=r1<radius  #logical function 1 if inside sphere, 0 if outside
out1=r1>=radius #logical function 0 if inside sphere, 1 if outside

e1x=pref*q1*np.divide(x2d-x1,np.power(r1,3)) #x component of E field valid outside sphere
e1y=pref*q1*np.divide(y2d-y1,np.power(r1,3)) #y component of E field valid outside sphere
#note that we don't need the z component here as we will be looking in a plane where ez=0, for a generic 3d position you need to calculate ez as well
v1=pref*q1*np.divide(np.ones(r1.shape),r1) # potential 1/r valid outside sphere

#sphere 2
q2=-q #charge on sphere 2 [C]
x2=0.5*xmax #x position of sphere 2 [m]
y2=0 #y position of sphere 2[m]

r2=np.sqrt(np.square(x2d-x2)+np.square(y2d-y2)) #distance to charge 1
in2=r2<radius  #logical function 1 if inside sphere, 0 if outside
out2=r2>=radius #logical function 0 if inside sphere, 1 if outside

e2x=pref*q2*np.divide(x2d-x2,np.power(r2,3)) #x component of E field valid outside sphere
e2y=pref*q2*np.divide(y2d-y2,np.power(r2,3)) #y component of E field valid outside sphere
#note that we don't need the z component here as we will be looking in a plane where ez=0, for a generic 3d position you need to calculate ez as well
v2=pref*q2*np.divide(np.ones(r2.shape),r2) # potential 1/r valid outside sphere


I am working on the file ".ipynb_checkpoints/dlnd_language_translation-checkpoint.ipynb".
The first task is:
Batch and pad the source and target sequences
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
def pad_sentence_batch(sentence_batch, pad_int):
    """Pad sentences with <PAD> so that each sentence of a batch has the same length"""
    max_sentence = max([len(sentence) for sentence in sentence_batch])
    return [sentence + [pad_int] * (max_sentence - len(sentence)) for sentence in sentence_batch]

def get_batches(sources, targets, batch_size, source_pad_int, target_pad_int):
    """Batch targets, sources, and the lengths of their sentences together"""
    for batch_i in range(0, len(sources)//batch_size):
        start_i = batch_i * batch_size

        # Slice the right amount for the batch
        sources_batch = sources[start_i:start_i + batch_size]
        targets_batch = targets[start_i:start_i + batch_size]

        # Pad
        pad_sources_batch = np.array(pad_sentence_batch(sources_batch, source_pad_int))
        pad_targets_batch = np.array(pad_sentence_batch(targets_batch, target_pad_int))

        # Need the lengths for the _lengths parameters
        pad_targets_lengths = []
        for target in pad_targets_batch:
            pad_targets_lengths.append(len(target))

        pad_source_lengths = []
        for source in pad_sources_batch:
            pad_source_lengths.append(len(source))

        yield pad_sources_batch, pad_targets_batch, pad_source_lengths, pad_targets_lengths


I am working on the file ".ipynb_checkpoints/dlnd_language_translation-checkpoint.ipynb".
The first task is:
Build the Graph
Build the graph using the neural network you implemented.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
save_path = 'checkpoints/dev'
(source_int_text, target_int_text), (source_vocab_to_int, target_vocab_to_int), _ = helper.load_preprocess()
max_target_sentence_length = max([len(sentence) for sentence in source_int_text])

train_graph = tf.Graph()
with train_graph.as_default():
    input_data, targets, lr, keep_prob, target_sequence_length, max_target_sequence_length, source_sequence_length = model_inputs()

    #sequence_length = tf.placeholder_with_default(max_target_sentence_length, None, name='sequence_length')
    input_shape = tf.shape(input_data)

    train_logits, inference_logits = seq2seq_model(tf.reverse(input_data, [-1]),
                                                   targets,
                                                   keep_prob,
                                                   batch_size,
                                                   source_sequence_length,
                                                   target_sequence_length,
                                                   max_target_sequence_length,
                                                   len(source_vocab_to_int),
                                                   len(target_vocab_to_int),
                                                   encoding_embedding_size,
                                                   decoding_embedding_size,
                                                   rnn_size,
                                                   num_layers,
                                                   target_vocab_to_int)

    training_logits = tf.identity(train_logits.rnn_output, name='logits')
    inference_logits = tf.identity(inference_logits.sample_id, name='predictions')

    masks = tf.sequence_mask(target_sequence_length, max_target_sequence_length, dtype=tf.float32, name='masks')

    with tf.name_scope("optimization"):
        # Loss function
        cost = tf.contrib.seq2seq.sequence_loss(
            training_logits,
            targets,
       ...


I am working on the file ".ipynb_checkpoints/dlnd_language_translation-checkpoint.ipynb".
The first task is:
Build the Neural Network
Apply the functions you implemented above to:

Apply embedding to the input data for the encoder.
Encode the input using your encoding_layer(rnn_inputs, rnn_size, num_layers, keep_prob,  source_sequence_length, source_vocab_size, encoding_embedding_size).
Process target data using your process_decoder_input(target_data, target_vocab_to_int, batch_size) function.
Apply embedding to the target data for the decoder.
Decode the encoded input using your decoding_layer(dec_input, enc_state, target_sequence_length, max_target_sentence_length, rnn_size, num_layers, target_vocab_to_int, target_vocab_size, batch_size, keep_prob, dec_embedding_size) function.
Can you write Python code for it?

def seq2seq_model(input_data, target_data, keep_prob, batch_size,
                  source_sequence_length, target_sequence_length,
                  max_target_sentence_length,
                  source_vocab_size, target_vocab_size,
                  enc_embedding_size, dec_embedding_size,
                  rnn_size, num_layers, target_vocab_to_int):
    """
    Build the Sequence-to-Sequence part of the neural network
    :param input_data: Input placeholder
    :param target_data: Target placeholder
    :param keep_prob: Dropout keep probability placeholder
    :param batch_size: Batch Size
    :param source_sequence_length: Sequence Lengths of source sequences in the batch
    :param target_sequence_length: Sequence Lengths of target sequences in the batch
    :param source_vocab_size: Source vocabulary size
    :param target_vocab_size: Target vocabulary size
    :param enc_embedding_size: Decoder embedding size
    :param dec_embedding_size: Encoder embedding size
    :param rnn_size: RNN Size
    :param num_layers: Number of layers
    :param target_vocab_to_int: Dictionary to go from the target words to an id
    :return: Tuple of (Training BasicDecoderOutput, Inference BasicDecoderOutput)
    """
    # TODO: Implement Function

    encode_output, encode_state = encoding_layer( input_data, rnn_size, num_layers, keep_prob,
                                source_sequence_length, source_vocab_size, enc_embedding_size )

    decode_input = process_decoder_input( target_data, target_vocab_to_int, batch_size )

    decode_outputs_train, decode_outputs_infer = decoding_layer( decode_input, encode_state,
                   target_sequence_length, tf.reduce_max( target_sequence_length ), rnn_size, num_layers,
                   target_vocab_to_int, target_vocab_size, batch_size, keep_prob, dec_embedding_size )

    return decode_outputs_train, decode_outputs_infer

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_seq2seq_model(seq2seq_model)


I am working on the file ".ipynb_checkpoints/dlnd_language_translation-checkpoint.ipynb".
The first task is:
Build the Neural Network
You'll build the components necessary to build a Sequence-to-Sequence model by implementing the following functions below:
- model_inputs
- process_decoder_input
- encoding_layer
- decoding_layer_train
- decoding_layer_infer
- decoding_layer
- seq2seq_model
Input
Implement the model_inputs() function to create TF Placeholders for the Neural Network. It should create the following placeholders:

Input text placeholder named "input" using the TF Placeholder name parameter with rank 2.
Targets placeholder with rank 2.
Learning rate placeholder with rank 0.
Keep probability placeholder named "keep_prob" using the TF Placeholder name parameter with rank 0.
Target sequence length placeholder named "target_sequence_length" with rank 1
Max target sequence length tensor named "max_target_len" getting its value from applying tf.reduce_max on the target_sequence_length placeholder. Rank 0.
Source sequence length placeholder named "source_sequence_length" with rank 1

Return the placeholders in the following the tuple (input, targets, learning rate, keep probability, target sequence length, max target sequence length, source sequence length)
Can you write Python code for it?

def model_inputs():
    """
    Create TF Placeholders for input, targets, learning rate, and lengths of source and target sequences.
    :return: Tuple (input, targets, learning rate, keep probability, target sequence length,
    max target sequence length, source sequence length)
    """
    # TODO: Implement Function

    input_ = tf.placeholder( tf.int32, [None, None], name = "input" )
    target_ = tf.placeholder( tf.int32, [None, None], name = "target" )
    learn_rate_ = tf.placeholder( tf.float32, None, name = "learn_rate" )
    keep_prob_ = tf.placeholder( tf.float32, None, name = "keep_prob" )
    target_sequence_length = tf.placeholder( tf.int32, [None], name="target_sequence_length" )
    max_target_sequence_length = tf.reduce_max( target_sequence_length )
    source_sequence_length = tf.placeholder( tf.int32, [None], name="source_sequence_length" )

    return input_, target_, learn_rate_, keep_prob_, target_sequence_length, max_target_sequence_length, source_sequence_length

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_model_inputs(model_inputs)


I am working on the file ".ipynb_checkpoints/dlnd_language_translation-checkpoint.ipynb".
The first task is:
Check Point
This is your first checkpoint. If you ever decide to come back to this notebook or have to restart the notebook, you can start from here. The preprocessed data has been saved to disk.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import numpy as np
import helper

(source_int_text, target_int_text), (source_vocab_to_int, target_vocab_to_int), _ = helper.load_preprocess()


I am working on the file ".ipynb_checkpoints/dlnd_language_translation-checkpoint.ipynb".
The first task is:
Checkpoint
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import tensorflow as tf
import numpy as np
import helper
import problem_unittests as tests

_, (source_vocab_to_int, target_vocab_to_int), (source_int_to_vocab, target_int_to_vocab) = helper.load_preprocess()
load_path = helper.load_params()


I am working on the file ".ipynb_checkpoints/dlnd_language_translation-checkpoint.ipynb".
The first task is:
Decoding - Training
Create a training decoding layer:
* Create a tf.contrib.seq2seq.TrainingHelper
* Create a tf.contrib.seq2seq.BasicDecoder
* Obtain the decoder outputs from tf.contrib.seq2seq.dynamic_decode
Can you write Python code for it?

def decoding_layer_train(encoder_state, dec_cell, dec_embed_input,
                         target_sequence_length, max_summary_length,
                         output_layer, keep_prob):
    """
    Create a decoding layer for training
    :param encoder_state: Encoder State
    :param dec_cell: Decoder RNN Cell
    :param dec_embed_input: Decoder embedded input
    :param target_sequence_length: The lengths of each sequence in the target batch
    :param max_summary_length: The length of the longest sequence in the batch
    :param output_layer: Function to apply the output layer
    :param keep_prob: Dropout keep probability
    :return: BasicDecoderOutput containing training logits and sample_id
    """
    # TODO: Implement Function

    decode_helper = tf.contrib.seq2seq.TrainingHelper( dec_embed_input, target_sequence_length )
    decoder = tf.contrib.seq2seq.BasicDecoder( dec_cell, decode_helper, encoder_state, output_layer )
    decoder_outputs, decoder_state = tf.contrib.seq2seq.dynamic_decode( decoder, impute_finished=True,
                                                                       maximum_iterations= max_summary_length )
    return decoder_outputs

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_decoding_layer_train(decoding_layer_train)


I am working on the file ".ipynb_checkpoints/dlnd_language_translation-checkpoint.ipynb".
The first task is:
Encoding
Implement encoding_layer() to create a Encoder RNN layer:
 * Embed the encoder input using tf.contrib.layers.embed_sequence
 * Construct a stacked tf.contrib.rnn.LSTMCell wrapped in a tf.contrib.rnn.DropoutWrapper
 * Pass cell and embedded input to tf.nn.dynamic_rnn()
Can you write Python code for it?

from imp import reload
reload(tests)

def encoding_layer(rnn_inputs, rnn_size, num_layers, keep_prob,
                   source_sequence_length, source_vocab_size,
                   encoding_embedding_size):
    """
    Create encoding layer
    :param rnn_inputs: Inputs for the RNN
    :param rnn_size: RNN Size
    :param num_layers: Number of layers
    :param keep_prob: Dropout keep probability
    :param source_sequence_length: a list of the lengths of each sequence in the batch
    :param source_vocab_size: vocabulary size of source data
    :param encoding_embedding_size: embedding size of source data
    :return: tuple (RNN output, RNN state)
    """
    # TODO: Implement Function
    encod_inputs = tf.contrib.layers.embed_sequence( rnn_inputs, source_vocab_size, encoding_embedding_size )

    rnn_cell = tf.contrib.rnn.MultiRNNCell( [ tf.contrib.rnn.LSTMCell( rnn_size ) for _ in range(num_layers) ] )
    # Adding dropout layer
    rnn_cell = tf.contrib.rnn.DropoutWrapper( rnn_cell, output_keep_prob = keep_prob )
    rnn_output, rnn_state = tf.nn.dynamic_rnn( rnn_cell, encod_inputs, source_sequence_length, dtype = tf.float32 )

    return rnn_output, rnn_state

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_encoding_layer(encoding_layer)


I am working on the file ".ipynb_checkpoints/dlnd_language_translation-checkpoint.ipynb".
The first task is:
Neural Network Training
Hyperparameters
Tune the following parameters:

Set epochs to the number of epochs.
Set batch_size to the batch size.
Set rnn_size to the size of the RNNs.
Set num_layers to the number of layers.
Set encoding_embedding_size to the size of the embedding for the encoder.
Set decoding_embedding_size to the size of the embedding for the decoder.
Set learning_rate to the learning rate.
Set keep_probability to the Dropout keep probability
Set display_step to state how many steps between each debug output statement
Can you write Python code for it?

# Number of Epochs
epochs = 10
# Batch Size
batch_size = 256
# RNN Size
rnn_size = 256
# Number of Layers
num_layers = 2
# Embedding Size
encoding_embedding_size = 128
decoding_embedding_size = 128
# Learning Rate
learning_rate = 0.01
# Dropout Keep Probability
keep_probability = 0.8
display_step = 10


I am working on the file ".ipynb_checkpoints/dlnd_language_translation-checkpoint.ipynb".
The first task is:
Preprocess all the data and save it
Running the code cell below will preprocess all the data and save it to file.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
helper.preprocess_and_save_data(source_path, target_path, text_to_ids)


I am working on the file ".ipynb_checkpoints/dlnd_language_translation-checkpoint.ipynb".
The first task is:
Train
Train the neural network on the preprocessed data. If you have a hard time getting a good loss, check the forms to see if anyone is having the same problem.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
def get_accuracy(target, logits):
    """
    Calculate accuracy
    """
    max_seq = max(target.shape[1], logits.shape[1])
    if max_seq - target.shape[1]:
        target = np.pad(
            target,
            [(0,0),(0,max_seq - target.shape[1])],
            'constant')
    if max_seq - logits.shape[1]:
        logits = np.pad(
            logits,
            [(0,0),(0,max_seq - logits.shape[1])],
            'constant')

    return np.mean(np.equal(target, logits))

# Split data to training and validation sets
train_source = source_int_text[batch_size:]
train_target = target_int_text[batch_size:]
valid_source = source_int_text[:batch_size]
valid_target = target_int_text[:batch_size]
(valid_sources_batch, valid_targets_batch, valid_sources_lengths, valid_targets_lengths ) = next(get_batches(valid_source,
                                                                                valid_target,
                                                                                batch_size,
                                                                                source_vocab_to_int['<PAD>'],
                                                                                target_vocab_to_int['<PAD>']))
with tf.Session(graph=train_graph) as sess:
    sess.run(tf.global_variables_initializer())

    for epoch_i in range(epochs):
        for batch_i, (source_batch, target_batch, sources_lengths, targets_lengths) in enumerate(
                get_batches(train_source, train_target, batch_size,
                            source_vocab_to_int['<PAD>'],
                            target_vocab_to_int['<PAD>'])):

            _, loss = sess.run(
                [train_op, cost],
                {input_data: source_batch,
                 targets:...


I am working on the file ".ipynb_checkpoints/dups_filter-checkpoint.ipynb".
The first task is:
A careful check also reveals a renaming of job title (from Marine Superintendent to Fleet Manager, employer: "K" LINE LOGISTICS SINGAPORE), though the posts are the same. This is also interesting and may need further analysis later. However, we first need to re-plot cluster distributions again to see if we really get rid of dups.
Can you write Python code for it?

df = pd.read_csv(SKILL_DAT + 'uniq_doc_index.csv')
df = df.set_index('index')

with(open(LDA_DIR + 'doc_topic_distr.mtx', 'r')) as f:
    doc_topic_distr = mmread(f)

# Global settings for all cluster plots
abbv_clusters = ['FIN', 'HWR', 'SCI', 'PROD', 'CON', 'LEG', 'CUS', 'LOG', 'MKT', 'DEV', 'MAN', 'HOS', 'AUD', 'COM', 'HR']
x = range(1, 16); labels = abbv_clusters

def getPosts2Check(job_title, industry, n_pair=10):
    sub_df = df[(df['title'] == job_title) & (df['industry'] == industry)]
    # Drop duplicated posts due to reposting

    scores = calScore(sub_df, doc_topic_distr)
    scores = scores.query('sim_score < 1').sort_values('sim_score', ascending=False)

    k_pairs = scores.head(n_pair)

    ids2check = np.unique(list(k_pairs['job_id1']) + list(k_pairs['job_id2']))
    print('# posts to check: %d' %len(ids2check))

    posts2check = sub_df[sub_df['job_id'].isin(ids2check)]
    return posts2check.sort_values('employer_name')

#     return k_pairs

def plotClusterDists(posts, figsize):
    n_post = posts.shape[0]
    fig, axarr = plt.subplots(n_post, sharex=True, figsize=figsize)

    for r in range(n_post):
        plt.subplot(n_post, 1, r+1)
        plotClusterDistAtRow(r, posts, doc_topic_distr)
        # Show ylabel at the middle
        if r==(n_post/2 - 1):
            plt.ylabel('Probability', fontsize=24)

    ## Fine tune the fig
    fig.subplots_adjust(hspace=.5)

    # Hide xticks on all subplots except the last one
    hide_xticks(fig)

    # Provide xtick labels only for the last subplot
    plt.xticks(x, labels, rotation=45)
    # Show xlabel only at the last subplot
    plt.xlabel('Skill Clusters', fontsize=20)
    return fig

posts2check = getPosts2Check(job_title="Software Engineer", industry="Financial and Insurance Activities", n_pair=5)
fig = plotClusterDists(posts2check, figsize=(6, 10))
# plt.savefig(LDA_DIR + 'fig/se_in_fin2.pdf')
plt.show(); plt.close()


I am working on the file ".ipynb_checkpoints/dups_filter-checkpoint.ipynb".
The first task is:
By reading the job posts of Capital Match Holdings, we found that they are actually different. However, the differences in their contents are subtle. Cluster distribution of the last post is a bit different from the first two because the last one has no salary detail while the first two provide salary and equity. A closer look at the skill sets in these posts reveals:

skill set of the first 2 posts = skill set of the last post + {equity}

salary in 1st post is more than salary in 2nd post (4K vs. 3K)

Other examples of suspicious dups:
Can you write Python code for it?

se_in_infocom = getPosts2Check(job_title='Software Engineer', industry='Information and Communications')

fig = plotClusterDists(se_in_infocom)
plt.savefig(LDA_DIR + 'fig/se_in_infocom2.pdf')
plt.show(); plt.close()


I am working on the file ".ipynb_checkpoints/dups_filter-checkpoint.ipynb".
The first task is:
Duplications seem still exist! Let us examine further.

Sample posts for SE:
Can you write Python code for it?

sample_employers = ["Capital Match Holdings Pte. Ltd.", "Comfortdelgro Corporation Limited", "Fujitsu Asia Pte Ltd",
             "Millennium Capital Management (Singapore) Pte. Ltd."]

sample_employers = map(str.upper, sample_employers)
sample_se_posts = df[(df.employer_name.isin(sample_employers)) & (df.title == "Software Engineer")]
se_fig = plotClusterDists(sample_se_posts)

plt.show(); plt.close()

dups = df[(df.employer_name.isin(sample_employers)) & (df.title == "Software Engineer")]
dups = dups.sort_values('employer_name')


I am working on the file ".ipynb_checkpoints/dups_filter-checkpoint.ipynb".
The first task is:
Removing reposts
Can you write Python code for it?

df = df.sort_values(['employer_name', 'doc'])
print('# posts bf filtering dups: %d' %df.shape[0])

df.head(10)

df = df.drop_duplicates(['employer_name', 'doc'])
print('# posts after filtering dups: %d' %df.shape[0])

df.head(10)

df = df.reset_index()
df.head()

df.to_csv(SKILL_DAT + 'uniq_doc_index.csv', index=False)


I am working on the file ".ipynb_checkpoints/dups_filter-checkpoint.ipynb".
The first task is:
Sample posts for Administrative Assistant:
Can you write Python code for it?

posts = getPosts2Check(job_title="Administrative Assistant", industry="Financial and Insurance Activities", n_pair=5)

fig = plotClusterDists(posts, figsize=(6, 12))
# plt.savefig(LDA_DIR + 'fig/admin_in_fin.jpg')
plt.show(); plt.close()

# Manager
posts = getPosts2Check(job_title="Manager", industry="Financial and Insurance Activities", n_pair=5)

fig = plotClusterDists(posts, figsize=(6, 12))
plt.savefig(LDA_DIR + 'fig/man_in_fin.jpg')
plt.savefig(LDA_DIR + 'fig/man_in_fin.pdf')
plt.show(); plt.close()


I am working on the file ".ipynb_checkpoints/dups_filter-checkpoint.ipynb".
The first task is:
Save duplicated and unique versions to files for further analysis:
Can you write Python code for it?

dups.to_csv(RES_DIR + 'tmp/se_dups.csv', index=False)
dups.drop_duplicates(['doc']).to_csv(RES_DIR + 'tmp/se_no_dups2.csv', index=False)


I am working on the file ".ipynb_checkpoints/dups_filter-checkpoint.ipynb".
The first task is:
These example tell us that we should also filter out the posts with are almost identical. Such posts can be detected based on the set of common skills (Jaccard similarity) as in the next section.
Detecting Almost-Identical Posts by Common Skills
In this section we use Jaccard similarity to detect posts that are almost-identical. We expect that such posts usually come from the same company. Thus, we first analyze the posts under the same company.
Can you write Python code for it?

cmh = df[df.employer_name == str.upper("Capital Match Holdings Pte. Ltd.")]
pairwiseJacSim(df=cmh)


I am working on the file ".ipynb_checkpoints/dynamic_web_pages-checkpoint.ipynb".
The first task is:
A local event loop is created next:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/dynamic_web_pages-checkpoint.ipynb".
The first task is:
After the web page has been completely loaded the event loop will exit and execution moves to the next line, where the resulting HTML can be extracted as usual:
Can you write Python code for it?

html = webview.page().mainFrame().toHtml()
tree = lxml.html.fromstring(html)
tree.cssselect('#result')[0].text_content()


I am working on the file ".ipynb_checkpoints/dynamic_web_pages-checkpoint.ipynb".
The first task is:
As expected we got the desired response. Notice that AJAX returns data in JSON format.
The question is, how do we find that right JS-like url.
<img src="https://s-media-cache-ak0.pinimg.com/236x/ab/80/e9/ab80e9fc1f771001c8b48bddf74f92d2.jpg">
Using WebKit
This is a web-rendering engine that executes JavaScript. We use it in order to simulate a legitimate web browser's JS request to the server and receive the correct response.
First lets try to receive data from this new url in the traditional manner:
Can you write Python code for it?

html = D(durl)
tree = lxml.html.fromstring(html.decode('utf-8'))
tree.cssselect('#result')[0].text_content()


I am working on the file ".ipynb_checkpoints/dynamic_web_pages-checkpoint.ipynb".
The first task is:
Clearly nothing happened. That is because the data is not resident in the downloaded web page. The actual data is obtained in response to a GET request generated by JavaScript code in the web page and sent to the server. This is called AJAX for Asynchronous JavaScript and XML, a technology for client-server data transfer without reloading the whole HTML page.
We can emulate JS by sending to the server an url just like the JS script would have sent, like this:
Can you write Python code for it?

url = base + 'ajax/search.json?page=0&page_size=10&search_term=a'
html = D(url)
res = json.loads(html.decode('utf-8'))
pprint.pprint(res)


I am working on the file ".ipynb_checkpoints/dynamic_web_pages-checkpoint.ipynb".
The first task is:
First attempt:
Just simple web scraping, downloading html page and searching for div results a
Can you write Python code for it?

D = Downloader()
html = D(surl)
tree = lxml.html.fromstring(html.decode('utf-8'))     # must decode else errors happen
tree.cssselect('div#results a')


I am working on the file ".ipynb_checkpoints/dynamic_web_pages-checkpoint.ipynb".
The first task is:
Next we input country names in the Name box. To select it we use its ID, 'search_term'. Once it is found, use the send_keys() command to input data simulating actual key input. To select all countries use the '.' metacharacter.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/dynamic_web_pages-checkpoint.ipynb".
The first task is:
Next we select the Search button. To select it we use its ID, 'search'. Once it is found, we simulate clicking it with the method click().
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/dynamic_web_pages-checkpoint.ipynb".
The first task is:
The loadFinished callback of the QWebView object is connected to the quit method of QEventLoop so that when the web page finishes loading the event loop will be stopped. The url to load must be wrapped in a QUrl object, which is passed to QWebView.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/dynamic_web_pages-checkpoint.ipynb".
The first task is:
This code was packed in the script webkit_render.py
Cool, heh?
<img src="https://media3.giphy.com/media/MF1kR4YmC2Z20/200_s.gif">
Using Selenium
The advantage of using WebKit is full control to customize the browser renderer to behave as we need it to. If such flexibility is not needed then Selenium is an alternative.
Lets redo the previous example using Selenium and its API. The first step is connect it to the current web browser (Firefox is current in this example).
Can you write Python code for it?

from selenium import webdriver
driver = webdriver.Firefox()


I am working on the file ".ipynb_checkpoints/dynamic_web_pages-checkpoint.ipynb".
The first task is:
We need to wait for AJAX to complete the request before using the results. That is done with a timeout:
Can you write Python code for it?

driver.implicitly_wait(30)                             # 30 seconds


I am working on the file ".ipynb_checkpoints/dynamic_web_pages-checkpoint.ipynb".
The first task is:
We want to get all countries in a single search. To achieve that we set the browser page to 1000, which is acomplished using JavaScript:
Can you write Python code for it?

js = "document.getElementById('page_size').options[1].text = '1000'"
driver.execute_script(js)


I am working on the file ".ipynb_checkpoints/dynamic_web_pages-checkpoint.ipynb".
The first task is:
When this command is run an empty browser window will pop up.
This is handy because with each command this window can be checked to see if Selenium worked as expected.
So next we load a web page in the browser using the get() method:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/educount-checkpoint.ipynb".
The first task is:

Language use in ECE

Statistics on language use in ECE, including tables on number of services, …

Can you write Python code for it?

edcounz = bs4.BeautifulSoup(eceall.text)

eduht = edcounz.findAll('a')

urldict = dict()

urldict.update({'text' : (edu.text)})
urldict.update({'url' : (edu.attrs['href'])})

#urldict.update('test':'blah')

urldict


I am working on the file ".ipynb_checkpoints/encontro04-checkpoint.ipynb".
The first task is:
Portanto, o algoritmo converge para uma direção que a multiplicação por $A$ não pode mudar. Isso corresponde à definição de autovetor dada acima!
Cabe enfatizar, porém, que nem toda matriz garante convergência.
Matriz de adjacência
Considere um grafo $(N, E)$ e uma matriz $A \in {0, 1}^{|N| \times |N|}$. Denotando por $\alpha_{ij}$ o elemento na linha $i$ e coluna $j$, dizemos que $A$ é a matriz de adjacência do grafo $(N, E)$ se:
$$\textrm{supondo } (N, E) \textrm{ não-dirigido}, \alpha_{ij} = 1 \textrm{ se } {i, j} \in E \textrm{ e } \alpha_{ij} = 0 \textrm{ caso contrário};$$
$$\textrm{supondo } (N, E) \textrm{ dirigido}, \alpha_{ij} = 1 \textrm{ se } (i, j) \in E \textrm{ e } \alpha_{ij} = 0 \textrm{ caso contrário}.$$
Vamos construir a matriz de adjacência de um grafo dos encontros anteriores.
Can you write Python code for it?

sn.graph_width = 320
sn.graph_height = 180

g = sn.load_graph('encontro02/3-bellman.gml', has_pos=True)

for n in g.nodes():
    g.node[n]['label'] = str(n)

sn.show_graph(g, nlab=True)

matrix = sn.build_matrix(g)

print(matrix)


I am working on the file ".ipynb_checkpoints/fcFinder Demo-checkpoint.ipynb".
The first task is:
Batches of Documents (not ready yet)
Can you write Python code for it?

### Need to work on this ###
def fcFinder_batches(batch_path, batch_schema,output_dir, targets=targets,modifiers=modifiers):
    batches = batch_path+'//%s'%batch_schema
    #return os.listdir(batch_path)
    listOfBatches = glob.glob(batches)
    counter = 0
    #return listOfBatches
    for batch in listOfBatches:
        #print(batch)
        corpus = os.path.join(batch,'corpus')
        for file in os.listdir(corpus):
            file_path = os.path.isfile(os.path.join(batch,corpus,file))
            #print(file)
            counter += 1
            if os.path.isfile(os.path.join(batch,corpus,file)):
                #print(file)
                if not os.path.exists(os.path.join(output_dir,batch)):
                    os.mkdir(os.path.exists(output_dir,bat))
                fcFinder(file_path,modifiers,targets,os.path.join(output_dir,batch))

    return counter
    return output_dir
    return listOfBatches
fcFinder_batches('/Users/alec/Box Sync/Adjudication','Batch*','/Users/alec/Desktop/fcFinder_test')

fcFinder(input_report, modifiers, targets, os.getcwd())

fcFinder(input_report2, modifiers, targets, os.getcwd())


I am working on the file ".ipynb_checkpoints/fcFinder Demo-checkpoint.ipynb".
The first task is:
One pipelined function for one file
Can you write Python code for it?

input_report2 = '/Users/alec/Box Sync/Adjudication/Batch_3/corpus/No_69411_129942_04-30-58.txt'
os.getcwd()

def fcFinder(input_report, output_dir, modifiers=modifiers, targets=targets):
    if not os.path.exists(output_dir):
        os.mkdir(os.path.join(output_dir,'saved'))
    report = ''
    with open(input_report,'r') as f0:
        report += f0.read()
    context = fc.create_context_doc(report, modifiers, targets)
    report_name = os.path.basename(input_report)
    annotations = fc.fluid_collection_classifier(context,input_report)
    XML_strings = [x.getXML() for x in annotations]
    return fc.writeKnowtator(XML_strings,report_name,os.path.join(output_dir,'saved'))

fcFinder(input_report,'/Users/alec/Desktop/fcFinder_test')


I am working on the file ".ipynb_checkpoints/fcFinder Demo-checkpoint.ipynb".
The first task is:
XML
This part of the pipeline takes all of the annotations from above and writes it into a Knowtator XML file. It is not yet able to be imported into eHOST
Can you write Python code for it?

#annotation1 = fcClasses.createAnnotations(definitive_mention,'fluid collection-definitive','Yes_28226_116465_05-29-93.txt')[0]
annotation1 = annotations1[0]
XML1 = annotation1.getXML()
print(XML1)

#XML1 = fcClasses.getXML(annotation1)

#print(XML1)
#importlib.reload(fc)

print(len(annotation_strings))
#print(len(fc.writeKnowtator(annotation_strings,report_name,os.path.join('..','XML_examples'))))
print(fc.writeKnowtator(annotation_strings,report_name,os.path.join('/Users/alec/Desktop/fcFinder_test','saved')))


I am working on the file ".ipynb_checkpoints/fcFinder Demo-checkpoint.ipynb".
The first task is:
fcFinder
I'm designing a function to work as a pipeline using the module fcFinder. Here is an outline of how the function works:
- As input, it takes:
    - a file name (the report)
    - an outpath to save the XML file
    - modifiers and targets as itemData objects

The function then follows this order, as demonstrated below:
Reads in the file
Creates a conText Document that is made up of markup objects
Classifies each tagObject as being either definitive evidence of fluid collection, negated evidence, or indication (need to get it to work for the relevant anatomic modifiers)
Creates mentionAnnotation objects out of those tabObjects
Writes one XML .txt.knowtator file. This still needs to be worked on so that it is compatible with eHOST, but the basic structure is there.
Can you write Python code for it?

#one annotation
modifiers = itemData.instantiateFromCSVtoitemData("/Users/alec/Box Sync/Bucher_Surgical_MIMICIII/pyConText_implement/modifiers.tsv")
targets = itemData.instantiateFromCSVtoitemData(
    "file:///Users/alec/Box Sync/Bucher_Surgical_MIMICIII/pyConText_implement/targets.tsv")

input_report = '/Users/alec/Box Sync/Bucher_Surgical_MIMICIII/Radiology_Annotation/Adjudication/Batch_3/corpus/Yes_28226_116465_05-29-93.txt'
report = ''
with open(input_report,'r') as f0:
    report += f0.read()
len(report)
report_name = os.path.basename(input_report)
report_name

context = fc.create_context_doc(report, modifiers, targets)

len(context.getSectionMarkups())

#importlib.reload(fc)

#Apply logic, classify tagObjects
annotations1=fc.fluid_collection_classifier(context,"textfile.txt")

annotation_strings = []
print(type(annotations1))
print(type(annotations1[0]))
for annotation in annotations1:
    annotation_strings.append(annotation.getXML())
    print(annotation.getMentionClass())
#definitive_mention = fc.fluid_collection_classifier(context)[0][0]
#definitive_mention.has_edge()
#Will this work to catch the appropriate anatomical modifiers?
#[x.getXML() for x in annotations1]

#print(type(definitive_mention))
#tagObject1 = definitive_mention.nodes()[0]
#tagObject1.getSpan()


I am working on the file ".ipynb_checkpoints/feat_extract-checkpoint.ipynb".
The first task is:
Build feature matrix
The matrix is a JD-Skill matrix where each entry $e(d, s)$ is the number of times skill $s$ occurs in job description $d$.
Can you write Python code for it?

doc_skill = buildDocSkillMat(jd_docs, skill_df, folder=SKILL_DIR)

with(open(SKILL_DIR + 'doc_skill.mtx', 'w')) as f:
    mmwrite(f, doc_skill)


I am working on the file ".ipynb_checkpoints/filter_user_reviews-checkpoint.ipynb".
The first task is:
Create a new dictionary with the following structure and then export as a json object:
{user id: [review, review, review], ..., user id: [review, review, review]}
Can you write Python code for it?

user_reviews = dict.fromkeys(user_list, 0)
for review in reviews.find():
    try:
        if user_reviews[review['_id']] == 0:
            print review['_id']
            print review
            break
    except KeyError:
        pass
#             user_reviews[review['_id']] = [review]
#         else:
#             user_reviews[review['_id']].append(review)

#     except KeyError:
#         pass

user_reviews[user_reviews.keys()[23]]

filtered_reviews = {}
for user in user_reviews.keys():
    if user_reviews[user] != 0:
        filtered_reviews[user] = user_reviews[user]

#We have this many users after our filtering
len(filtered_reviews)

#Dump file of cleaned up user data
with open('merged_user_reviews.json', 'w') as fp:
    json.dump(user_reviews, fp)


I am working on the file ".ipynb_checkpoints/filter_user_reviews-checkpoint.ipynb".
The first task is:
The business ID field has already been filtered for only restaurants
We want to filter the users collection for the following:
    1. User must have at least 20 reviews
    2. For users with 20 reviews, identify the reviews which are for businesses
    3. For each user, keep only those reviews which are related to a business in
    the list of restaurant business IDs
    4. Keep only users who have at least 20 reviews after finishing step 3
Can you write Python code for it?

#Find a list of users with at least 20 reviews
user_list = []
for user in users.find():
    if user['review_count'] >= 20:
        user_list.append(user['_id'])
    else:
        pass


I am working on the file ".ipynb_checkpoints/games-checkpoint.ipynb".
The first task is:
A random_player should never win against an alphabeta_player. Let's test that.
Can you write Python code for it?

for _ in range(10):
    print(play_game(ttt, random_player, alphabeta_player))


I am working on the file ".ipynb_checkpoints/games-checkpoint.ipynb".
The first task is:
Canvas_TicTacToe(Canvas)
This subclass is used to play TicTacToe game interactively in Jupyter notebooks. TicTacToe class is called while initializing this subclass.
Let's have match between random_player and alphabeta_player. Click on the board to call players to make a move.
Can you write Python code for it?

bot_play = Canvas_TicTacToe('bot_play', 'random', 'alphabeta')


I am working on the file ".ipynb_checkpoints/games-checkpoint.ipynb".
The first task is:
First we try out our random_player(game, state). Given a game state it will give us a random move every time:
Can you write Python code for it?

print(random_player(game52, 'A'))
print(random_player(game52, 'A'))


I am working on the file ".ipynb_checkpoints/games-checkpoint.ipynb".
The first task is:
GameState namedtuple
GameState is a namedtuple which represents the current state of a game. Let it be Tic-Tac-Toe or any other game.
Game class
Let's have a look at the class Game in our module. We see that it has functions, namely actions, result, utility, terminal_test, to_move and display.
We see that these functions have not actually been implemented. This class is actually just a template class; we are supposed to create the class for our game, TicTacToe by inheriting this Game class and implement all the methods mentioned in Game. Do not close the popup so that you can follow along the description of code below.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/games-checkpoint.ipynb".
The first task is:
Hmm, so that's the initial state of the game; no X's and no O's.
Let us create a new game state by ourselves to experiment:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/games-checkpoint.ipynb".
The first task is:
Note that, here, if you are the first player, the alphabeta_player plays as MIN, and if you are the second player, the alphabeta_player plays as MAX. This happens because that's the way the game is defined in the class Fig52Game. Having a look at the code of this class should make it clear.
TicTacToe game
Now let's play TicTacToe. First we initialize the game by creating an instance of the subclass TicTacToe inherited from the class Game:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/games-checkpoint.ipynb".
The first task is:
The alphabeta_player(game, state) will always give us the best move possible:
Can you write Python code for it?

print( alphabeta_player(game52, 'A') )
print( alphabeta_player(game52, 'B') )
print( alphabeta_player(game52, 'C') )


I am working on the file ".ipynb_checkpoints/games-checkpoint.ipynb".
The first task is:
The class TicTacToe has been inherited from the class Game. As mentioned earlier, you really want to do this. Catching bugs and errors becomes a whole lot easier.
Additional methods in TicTacToe:

__init__(self, h=3, v=3, k=3) :  When you create a class inherited from the Game class (class TicTacToe in our case), you'll have to create an object of this inherited class to initialize the game. This initialization might require some additional information which would be passed to __init__ as variables. For the case of our TicTacToe game, this additional information would be the number of rows h, number of columns v and how many consecutive X's or O's are needed in a row, column or diagonal for a win k. Also, the initial game state has to be defined here in __init__.

compute_utility(self, board, move, player) : A method to calculate the utility of TicTacToe game. If 'X' wins with this move, this method returns 1; if 'O' wins return -1; else return 0.

k_in_row(self, board, move, player, delta_x_y) : This method returns True if there is a line formed on TicTacToe board with the latest move else False.

GameState in TicTacToe game
Now, before we start implementing our TicTacToe game, we need to decide how we will be representing our game state. Typically, a game state will give you all the current information about the game at any point in time. When you are given a game state, you should be able to tell whose turn it is next, how the game will look like on a real-life board (if it has one) etc. A game state need not include the history of the game. If you can play the game further given a game state, you game state representation is acceptable. While we might like to include all kinds of information in our game state, we wouldn't want to put too much information into it. Modifying this game state to generate a new one would be a real pain then.
Now, as for our TicTacToe game state, would storing only the positions of all the X's and O's be sufficient to represent all the game information at that point in...
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/games-checkpoint.ipynb".
The first task is:
The output is -1, hence random_player loses implies alphabeta_player wins.
Since, an alphabeta_player plays perfectly, a match between two alphabeta_players should always end in a draw. Let's see if this happens:
Can you write Python code for it?

for _ in range(10):
    print(play_game(ttt, alphabeta_player, alphabeta_player))


I am working on the file ".ipynb_checkpoints/games-checkpoint.ipynb".
The first task is:
The random_player will behave how he is supposed to i.e. pseudo-randomly:
Can you write Python code for it?

random_player(ttt, my_state)

random_player(ttt, my_state)


I am working on the file ".ipynb_checkpoints/games-checkpoint.ipynb".
The first task is:
We can print a state using the display method:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/games-checkpoint.ipynb".
The first task is:
What the alphabeta_player does is, it simply calls the method alphabeta_full_search. They both are essentially the same. In the module, both alphabeta_full_search and minimax_decision have been implemented. They both do the same job and return the same thing, which is, the best move in the current state. It's just that alphabeta_full_search is more efficient w.r.t time because it prunes the search tree and hence, explores lesser number of states.
Can you write Python code for it?

minimax_decision('A', game52)

alphabeta_full_search('A', game52)


I am working on the file ".ipynb_checkpoints/games-checkpoint.ipynb".
The first task is:
Yay! We win. But we cannot win against an alphabeta_player, however hard we try.
Can you write Python code for it?

ab_play = Canvas_TicTacToe('ab_play', 'human', 'alphabeta')


I am working on the file ".ipynb_checkpoints/getTweetsByID-checkpoint.ipynb".
The first task is:
Setup Twitter Access
Can you write Python code for it?

# Twitter OAuth Credentials
consumer_key = ""
consumer_secret = ""
access_token = ""
access_secret = ""


I am working on the file ".ipynb_checkpoints/guardian_first_attempt-checkpoint.ipynb".
The first task is:
Create comment stemmer and TFIDF vectorizer
We will do some stemming in the comment text in order to create shorter vectors that represent each comment.
Can you write Python code for it?

from sklearn.feature_extraction.text import TfidfVectorizer
import nltk.stem

english_stemmer = nltk.stem.SnowballStemmer('english')

class StemmedTfidfVectorizer(TfidfVectorizer):
    def build_analyzer(self):
        analyzer=super(StemmedTfidfVectorizer,self).build_analyzer()
        return lambda doc:(english_stemmer.stem(w) for w in analyzer(doc))


I am working on the file ".ipynb_checkpoints/guardian_first_attempt-checkpoint.ipynb".
The first task is:
Vectorize extracted comments
Can you write Python code for it?

stem_vectorizer = StemmedTfidfVectorizer(min_df=1, stop_words='english')
stem_analyze = stem_vectorizer.build_analyzer()
# print [tok for tok in stem_analyze ("When we have a real living wage, there will no longer need to be 'stupid tax credits'. Until then, people need a top up to support themselves, because the companies they work for, don't want to give people their dues.")]

comment_vectors = stem_vectorizer.fit_transform(parsed_comments)

print "%d features found" % (len(stem_vectorizer.get_feature_names()))
print stem_vectorizer.get_feature_names()


I am working on the file ".ipynb_checkpoints/homework7_billionaire_shengyingzhao-checkpoint.ipynb".
The first task is:
'What is relationship to company'? And what are the most common relationships?
Can you write Python code for it?

recent['relationshiptocompany']

recent['relationshiptocompany'].describe()
# the most common relationship to company is founder


I am working on the file ".ipynb_checkpoints/homework7_billionaire_shengyingzhao-checkpoint.ipynb".
The first task is:
How old are billionaires? How old are billionaires self made vs. non self made? or different industries?
Can you write Python code for it?

recent.sort_values(by='age',ascending=False).head()

columns_want = recent[['name', 'age', 'selfmade','industry']]  #[[]]:dataframe
columns_want.head()


I am working on the file ".ipynb_checkpoints/homework7_billionaire_shengyingzhao-checkpoint.ipynb".
The first task is:
Most common source of wealth? Male vs. female?
Can you write Python code for it?

recent['sourceofwealth'].describe()
# the most common source of wealth is real estate

recent.groupby('gender')['sourceofwealth'].describe() #describe the content of a given column
# the most common source of wealth for male is real estate, while for female is diversified


I am working on the file ".ipynb_checkpoints/homework7_billionaire_shengyingzhao-checkpoint.ipynb".
The first task is:
What country are most billionaires from? For the top ones, how many billionaires per billion people?
Can you write Python code for it?

recent = df[df['year'] == 2014] #recent is a variable, a variable can be assigned to different things, here it was assigned to a data frame
recent.head()

recent.columns.values


I am working on the file ".ipynb_checkpoints/homework7_billionaire_shengyingzhao-checkpoint.ipynb".
The first task is:
What's the average wealth of a billionaire? Male? Female?
Can you write Python code for it?

recent['networthusbillion'].describe()
# the average wealth of a billionaire is $3.9 billion

recent.groupby('gender')['networthusbillion'].describe()#group by is a function, group everything by gender, and show the billionnetworth
# female mean is 3.920556 billion
# male mean is 3.902716 billion


I am working on the file ".ipynb_checkpoints/homework_5_shengying_zhao-checkpoint.ipynb".
The first task is:
Pick two of your favorite Lils to fight it out, and use their IDs to print out their top tracks
Can you write Python code for it?

for artist in artist_info:
    print(artist['name'], artist['id'])

#I chose Lil Fate and Lil' Flip, first I want to figure out the top track of Lil Fate
response = requests.get("https://api.spotify.com/v1/artists/6JUnsP7jmvYmdhbg7lTMQj/top-tracks?country=US")

print(response.text)

data = response.json()

type(data)

data.keys()

type(data['tracks'])

print(data['tracks'])

data['tracks'][0]

for item in data['tracks']:
    print(item['name'])

# now to figure out the top track of Lil' Flip #things within {} or ALL Caps means to replace them
response = requests.get("https://api.spotify.com/v1/artists/4Q5sPmM8j4SpMqL4UA1DtS/top-tracks?country=US")

print(response.text)

data = response.json()

type(data)

data.keys()

type(data['tracks'])

for item in data['tracks']:
    #type(item): dict
    #print(item.keys()), saw 'name'
    print(item['name'])


I am working on the file ".ipynb_checkpoints/homework_5_shengying_zhao-checkpoint.ipynb".
The first task is:
Since we're talking about Lils, what about Biggies? How many total "Biggie" artists are there? How many total "Lil"s? If you made 1 request every 5 seconds, how long would it take to download information on all the Lils vs the Biggies?
Can you write Python code for it?

response = requests.get('https://api.spotify.com/v1/search?q=Lil&type=artist&market=US')
all_lil = response.json()

print(response.text)

all_lil.keys()

all_lil['artists'].keys()

print(all_lil['artists']['total'])

response = requests.get('https://api.spotify.com/v1/search?q=Biggie&type=artist&market=US')
all_biggies = response.json()
print(all_biggies['artists']['total'])


I am working on the file ".ipynb_checkpoints/homework_5_shengying_zhao-checkpoint.ipynb".
The first task is:
Will the world explode if a musicians swears? Get an average popularity for their explicit songs vs. their non-explicit songs. How many minutes of explicit songs do they have? Non-explicit?
Can you write Python code for it?

#for Lil' fate's top tracks
explicit_count = 0
non_explicit_count = 0
popularity_explicit = 0
popularity_non_explicit = 0
minutes_explicit = 0
minutes_non_explicit = 0
for track in data['tracks']:
    if track['explicit']== True:

        explicit_count = explicit_count + 1
        popularity_explicit = popularity_explicit + track['popularity']
        minutes_explicit = minutes_explicit + track['duration_ms']

    elif track['explicit']== False:
        non_explicit_count = non_explicit_count + 1
        popularity_non_explicit = popularity_non_explicit + track['popularity']
        minutes_non_explicit = minutes_non_explicit + track['duration_ms']

print("Lil' Flip has", (minutes_explicit/1000)/60, "of explicit songs")
print("Lil' Flip has", (minutes_non_explicit/1000)/60, "of non-explicit songs")
print("The average popularity of Lil' Flip explicits songs is", popularity_explicit/explicit_count)
print("The average popularity of Lil' Flip non-explicits songs is", popularity_non_explicit/non_explicit_count)


I am working on the file ".ipynb_checkpoints/homework_5_shengying_zhao-checkpoint.ipynb".
The first task is:
print a list of Lil's that are more popular than Lil's Kim
Can you write Python code for it?

for artist in artist_info:
    print(artist['name'])
    if artist['name']== "Lil' Kim":
        print("Found Lil Kim")
        print(artist['popularity'])
    else:
        pass #print

Lil_kim_popularity = 62
more_popular_than_Lil_kim = []
for artist in artist_info:
    if artist['popularity'] > Lil_kim_popularity:
        #If yes, let's add them to our list
        print(artist['name'], "is more popular with a score of", artist['popularity'])
        more_popular_than_Lil_kim.append(artist['name'])
    else:
        print(artist['name'], "is less popular with a score of", artist['popularity'])
for artist_name in more_popular_than_Lil_kim:
    print(artist_name)


I am working on the file ".ipynb_checkpoints/iching_intro-checkpoint.ipynb".
The first task is:
First Change
Can you write Python code for it?

data = 50 - 1
sky, earth, firstChange, data = iching.getChange(data)
print sky, '\n', earth, '\n',firstChange, '\n', data


I am working on the file ".ipynb_checkpoints/iching_intro-checkpoint.ipynb".
The first task is:
Plot transitions
Can you write Python code for it?

%matplotlib inline
iching.plotTransition(100, w = 50)

%matplotlib inline
import matplotlib.pyplot as plt

fig = plt.figure(figsize=(15, 10),facecolor='white')
plt.subplot(2, 2, 1)
iching.plotTransition(1000, w = 50)
plt.subplot(2, 2, 2)
iching.plotTransition(1000, w = 50)
plt.subplot(2, 2, 3)
iching.plotTransition(1000, w = 50)
plt.subplot(2, 2, 4)
iching.plotTransition(1000, w = 50)


I am working on the file ".ipynb_checkpoints/iching_intro-checkpoint.ipynb".
The first task is:
Second Change
Can you write Python code for it?

sky, earth, secondChange, data = iching.getChange(data)
print sky, '\n', earth, '\n',secondChange, '\n', data


I am working on the file ".ipynb_checkpoints/if_statements-checkpoint.ipynb".
The first task is:
As you can see, the if-elif-else chain lets you respond in very specific ways to any given situation.

Exercises

Three is a Crowd

Make a list of names that includes at least four people.
Write an if test that prints a message about the room being crowded, if there are more than three people in your list.
Modify your list so that there are only two people in it. Use one of the methods for removing people from the list, don't just redefine the list.
Run your if test again. There should be no output this time, because there are less than three people in the list.
Bonus: Store your if test in a function called something like crowd_test.

Three is a Crowd - Part 2

Save your program from Three is a Crowd under a new name.
Add an else statement to your if tests. If the else statement is run, have it print a message that the room is not very crowded.

Six is a Mob

Save your program from Three is a Crowd - Part 2 under a new name.
Add some names to your list, so that there are at least six people in the list.
Modify your tests so that
If there are more than 5 people, a message is printed about there being a mob in the room.
If there are 3-5 people, a message is printed about the room being crowded.
If there are 1 or 2 people, a message is printed about the room not being crowded.
If there are no people in the room, a message is printed abou the room being empty.

top
More than one passing test
In all of the examples we have seen so far, only one test can pass. As soon as the first test passes, the rest of the tests are ignored. This is really good, because it allows our code to run more efficiently. Many times only one condition can be true, so testing every condition after one passes would be meaningless.
There are situations in which you want to run a series of tests, where every single test runs. These are situations where any or all of the tests could pass, and you want to respond to each passing test. Consider the following example, where we want to greet each dog that is present:
Can you write Python code for it?

dogs = ['willie', 'hootz']

if 'willie' in dogs:
    print("Hello, Willie!")
if 'hootz' in dogs:
    print("Hello, Hootz!")
if 'peso' in dogs:
    print("Hello, Peso!")
if 'monty' in dogs:
    print("Hello, Monty!")


I am working on the file ".ipynb_checkpoints/if_statements-checkpoint.ipynb".
The first task is:
If we had done this using an if-elif-else chain, only the first dog that is present would be greeted:
Can you write Python code for it?

###highlight=[6,7,8,9,10,11]
dogs = ['willie', 'hootz']

if 'willie' in dogs:
    print("Hello, Willie!")
elif 'hootz' in dogs:
    print("Hello, Hootz!")
elif 'peso' in dogs:
    print("Hello, Peso!")
elif 'monty' in dogs:
    print("Hello, Monty!")


I am working on the file ".ipynb_checkpoints/if_statements-checkpoint.ipynb".
The first task is:
In this case, we only print a message if there is at least one dog present. Of course, you could add a new else clause to respond to the situation in which there are no dogs at all:
Can you write Python code for it?

###highlight=[10,11]
dogs = []

if len(dogs) >= 5:
    print("Holy mackerel, we might as well start a dog hostel!")
elif len(dogs) >= 3:
    print("Wow, we have a lot of dogs here!")
elif len(dogs) >= 1:
    print("Okay, this is a reasonable number of dogs.")
else:
    print("I wish we had a dog here.")


I am working on the file ".ipynb_checkpoints/if_statements-checkpoint.ipynb".
The first task is:
In this situation, nothing happens if the test does not pass.
Can you write Python code for it?

###highlight=[2]
dogs = ['willie', 'hootz']

if len(dogs) > 3:
    print("Wow, we have a lot of dogs here!")


I am working on the file ".ipynb_checkpoints/if_statements-checkpoint.ipynb".
The first task is:
In this situation, the first two tests fail, so the statement in the else clause is executed. Note that this statement would be executed even if there are no dogs at all:
Can you write Python code for it?

###highlight=[2]
dogs = []

if len(dogs) >= 5:
    print("Holy mackerel, we might as well start a dog hostel!")
elif len(dogs) >= 3:
    print("Wow, we have a lot of dogs here!")
else:
    print("Okay, this is a reasonable number of dogs.")


I am working on the file ".ipynb_checkpoints/if_statements-checkpoint.ipynb".
The first task is:
Note that you don't have to take any action at all when you start a series of if statements. You could simply do nothing in the situation that there are no dogs by replacing the else clause with another elif clause:
Can you write Python code for it?

###highlight=[8]
dogs = []

if len(dogs) >= 5:
    print("Holy mackerel, we might as well start a dog hostel!")
elif len(dogs) >= 3:
    print("Wow, we have a lot of dogs here!")
elif len(dogs) >= 1:
    print("Okay, this is a reasonable number of dogs.")


I am working on the file ".ipynb_checkpoints/if_statements-checkpoint.ipynb".
The first task is:
Notice that there are no errors. The condition len(dogs) > 3 evaluates to False, and the program moves on to any lines after the if block.
if-else statements
Many times you will want to respond in two possible ways to a test. If the test evaluates to True, you will want to do one thing. If the test evaluates to False, you will want to do something else. The if-else structure lets you do that easily. Here's what it looks like:
Can you write Python code for it?

dogs = ['willie', 'hootz', 'peso', 'juno']

if len(dogs) > 3:
    print("Wow, we have a lot of dogs here!")
else:
    print("Okay, this is a reasonable number of dogs.")


I am working on the file ".ipynb_checkpoints/if_statements-checkpoint.ipynb".
The first task is:
Of course, this could be written much more cleanly using lists and for loops. See if you can follow this code.
Can you write Python code for it?

dogs_we_know = ['willie', 'hootz', 'peso', 'monty', 'juno', 'turkey']
dogs_present = ['willie', 'hootz']

# Go through all the dogs that are present, and greet the dogs we know.
for dog in dogs_present:
    if dog in dogs_we_know:
        print("Hello, %s!" % dog.title())


I am working on the file ".ipynb_checkpoints/if_statements-checkpoint.ipynb".
The first task is:
Our results have not changed in this case, because if the test evaluates to True only the statements under the if statement are executed. The statements under else area only executed if the test fails:
Can you write Python code for it?

###highlight=[2]
dogs = ['willie', 'hootz']

if len(dogs) > 3:
    print("Wow, we have a lot of dogs here!")
else:
    print("Okay, this is a reasonable number of dogs.")


I am working on the file ".ipynb_checkpoints/if_statements-checkpoint.ipynb".
The first task is:
The test evaluated to False, so only the statement under else is run.
if-elif...else chains
Many times, you will want to test a series of conditions, rather than just an either-or situation. You can do this with a series of if-elif-else statements
There is no limit to how many conditions you can test. You always need one if statement to start the chain, and you can never have more than one else statement. But you can have as many elif statements as you want.
Can you write Python code for it?

dogs = ['willie', 'hootz', 'peso', 'monty', 'juno', 'turkey']

if len(dogs) >= 5:
    print("Holy mackerel, we might as well start a dog hostel!")
elif len(dogs) >= 3:
    print("Wow, we have a lot of dogs here!")
else:
    print("Okay, this is a reasonable number of dogs.")


I am working on the file ".ipynb_checkpoints/if_statements-checkpoint.ipynb".
The first task is:
This is the kind of code you should be aiming to write. It is fine to come up with code that is less efficient at first. When you notice yourself writing the same kind of code repeatedly in one program, look to see if you can use a loop or a function to make your code more efficient.
top
True and False values
Every value can be evaluated as True or False. The general rule is that any non-zero or non-empty value will evaluate to True. If you are ever unsure, you can open a Python terminal and write two lines to find out if the value you are considering is True or False. Take a look at the following examples, keep them in mind, and test any value you are curious about. I am using a slightly longer test just to make sure something gets printed each time.
Can you write Python code for it?

###highlight=[2]
if 0:
    print("This evaluates to True.")
else:
    print("This evaluates to False.")

###highlight=[2]
if 1:
    print("This evaluates to True.")
else:
    print("This evaluates to False.")

###highlight=[2,3]
# Arbitrary non-zero numbers evaluate to True.
if 1253756:
    print("This evaluates to True.")
else:
    print("This evaluates to False.")

###highlight=[2,3]
# Negative numbers are not zero, so they evaluate to True.
if -1:
    print("This evaluates to True.")
else:
    print("This evaluates to False.")

###highlight=[2,3]
# An empty string evaluates to False.
if '':
    print("This evaluates to True.")
else:
    print("This evaluates to False.")

###highlight=[2,3]
# Any other string, including a space, evaluates to True.
if ' ':
    print("This evaluates to True.")
else:
    print("This evaluates to False.")

###highlight=[2,3]
# Any other string, including a space, evaluates to True.
if 'hello':
    print("This evaluates to True.")
else:
    print("This evaluates to False.")

###highlight=[2,3]
# None is a special object in Python. It evaluates to False.
if None:
    print("This evaluates to True.")
else:
    print("This evaluates to False.")


I am working on the file ".ipynb_checkpoints/if_statements-checkpoint.ipynb".
The first task is:
What happens in this program?

The program starts out with a list of desserts, and one dessert is identified as a favorite.
The for loop runs through all the desserts.
Inside the for loop, each item in the list is tested.
If the current value of dessert is equal to the value of favorite_dessert, a message is printed that this is my favorite.
If the current value of dessert is not equal to the value of favorite_dessert, a message is printed that I just like the dessert.

You can test as many conditions as you want in an if statement, as you will see in a little bit.
Logical Tests
Every if statement evaluates to True or False. True and False are Python keywords, which have special meanings attached to them. You can test for the following conditions in your if statements:

equality (==)
inequality (!=)
other inequalities
greater than (>)
greater than or equal to (>=)
less than (<)
less than or equal to (<=)

You can test if an item is in a list.

Whitespace
Remember learning about PEP 8? There is a section of PEP 8 that tells us it's a good idea to put a single space on either side of all of these comparison operators. If you're not sure what this means, just follow the style of the examples you see below.
Equality
Two items are equal if they have the same value. You can test for equality between numbers, strings, and a number of other objects which you will learn about later. Some of these results may be surprising, so take a careful look at the examples below.
In Python, as in many programming languages, two equals signs tests for equality.
Watch out! Be careful of accidentally using one equals sign, which can really throw things off because that one equals sign actually sets your item to the value you are testing for!
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/if_statements-checkpoint.ipynb".
The first task is:
greater than or equal to
Can you write Python code for it?