I am working on the file "MoL_June15/lab-cky.ipynb".
The first task is:
CKY parser
We will implement a generalised CKY which can deal with an arbitrary epsilon-free PCFG.
Item
We are using dotted rules to represent items in CKY. A dotted rule is basically a context-free production agumented with a list of positions of the input sentence.
Can you write Python code for it?

class Item(object):  # this class is also available in `item.py`
    """A dotted rule used in CKY/Earley."""

    def __init__(self, rule, dots):
        assert len(dots) > 0, 'I do not accept an empty list of dots'
        self.rule_ = rule
        self.dots_ = tuple(dots)

    def __eq__(self, other):
        return self.rule_ == other.rule_ and self.dots_ == other.dots_

    def __ne__(self, other):
        return not(self == other)

    def __hash__(self):
        return hash((self.rule_, self.dots_))

    def __repr__(self):
        return '{0} ||| {1}'.format(self.rule_, self.dots_)

    def __str__(self):
        return '{0} ||| {1}'.format(self.rule_, self.dots_)

    @property
    def lhs(self):
        return self.rule_.lhs

    @property
    def rule(self):
        return self.rule_

    @property
    def dot(self):
        return self.dots_[-1]

    @property
    def start(self):
        return self.dots_[0]

    @property
    def next(self):
        """return the symbol to the right of the dot (or None, if the item is complete)"""
        if self.is_complete():
            return None
        return self.rule_.rhs[len(self.dots_) - 1]

    def state(self, i):
        return self.dots_[i]

    def advance(self, dot):
        """return a new item with an extended sequence of dots"""
        return Item(self.rule_, self.dots_ + (dot,))

    def is_complete(self):
        """complete items are those whose dot reached the end of the RHS sequence"""
        return len(self.rule_.rhs) + 1 == len(self.dots_)

r = Rule('[S]', ['[X]'], 0.0)
i1 = Item(r, [0])
i2 = i1.advance(1)

print i1
print i2

i1 != i2

i1.is_complete()

i2.is_complete()

i1.next

i2.next


I am working on the file "MoL_June15/lab-cky.ipynb".
The first task is:
Complete program
Start with axioms and exhaustively apply inference rules
Can you write Python code for it?

def cky(cfg, sentence):
    A = Agenda()
    for item in axioms(cfg, sentence):
        A.push(item)
    while A:
        item = A.pop()
        if item.is_complete() or is_nonterminal(item.next):
            for new in complete(item, A):
                A.push(new)
        else:
            new = scan(item, sentence)
            if new is not None:
                A.push(new)
        A.make_passive(item)
    return make_forest(A.itercomplete())

forest = cky(G1, sentence)

print forest


I am working on the file "MoL_June15/lab-cky.ipynb".
The first task is:
Forest from complete items
Each complete item in the (passive) agenda can be mapped to a new CFG rule (annotated with input spans).
For example, an item such as
    [X -> A x B *, [0,1,2,3]]

results in the rule
    X:0-3 -> A:0-1 x B:2-3

observe how only nonterminal nodes get annotated.
Can you write Python code for it?

def make_forest(complete_items):
    forest = WCFG()
    for item in complete_items:
        lhs = make_symbol(item.lhs, item.start, item.dot)
        rhs = []
        for i, sym in enumerate(item.rule.rhs):
            rhs.append(make_symbol(sym, item.state(i), item.state(i + 1)))
        forest.add(Rule(lhs, rhs, item.rule.prob))
    return forest


I am working on the file "MoL_June15/lab-cky.ipynb".
The first task is:
Goal
We know that the goal symbol is basically the original start symbol spanning from 0 to n (the length of the sentence).
Can you write Python code for it?

goal = make_symbol('[E]', 0, len(sentence))
goal


I am working on the file "MoL_June15/lab-cky.ipynb".
The first task is:
Let's see how it works
Can you write Python code for it?


I am working on the file "MoL_June15/lab-cky.ipynb".
The first task is:
Optimised version
For every rule X -> w_i alpha such that w_i is an input word at position i, we have an item of the kind:

[X -> * w_i alpha, i - 1, i - 1]

which gets represented as

X -> w_i alpha, [i-1]

Don't forget that these optimised axioms require an additional inference rule (some sort of delayed axioms which resembles a combination of prediction with completion).
Can you write Python code for it?

def axioms2(cfg, sentence):
    """
    :params cfg: a context-free grammar (an instance of WCFG)
    :params sentence: the input sentence (as a list or tuple)
    :returns: a list of items
    """
    by_rhs0 = defaultdict(list)
    for rule in cfg:
        by_rhs0[rule.rhs[0]].append(rule)
    items = []
    for i, word in enumerate(sentence):
        for rule in by_rhs0.get(word, []):
            items.append(Item(rule, [i]))
    return items

axioms2(G1, sentence)


I am working on the file "MoL_June15/lab-cky.ipynb".
The first task is:
and the agenda will make sure there are no duplicates
Can you write Python code for it?


I am working on the file "MoL_June15/lab-cky.ipynb".
The first task is:
we can push items into the agenda
Can you write Python code for it?

A.push(Item(r1, [0]))  # S -> S X, [0]  (earley axiom)


I am working on the file "MoL_June15/lab-inference.ipynb".
The first task is:
Counting derivations
Another interesting question is

how many analyses of a given sentence do we have?

This question is very simple to answer for acyclic hypergraphs and it turns out to be a special case of the inside recursion.
    N(v) =
        1                           if v is terminal
        0                           if v is nonterminal and BS(v) is empty
        \sum_{e \in BS(v)} 1 * \prod_{u \in tail(e)} N(u)
                                    otherwise

Compare the definition above with the inside recursion presented earlier.
Also compare the program below with the inside computation and comment on the differences.
Can you explain this recursion?
Can you write Python code for it?

def counting(forest, start):  # acyclic hypergraph
    N = dict()

    def get_count(symbol):
        w = N.get(symbol, None)
        if w is not None:
            return w
        incoming = forest.get(symbol, set())
        if len(incoming) == 0:  # terminals have already been handled, this must be a nonterminal dead end
            N[symbol] = w
            return 0
        w = 0
        for rule in incoming:
            k = 1
            for child in rule.rhs:
                k *= get_count(child)
            w += k
        N[symbol] = w
        return w

    # handles terminals
    for sym in forest.terminals:
        N[sym] = 1
    # handles nonterminals
    #for sym in forest.nonterminals:
    #    get_inside(sym)
    get_count(start)

    return N

N = counting(forest, goal)


I am working on the file "MoL_June15/lab-inference.ipynb".
The first task is:
Empirical distribution
A nice thing to do whe we can sample is to obtain an empirical distribution.
That is, we draw a number of independent samples from the underlying distribution and approximate their probabilities by their relative frequency in the sample. The law of large numbers says that our estimates converge to true probabilities as we sample more. That also holds for expectations that we might want to compute based on the underlying distribution.
Note how our estimates are pretty close to the true probabilities. This is because we are drawing indepent samples from the exact conditional distribution p(d|x).
Can you write Python code for it?

# an example of how to estimate an empirical distribution out of 100 samples
from collections import defaultdict
counts = defaultdict(int)
n_samples = 1000
# here we sample a number of derivations
for i in range(n_samples):
    d = tuple(sample(forest, I, goal))
    counts[d] += 1  # counting how often we they get sampled

# here we sort them by frequency from most frequent to least frequent
for d, n in sorted(counts.iteritems(), key=lambda (d, n): n, reverse=True):
    # here we compute the exact probability (for comparison)
    prob = joint_probability(d)/I[goal]
    # here we compute an empirical estimate
    estimate = float(n)/n_samples
    t = make_nltk_tree(d)
    print '# n=%d prob=%s estimate=%s\n%s' % (n, prob, estimate, str(t))
    print


I am working on the file "MoL_June15/lab-inference.ipynb".
The first task is:
Here we draw a random sample
Can you write Python code for it?


I am working on the file "MoL_June15/lab-inference.ipynb".
The first task is:
Reminder: the goal symbol after parsing is the original start symbol annotated from 0 to n (the length of the sentence).
Can you write Python code for it?


I am working on the file "MoL_June15/lab-inference.ipynb".
The first task is:
The conditional probability p(d|x) is given by
p(d|x) = p(d,x)/p(x)

and we know that
p(x) = \sum_d p(d,x)

is given by the inside at the root.
Thus, the following is the conditional probability of the best derivation:
Can you write Python code for it?

joint_probability(d)/I[goal]

# We can use this utilitary method to draw trees
from util import make_nltk_tree

t = make_nltk_tree(d)
print t


I am working on the file "MoL_June15/lab-inference.ipynb".
The first task is:
The inside at the root represents the probability of the sentence
p(x) = \sum_d p(x, d)
Can you write Python code for it?


I am working on the file "MoL_June15/lab-inference.ipynb".
The first task is:
The total number of derivations is associated with the value of N at the root.
Can you write Python code for it?


I am working on the file "Mobile_customer.ipynb".
The first task is:
2. Изучим данные и обработаем пропущенные значения
Can you write Python code for it?

train.info()

test.info()

# определим функцию для заполнения пропущенных значений
def replace_nan(data):
    # в столбцах 'START_PACK' и 'OFFER_GROUP' заменим NaN на 'Unknown'
    data['START_PACK'] = data['START_PACK'].fillna('Unknown')
    data['OFFER_GROUP'] = data['OFFER_GROUP'].fillna('Unknown')

    # столбцы с датами приведем к формату datetime
    data['ACT_DATE'] = pd.to_datetime(data['ACT_DATE'], format='%Y-%m-%d', errors='ignore')
    data['BIRTHDAY'] = pd.to_datetime(data['BIRTHDAY'], format='%Y-%m-%d', errors='ignore')

    # в столбце GENDER заменим NaN на M, так как 16034 из 28600 записей имеют значение M
    data['GENDER'] = data['GENDER'].fillna('M')

    # по условию задачи, NaN в столбце 'MLLS_STATE' означает что абонент не является участником программы лояльности
    data['MLLS_STATE'] = data['MLLS_STATE'].fillna('No')

    # по условиям задачи NaN в столбце 'OBLIG_NUM' означает, что абонент не пользовался рассрочкой
    data['OBLIG_NUM'] = data['OBLIG_NUM'].fillna(0.0)

    # NaN в столбце 'ASSET_TYPE_LAST' вероятно означает, что абонент не приобретал оборудование в компании
    data['ASSET_TYPE_LAST'] = data['ASSET_TYPE_LAST'].fillna('Not buying')

    # в столбце 'USAGE_AREA' заменим NaN на 'Undefined'
    data['USAGE_AREA'] = data['USAGE_AREA'].fillna('Undefined')

    # в остальных столбцах заменим NaN  на 0.0, считая что отсутствие данных означает отсутствие активности
    data['REFILL_OCT_16'] = data['REFILL_OCT_16'].fillna(0.0)
    data['REFILL_NOV_16'] = data['REFILL_NOV_16'].fillna(0.0)
    data['OUTGOING_OCT_16'] = data['OUTGOING_OCT_16'].fillna(0.0)
    data['OUTGOING_NOV_16'] = data['OUTGOING_NOV_16'].fillna(0.0)
    data['GPRS_OCT_16'] = data['GPRS_OCT_16'].fillna(0.0)
    data['GPRS_NOV_16'] = data['GPRS_NOV_16'].fillna(0.0)
    data['REVENUE_OCT_16'] = data['REVENUE_OCT_16'].fillna(0.0)
    data['REVENUE_NOV_16'] = data['REVENUE_NOV_16'].fillna(0.0)

# переведем BYR  в BYN
def byr_to_byn(data):
    data['REFILL_OCT_16'] =...


I am working on the file "Mobile_customer.ipynb".
The first task is:
4. Построим первую модель на всех признаках
Can you write Python code for it?

# ансамбль классификаторов методом Weighted Average Probabilities
clf1 = LogisticRegression(random_state=42)
clf2 = RandomForestClassifier(random_state=42)
clf3 = SGDClassifier(loss='log', random_state=42)

eclf = VotingClassifier(estimators=[('lr', clf1), ('rf', clf2), ('sgd', clf3)], voting='soft', weights=[1,1,1])

# проверка качества модели кросс-валидацией с вычислением ROC AUC на всех признаках
for clf, label in zip([clf1, clf2, clf3, eclf],
                      ['Logistic Regression', 'Random Forest', 'SGD', 'Ensemble']):
    scores2 = cross_val_score(estimator=clf, X=X_train, y=y_train, cv=10, scoring='roc_auc')
    print("ROC AUC: %0.6f (+/- %0.6f) [%s]" % (scores2.mean(), scores2.std(), label))


I am working on the file "Mobile_customer.ipynb".
The first task is:
7. Построение классификатора по тестовым данным
Can you write Python code for it?

# roc curve по тестовым данным
colors = ['black', 'orange', 'blue', 'green']
linestyles = [':', '--', '-.', '-']
for clf, label, clr, ls in zip([clf1, clf2, clf3, eclf],
                               ['Logistic Regression', 'Random Forest', 'SGD', 'Ensemble'],
                               colors, linestyles):
    y_pred = clf.fit(X_train[best_features], y_train).predict_proba(X_test[best_features])[:, 1]
    fpr, tpr, thresholds = roc_curve(y_true=y_test, y_score=y_pred)
    roc_auc = auc(x=fpr, y=tpr)
    plt.plot(fpr, tpr, color=clr, linestyle=ls, label='%s (auc = %0.2f)' % (label, roc_auc))
plt.legend(loc='lower right')
plt.plot([0, 1], [0, 1], linestyle='--', color='gray', linewidth=2)
plt.xlim([-0.1, 1.1])
plt.ylim([-0.1, 1.1])
plt.grid()
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.show()


I am working on the file "Mobile_customer.ipynb".
The first task is:
На тренировочных данных наилучший результат дает ансамбль из трех алгоритмов
5. Проверим важность признаков методом Random Forest
Can you write Python code for it?

# Построим лес и подсчитаем важность признаков
forest = ExtraTreesClassifier(n_estimators=250,
                              random_state=0)

forest.fit(X_train, y_train)
importances = forest.feature_importances_
std = np.std([tree.feature_importances_ for tree in forest.estimators_],
             axis=0)
indices = np.argsort(importances)[::-1]

# Выведем ранг признаков по важности
print("Feature ranking:")

for f in range(X_train.shape[1]):
    print("%d. %s (%f)" % (f + 1, list(X_train.columns)[indices[f]], importances[indices[f]]))

# Сделаем график важности признаков
plt.figure()
plt.title("Feature importances")
plt.bar(range(X_train.shape[1]), importances[indices],
       color="r", yerr=std[indices], align="center")
plt.xticks(range(X_train.shape[1]), indices)
plt.xlim([-1, X_train.shape[1]])
plt.show()


I am working on the file "Mobile_customer.ipynb".
The first task is:
Показатели ROC AUC полученые на кросс валидации и на тестовой выборке совпадают, что говорит о том, что модель не переобучена и не недообучена.
8. Получение итогового результата
Can you write Python code for it?

result_pred = eclf.fit(X_train[best_features], y_train).predict_proba(test[best_features])
result = pd.DataFrame(test['USER_ID'])
result['ACTIVITY_DEC_16_PROB'] = list(result_pred[:, 1])
result.to_csv('result.csv', encoding='utf8', index=None)


I am working on the file "Mobile_customer.ipynb".
The first task is:
Теперь у нас есть наборы данных test и train без отсутствующих данных и с несколькими новыми признаками
3. Подготовка данных для машинного обучения
Can you write Python code for it?

# преобразование категориальных данных
le = LabelEncoder()
for n in ['STATUS', 'TP_CURRENT', 'START_PACK', 'OFFER_GROUP', 'GENDER', 'MLLS_STATE',
         'PORTED_IN', 'PORTED_OUT', 'OBLIG_ON_START', 'ASSET_TYPE_LAST', 'DEVICE_TYPE_BUS', 'USAGE_AREA']:
    le.fit(train[n])
    train[n] = le.transform(train[n])
    test[n] = le.transform(test[n])

# стандартизация данных
features = list(train.columns)
del features[0]
del features[22]
scaler = StandardScaler()
for n in features:
    scaler.fit(train[n])
    train[n] = scaler.transform(train[n])
    test[n] = scaler.transform(test[n])

# разбиваем train на тренировочный и тестовый набор
X_train, X_test, y_train, y_test = train_test_split(train[features],
                                                    train.ACTIVITY_DEC_16,
                                                    test_size=0.20,
                                                    random_state=123)


I am working on the file "Mod_Fun.ipynb".
The first task is:

As every grade schooler knows, if at all aware of their heritage, said VE inscribes inside the octahedron of volume 4, as does an Icosahedron with flush faces.  We do a kind of jitterbugging that makes the VE larger instead of smaller.  Two applications of the S Factor does the trick.
Can you write Python code for it?

skew_icosa = small_ve * s_factor * s_factor

skew_icosa

skew_icosa + (24 * smod)


I am working on the file "Mod_Fun.ipynb".
The first task is:
Another way to reach the SmallGuy is to start with the volume 20 cubocta and shrink its edges by the S Factor, which means volume shrinks by a factor of the reciprocal of said S Factor to the 3rd power or $1/s_factor ^{3}$
Can you write Python code for it?

ve * (one/s_factor)**3

SmallGuy_edge = two * (one/s_factor) # effect on edges
SmallGuy_edge


I am working on the file "Mod_Fun.ipynb".
The first task is:
In showing off the Decimal type, I'm advertising high precision, but not "infinite precision".  Please be tolerant of our epsilons (tiny abberations).
Can you write Python code for it?


I am working on the file "Mod_Fun.ipynb".
The first task is:
JITTERBUG TRANSFORMATION
We call this the S Factor by the way.  VE:Icosa :: S:E is what to remember.  VE is the 12-around-1 nuclear sphere based agglomeration whereas Icosa is dervied from Jitterbugging, a mathematical transformation with a more technical name if you're a math snob (I can be).

Can you write Python code for it?

two * sqrt2 * (Ø**-2)

icosa = five * sqrt2 * Ø ** 2
icosa

ve = Decimal(20)

s_factor = ve / icosa

s_factor  # see?


I am working on the file "Mod_Fun.ipynb".
The first task is:
RHOMBIC TRIACONTAHEDRON (RT)

Can you write Python code for it?


I am working on the file "Mod_Fun.ipynb".
The first task is:
S3 is our conversion constant for going between XYZ cube volumes and IVM tetra volumes.  The two mensuration systems each have their own unit volume, by convention a .5 radius edge cube versus a 1.0 diametered edged tetrahedron, or use edges 1 and 2 if preferred, their ratio will be the same, with the cube a bit bigger.

SuperRT is the RT (rhombic triacontahedron) formed by the Icosa and its dual, the Pentagonal Dodecahedron, the two five-fold symmetric shapes in the Platonic set of five polys.  The Icosa we're talking about is the one above, derived from the VE of volume 20, through Jitterbugging.
If we shrink SuperRT down by $\phi^{-3}$ volume-wise (all edges are now $\phi^{-1}$ their initial length), and carve it into 120 modules (60 left, 60 right), then lo and behold, we have the E modules.
Another expression for SuperRT volume is $15\sqrt{2}$.
Can you write Python code for it?


I am working on the file "Mod_Fun.ipynb".
The first task is:
The S factor again, yes?
$\sqrt{2}-(\sqrt{2}(\phi^{-3}))= 2\sqrt{2}(\phi^{-2})$ = S Factor.
Another expression for the S Factor is $24E + 8e3$ where E means emod, and $e3$ means $E * \phi^{-3}$.
Can you write Python code for it?

Decimal(24) * emod + Decimal(8) * emod * Ø**-3


I am working on the file "Model Evaluation/Model Evaluation .ipynb".
The first task is:
Bagging KNN Classifier. Resulted in a small decrease in the score (from .944 to .940).
Can you write Python code for it?

from sklearn.ensemble import BaggingClassifier
baggingknn = BaggingClassifier(KNeighborsClassifier())

evaluate_model(baggingknn)

bagging_params = {'n_estimators': [10, 20],
                  'max_samples': [0.7, 1.0],
                  'max_features': [0.7, 1.0],
                  'bootstrap_features': [True, False]}

gsbaggingknn = GridSearchCV(baggingknn,
                            bagging_params, n_jobs=-1,
                            cv=KFold(len(y), n_folds=3, shuffle=True))

gsbaggingknn.fit(X, y)

gsbaggingknn.best_params_

various_models['gsbaggingknn'] = {'model': gsbaggingknn.best_estimator_,
                     'score': evaluate_model(gsbaggingknn.best_estimator_)}


I am working on the file "Model Evaluation/Model Evaluation .ipynb".
The first task is:
Checking the unique values for each of the columns.
Can you write Python code for it?

print df.buying.unique()

print df.maint.unique()

print df.doors.unique()

print df.persons.unique()

print df.lug_boot.unique()

print df.safety.unique()

print df.acceptability.unique()


I am working on the file "Model Evaluation/Model Evaluation .ipynb".
The first task is:
Creating a dataframe to compare the models.
Can you write Python code for it?

scores = pd.DataFrame([(k, v['score']) for k, v in various_models.iteritems()],
             columns=['model', 'score']).set_index('model').sort_values('score', ascending=False)

plt.style.use('fivethirtyeight')
scores.plot(kind='bar')
plt.ylim(0.5, 1.05)

scores


I am working on the file "Model Evaluation/Model Evaluation .ipynb".
The first task is:
Decision Trees are next.
Can you write Python code for it?

from sklearn.tree import DecisionTreeClassifier

dt = DecisionTreeClassifier()
various_models['dt'] = {'model': dt,
                    'score': evaluate_model(dt)}

params = {'criterion': ['gini', 'entropy'],
          'splitter': ['best', 'random'],
          'max_depth': [None, 5, 10],
          'min_samples_split': [2, 5],
          'min_samples_leaf': [1, 2, 3]}

gsdt = GridSearchCV(dt,
                    params, n_jobs=-1,
                    cv=KFold(len(y), n_folds=3, shuffle=True))

gsdt.fit(X, y)
print gsdt.best_params_
print gsdt.best_score_

various_models['gsdt'] = {'model': gsdt.best_estimator_,
                      'score': evaluate_model(gsdt.best_estimator_)}

gsbaggingdt = GridSearchCV(BaggingClassifier(gsdt.best_estimator_),
                           bagging_params, n_jobs=-1,
                           cv=KFold(len(y), n_folds=3, shuffle=True))

gsbaggingdt.fit(X, y)

print gsbaggingdt.best_params_
print gsbaggingdt.best_score_

various_models['gsbaggingdt'] = {'model': gsbaggingdt.best_estimator_,
                             'score': evaluate_model(gsbaggingdt.best_estimator_)}


I am working on the file "Model Evaluation/Model Evaluation .ipynb".
The first task is:
KNN Classifier
Can you write Python code for it?

from sklearn.neighbors import KNeighborsClassifier

a = evaluate_model(KNeighborsClassifier())

from sklearn.grid_search import GridSearchCV

params = {'n_neighbors': range(2,60)}

gsknn = GridSearchCV(KNeighborsClassifier(),
                     params, n_jobs=-1,
                     cv=KFold(len(y), n_folds=3, shuffle=True))

gsknn.fit(X, y)

gsknn.best_params_

gsknn.best_score_

evaluate_model(gsknn.best_estimator_)

various_models['knn'] = {'model': gsknn.best_estimator_,
                     'score': a}


I am working on the file "Model Evaluation/Model Evaluation .ipynb".
The first task is:
Now moving onto Logistic Regression.
Can you write Python code for it?

from sklearn.linear_model import LogisticRegression

lr = LogisticRegression()
various_models['lr'] = {'model': lr,
                    'score': evaluate_model(lr)}

params = {'C': [0.001, 0.01, 0.1, 1.0, 10.0, 100.0],
          'penalty': ['l1', 'l2']}

gslr = GridSearchCV(lr,
                    params, n_jobs=-1,
                    cv=KFold(len(y), n_folds=3, shuffle=True))

gslr.fit(X, y)

print gslr.best_params_
print gslr.best_score_

various_models['gslr'] = {'model': gslr.best_estimator_,
                             'score': evaluate_model(gslr.best_estimator_)}

gsbagginglr = GridSearchCV(BaggingClassifier(gslr.best_estimator_),
                           bagging_params, n_jobs=-1,
                           cv=KFold(len(y), n_folds=3, shuffle=True))

gsbagginglr.fit(X, y)

print gsbagginglr.best_params_
print gsbagginglr.best_score_

various_models['gsbagginglr'] = {'model': gsbagginglr.best_estimator_,
                             'score': evaluate_model(gsbagginglr.best_estimator_)}


I am working on the file "Model Evaluation/Model Evaluation .ipynb".
The first task is:
Train test split and creating a function to evaluate the models being created next.
Can you write Python code for it?

from sklearn.cross_validation import train_test_split, KFold
from sklearn.metrics import accuracy_score, precision_score, recall_score, confusion_matrix, classification_report

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42, stratify=y)

def evaluate_model(model):
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)

    a = accuracy_score(y_test, y_pred)

    cm = confusion_matrix(y_test, y_pred)
    cr = classification_report(y_test, y_pred)

    print cm
    print cr

    return a

various_models = {}


I am working on the file "Model Evaluation/Model Evaluation .ipynb".
The first task is:
Using the information in the cell above, maps will be used to create a scale.
Can you write Python code for it?


I am working on the file "Model Evaluation/Model_Evaluation-exercise.ipynb".
The first task is:
2 - Build a pipeline for polynomial fitting, fit polynomials of degree 1 to the number of features, and plot your training and testing errors for each.  Comment on your results
Can you write Python code for it?

# scaling+pol+linear regression
pipe = Pipeline([
    ('scl', StandardScaler()),
    ('poly', PolynomialFeatures()),
    ('lr', LinearRegression())
])

scores = []
# looping through degrees
for n in range(1, Xtrain.shape[1]+1):
    # set pipeline params
    pipe.set_params(poly__degree=n)
    # fit and score
    pipe.fit(Xtrain, ytrain)
    scores.append([n, pipe.score(Xtrain, ytrain), pipe.score(Xtest, ytest),
                   mean_squared_error(pipe.predict(Xtrain), ytrain), mean_squared_error(pipe.predict(Xtest), ytest)])

scores = pd.DataFrame(scores)
scores.columns = ['degree', 'train score', 'test score', 'train mse', 'test mse']

scores

# plot on two separate axes cause the scales are very different
fig, ax = plt.subplots(2, figsize=(10, 8))
ax[0].plot(scores.degree, scores['train score'], label='train score')
ax[1].plot(scores.degree, scores['test score'], label='test score');

# single plot for n=1,2,3
fig, ax = plt.subplots(figsize=(10, 8))
ax.plot(scores.iloc[:3, 0], scores.iloc[:3, 1], label='train score')
ax.plot(scores.iloc[:3, 0], scores.iloc[:3, 2], label='test score');


I am working on the file "Model Evaluation/Model_Evaluation-exercise.ipynb".
The first task is:
2 - Fit a machine learning algorithm of your choice to the data, tuning hyperparameters using the Better Holdout Method.  Generate training and validation plots and comment on your results.
Can you write Python code for it?

# in solution he encodes all variables once using df.apply(le.fit_transform)


I am working on the file "Model Evaluation/Model_Evaluation-exercise.ipynb".
The first task is:
2 - Fit a machine learning algorithm of your choice to the data, tuning hyperparameters using the Better Holdout Method. Generate training and validation plots and comment on your results.
Can you write Python code for it?

# scaling and ridge regression
pipe = Pipeline([
    ('stsc', StandardScaler()),
    ('ridge', Ridge())
])

# better hold out
Xtr, Xcv, ytr, ycv = train_test_split(Xtrain, ytrain, test_size=0.2, random_state=42)
scores = []
# looping through regularization values
for a in [0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1, 3, 10, 30, 100, 300, 1000, 3000]:
    pipe.set_params(ridge__alpha=a)
    pipe.fit(Xtr, ytr)
    tr_score = pipe.score(Xtr, ytr)
    cv_score = pipe.score(Xcv, ycv)
    scores.append([a, tr_score, cv_score])

scores = np.array(scores)

scores

fig, ax = plt.subplots(figsize=(16, 8), subplot_kw=dict(xscale='log'))

ax.plot(scores[:, 0], scores[:, 1], label='Train')
ax.plot(scores[:, 0], scores[:, 2], label='Hold Out')
ax.set_title(r'Ridge Regression with different $\alpha$')
ax.set_ylabel('$R^2$')
ax.set_xlabel(r'\alpha')
ax.legend();


I am working on the file "Model Evaluation/Model_Evaluation-exercise.ipynb".
The first task is:
2 - Separate your data into train and test sets, of portions ranging from test_size = 0.99 to test_size = 0.01, fitting a logistic regression model to each and computing the training and test errors.  Plot the errors of the training and test sets on the same plot.  Do this without using sklearn.model_select.learning_curve.  Comment on your results.
Can you write Python code for it?

# scaling and logistic regression
pipe = Pipeline([
        ('scl', StandardScaler()),
        ('lr', LogisticRegression())
    ])

scores = []
# looping through test sizes from 0.98 to 0.02 (0.99 gave error because I didn't have all the classes in the split)
# for tsize in np.arange(0.98, 0.01, -0.01):
for tsize in np.arange(0.01, 0.99, 0.01):
    # splitting
    Xtr, Xts, ytr, yts = train_test_split(wine.drop('class', axis=1),
                                            wine['class'],
                                            test_size=tsize,
                                            random_state=1,
                                            # stratify=wine['class'] # not used in solutions
                                         )
    # fit and score
    pipe.fit(Xtr, ytr)
    scores.append([tsize, pipe.score(Xtr, ytr), pipe.score(Xts, yts)])

scores = np.array(scores)

# plot train and test scores by increasing train size (1-test size)
fig, ax = plt.subplots(figsize=(16, 8))
ax.plot(1 - scores[:, 0], scores[:, 1], label='Train')
ax.plot(1 - scores[:, 0], scores[:, 2], label='Test')
ax.set_xlim([0, 1])
ax.set_ylim([0, 1.05])
ax.set_xlabel('Train size (%)')
ax.set_ylabel('Accuracy')
ax.set_title('Learning Curves')
ax.legend();


I am working on the file "Model Evaluation/Model_Evaluation-exercise.ipynb".
The first task is:
5 - Check your results in part (4) using sklearn.
Can you write Python code for it?

print(confusion_matrix(y_true=ytest, y_pred=ypred))
print('TPR', recall_score(ytest, ypred))
print('PRE', precision_score(y_true=ytest, y_pred=ypred))
print('F1', f1_score(y_true=ytest, y_pred=ypred))
print('ACC', accuracy_score(y_true=ytest, y_pred=ypred))


I am working on the file "Model Evaluation/Model_Evaluation-exercise.ipynb".
The first task is:
5 - Tune the regularization strength parameter as in part (4), but this time using sklearn.model_select.evaluation_curve.  Comment on your results.
Can you write Python code for it?

scores = {}
best_scores = {}

for k in [2, 3, 4, 5]:
    # validation cruve uses cv and wants params similar to grid search
    train_scores, test_scores = validation_curve(estimator=pipe,
                                                X=wine.drop('class', axis=1),
                                                y=wine['class'],
                                                param_name='lr__C',
                                                param_range=param_range,
                                                cv=k)
    # saving mean scores
    mean_train_scores = train_scores.mean(axis=1)
    mean_test_scores = test_scores.mean(axis=1)
    # and best scores
    best_scores[k] = [param_range[mean_test_scores.argmax()], mean_test_scores[mean_test_scores.argmax()]]
    scores[k] = np.array([param_range, mean_train_scores, mean_test_scores]).T

best_scores

fig, ax = plt.subplots(2, 2, figsize=(16, 8), sharex='col', sharey='row', subplot_kw=dict(xscale='log'))
i, j = 0, 0

for k in scores:
    ax[i, j].plot(scores[k][:, 0], scores[k][:, 1], label='Train')
    ax[i, j].plot(scores[k][:, 0], scores[k][:, 2], label='Test')
    ax[i, j].plot(best_scores[k][0], best_scores[k][1], 'ro', label='Best Score')
    ax[i, j].set_title('{}-fold CV'.format(k))
    if j == 0:
        ax[i, j].set_ylabel('Accuracy')
    if i == 1:
        ax[i, j].set_xlabel('C value')
    ax[i, j].legend()
    j += 1
    if j == 2:
        j = 0
        i += 1


I am working on the file "Model Evaluation/Model_Evaluation-exercise.ipynb".
The first task is:
6 - Plot the precision and recall curve for your fit.
Can you write Python code for it?

ypredprob = pipe.predict_proba(Xtest)

# get precision and recall values
precision, recall, _ = precision_recall_curve(ytest, ypredprob[:, 1])
# plot them and fill the curve
plt.figure(figsize=(16, 8))
plt.step(recall, precision, color='b', alpha=0.2, where='post')
plt.fill_between(recall, precision, step='post', alpha=0.2, color='b')

plt.xlabel('Recall')
plt.ylabel('Precision')
plt.ylim([0.0, 1.05])
plt.xlim([0.0, 1.0])
plt.title('Precision-Recall curve');


I am working on the file "Model Evaluation/Model_Evaluation-exercise.ipynb".
The first task is:
7 - Plot the ROC curve for your fit.
Can you write Python code for it?

# getting FPR and TPR for the ROC curve and the area under it
fpr, tpr, _ = roc_curve(ytest, ypredprob[:, 1])
roc_auc = roc_auc_score(ytest, ypredprob[:, 1])
# plot and fill the curve
plt.figure(figsize=(16, 8))
plt.step(fpr, tpr, color='b', alpha=0.2, where='post')
plt.fill_between(fpr, tpr, step='post', alpha=0.2, color='b')

plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.ylim([0.0, 1.05])
plt.xlim([0.0, 1.0])
plt.title('ROC curve, area = {:.5f}'.format(roc_auc));


I am working on the file "Model Evaluation/Model_Evaluation-exercise.ipynb".
The first task is:
After adding the sixth variable we have little improvement in the results.
4 - Build a pipeline that standardizes your data, performs feature selection via regularization, and then fits a model of your choice.  Produce the same plots as above and comment on your results.
Can you write Python code for it?

# scaling, select from model using lassoCV and lassoCV
pipe = Pipeline([
    ('scl', StandardScaler()),
    ('sfmod', SelectFromModel(LassoCV())),
    # ('lasso', LassoCV())
    ('forest', RandomForestRegressor()) # used in solution
])

scores = []
# looping through threshold values
for c in np.arange(0.1, 2.1, 0.1):
    pipe.set_params(sfmod__threshold=str(c) + '*mean')
    pipe.fit(Xtrain, ytrain)
    scores.append([c, pipe.score(Xtrain, ytrain), pipe.score(Xtest, ytest),
                   mean_squared_error(pipe.predict(Xtrain), ytrain), mean_squared_error(pipe.predict(Xtest), ytest)])
#     sel = pipe.named_steps['sfmod']
#     print(Xtrain.columns[sel.transform(np.arange(len(Xtrain.columns)).reshape(1, -1))])

scores = pd.DataFrame(scores)
scores.columns = ['threshold', 'train score', 'test score', 'train mse', 'test mse']

scores

fig, ax = plt.subplots(figsize=(10, 8))
ax.plot(scores.threshold, scores['train score'], label='train score')
ax.plot(scores.threshold, scores['test score'], label='test score')
ax.legend();

fig, ax = plt.subplots(figsize=(10, 8))
ax.plot(scores.threshold, scores['train mse'], label='train mse')
ax.plot(scores.threshold, scores['test mse'], label='test mse')
ax.legend();


I am working on the file "Model Evaluation/Model_Evaluation-exercise.ipynb".
The first task is:
Also, I'm encoding the class column with 0s and 1s:
Can you write Python code for it?

# 2 for benign, 4 for malignant
le = LabelEncoder()
le.fit([2, 4])
breast['class'] = le.transform(breast['class'])

breast.describe().T

Xtrain, Xtest, ytrain, ytest = train_test_split(breast.drop('class', axis=1), breast['class'], test_size=0.2, random_state=0)


I am working on the file "Model Evaluation/Model_Evaluation-exercise.ipynb".
The first task is:
Also, I'm relabeling the poisonous mushrooms as 1 and not poisonous ones as 0.
Can you write Python code for it?

ytrain = ytrain.apply(lambda x: 1 if x == 'p' else 0)
ytest = ytest.apply(lambda x: 1 if x == 'p' else 0)


I am working on the file "Model Evaluation/Model_Evaluation-exercise.ipynb".
The first task is:
Bare nuclei is an object: this is due to the presence of ? for missing values, which I'm going to replace with 1s.
Can you write Python code for it?

breast.bare_nuclei.value_counts()

breast.bare_nuclei = breast.bare_nuclei.apply(lambda x: x.replace('?', '1'))
breast.bare_nuclei = pd.to_numeric(breast.bare_nuclei)


I am working on the file "Model Evaluation/Model_Evaluation-exercise.ipynb".
The first task is:
I guess we can try with 12 components
Can you write Python code for it?

pipe = Pipeline([
    ('scl', StandardScaler()),
    ('pca', PCA(n_components=12)),
    ('ridge', Ridge())
])

scores = []

for a in [0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1, 3, 10, 30, 100, 300, 1000, 3000]:
    kfold = KFold(n_splits=10, random_state=1).split(Xtrain, ytrain)
    fold_scores = []
    pipe.set_params(ridge__alpha=a)
    print('alpha = {}'.format(a))
    for i, (train, test) in enumerate(kfold):
        Xtr, ytr = Xtrain.iloc[train, :], ytrain.iloc[train]
        Xcv, ycv = Xtrain.iloc[test, :], ytrain.iloc[test]
        pipe.fit(Xtr, ytr)

        tr_score = pipe.score(Xtr, ytr)
        cv_score = pipe.score(Xcv, ycv)
        fold_scores.append([i, tr_score, cv_score])
        print('Fold {} done'.format(i+1))

    fold_scores = np.array(fold_scores)
    scores.append([a, fold_scores.mean(axis=0)[1], fold_scores.mean(axis=0)[2],
                   fold_scores.std(axis=0)[1], fold_scores.std(axis=0)[2]])

scores = np.array(scores)

scores

fig, ax = plt.subplots(figsize=(16, 8), subplot_kw=dict(xscale='log'))

ax.plot(scores[:, 0], scores[:, 1], label='Train')
ax.plot(scores[:, 0], scores[:, 2], label='CV')
ax.fill_between(scores[:, 0],
                 scores[:, 1] + scores[:, 3],
                 scores[:, 1] - scores[:, 3],
                 alpha=0.15, color='blue')
ax.fill_between(scores[:, 0],
                 scores[:, 2] + scores[:, 4],
                 scores[:, 2] - scores[:, 4],
                 alpha=0.15, color='orange')
ax.set_title(r'Ridge Regression with different $\alpha$')
ax.set_ylabel('$R^2$')
ax.set_xlabel(r'$\alpha$')
ax.legend();


I am working on the file "Model Evaluation/Model_Evaluation-exercise.ipynb".
The first task is:
In solution he uses dummies for subject#, which I don't think is really "honest".
Can you write Python code for it?

parkinson.head()

parkinson.info()

parkinson.describe().T

# I'm dropping subject# because it is the patient id
Xtrain, Xtest, ytrain, ytest = train_test_split(parkinson.drop(['subject#', 'motor_UPDRS'], axis=1),
                                                parkinson.motor_UPDRS,
                                                test_size=0.2,
                                                random_state=54)


I am working on the file "Model Evaluation/Model_Evaluation-exercise.ipynb".
The first task is:
Not much of an improvement...
6 - Pick a regression algorithm that has at least two hyperparameters and tune them using GridSeachCV.  Comment on your results.
I'm going to try and add some polynomial features to the Ridge regression to see if it can perform better:
Can you write Python code for it?

pipe = Pipeline([
    ('scl', StandardScaler()),
    ('pca', PCA(n_components=9)),
    ('poly', PolynomialFeatures()),
    ('ridge', Ridge())
])

alpha_range = [0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1, 3, 10, 30]
degree_range = np.arange(1, 4)
# using grid search to tune regularization and degree of polynomials
param_grid = [{'poly__degree':degree_range, 'ridge__alpha': alpha_range}]
gs = GridSearchCV(estimator=pipe,
                  param_grid=param_grid,
                  cv=10,
                  n_jobs=2)

gs.fit(Xtrain, ytrain)

print(gs.best_score_)
print(gs.best_params_)

pipe.set_params(poly__degree=2, ridge__alpha=30)
pipe.fit(Xtrain, ytrain)
pipe.score(Xtest, ytest)


I am working on the file "Model Evaluation/Model_Evaluation-exercise.ipynb".
The first task is:
Seven is the number of neighbors with the best CV score:
Can you write Python code for it?

pipe.set_params(knn__n_neighbors=7)
pipe.fit(Xtrain, ytrain)
pipe.score(Xtest, ytest)


I am working on the file "Model Evaluation/Model_Evaluation-exercise.ipynb".
The first task is:
So, apparently GridSearchCV gets stuck if I use custom scorers or if I'm multithreading or something like that, so I'm going to try to encode my labels outside the pipeline...
Can you write Python code for it?

mle = MultiLabelEncoder()
Xtrain_enc = mle.fit_transform(Xtrain)
Xtest_enc = mle.transform(Xtest)

pipe = Pipeline([
    ('sel', RFE(estimator=LogisticRegression())),
    ('lr', LogisticRegression())
])

# C_range = [0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1, 3, 10, 30, 100, 300]
C_range = [0.01, 0.03, 0.1, 0.3, 1, 3]
comp_range = np.arange(1, 16)
# using grid search to tune regularization and number of PCs
param_grid = [{'sel__n_features_to_select':comp_range, 'lr__C': C_range}]
gs = GridSearchCV(estimator=pipe,
                  param_grid=param_grid,
                  cv=10)
# this could take a very long time...
gs.fit(Xtrain_enc, ytrain)

print(gs.best_score_)
print(gs.best_params_)

pipe.set_params(sel__n_features_to_select=13, lr__C=0.3)
pipe.fit(Xtrain_enc, ytrain)
pipe.score(Xtest_enc, ytest)


I am working on the file "Model Evaluation/Model_Evaluation-exercise.ipynb".
The first task is:
Test score is better than training using LassoCV as an estimator, using random forest we get more overfitting but also better overall performances than all the previous methods. The best choice for the threshold seems to be around 0.5.
5 - Create two pipelines for feature selection of a technique of your choice, scaling the data before hand.  Then, join these two pipelines with a FeatureUnion, and fit a polynomial model, also in a pipeline.  Comment on your results.
Can you write Python code for it?

# joining select from model using lasso and select k best using the best parameters from before
pipe = Pipeline([
    ('scl', StandardScaler()),
    ('featsel', FeatureUnion([
        ('sfmod', SelectFromModel(LassoCV(), threshold='0.5*mean')),
        ('best', SelectKBest(k=6))
    ])),
    ('poly', PolynomialFeatures(degree=3)),
    ('lr', LinearRegression())
])

pipe.fit(Xtrain, ytrain)
print(pipe.score(Xtrain, ytrain))
print(pipe.score(Xtest, ytest))

print(mean_squared_error(pipe.predict(Xtrain), ytrain))
print(mean_squared_error(pipe.predict(Xtest), ytest))

pipe.named_steps['poly'].n_input_features_


I am working on the file "Model Evaluation/Model_Evaluation-exercise.ipynb".
The first task is:
The accuracy is almost 98%, which is quite good!
3 - Using your model in part (2), compute the following quantities, without using skelarn.metrics.
    - True Positives
    - True Negatives
    - False Positives
    - False Negatives
Can you write Python code for it?

ypred = pipe.predict(Xtest)

TP = sum((ypred[ytest==1]==ytest.values[ytest.values==1]))
TN = sum((ypred[ytest==0]==ytest.values[ytest.values==0]))
FP = sum((ypred[ytest==0]!=ytest.values[ytest.values==0]))
FN = sum((ypred[ytest==1]!=ytest.values[ytest.values==1]))

print('True Positives: {}\nTrue Negatives: {}\nFalse Positives: {}\nFalse Negatives: {}'.format(TP, TN, FP, FN))


I am working on the file "Model Evaluation/Model_Evaluation-exercise.ipynb".
The first task is:
The plots are the same as above.
6 - Fit another classification algorithm to the data, tuning the parameter using LOOCV.  Comment on your results.
Can you write Python code for it?

pipe = Pipeline([
    ('scl', StandardScaler()),
    ('svm', SVC())
])
# using grid search with LeaveOneOut as cv
param_range = [0.0001, 0.0003, 0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1, 3, 10, 30, 100, 300, 1000, 3000]
param_grid = [{'svm__C': param_range}]

# in solution: cross_val_score(pipe, X, y, cv=LeaveOneOut()) used looping through params values
gs = GridSearchCV(estimator=pipe,
                  param_grid=param_grid,
                  cv=LeaveOneOut())

gs.fit(wine.drop('class', axis=1), wine['class'])
gs.best_params_['svm__C'], gs.best_score_


I am working on the file "Model Evaluation/Model_Evaluation-exercise.ipynb".
The first task is:
The score is very bad, the best choice for the regularization parameter seems to be 100 but there seems to be a problem of serious bias in this dataset. We may need more features to get accurate predictions.
3 - Repeat part (2), this time using cross validation.  Comment on your results.
Again in the solution he uses cross_validation_curve...
Can you write Python code for it?

scores = []

for a in [0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1, 3, 10, 30, 100, 300, 1000, 3000]:
    kfold = KFold(n_splits=10, random_state=1).split(Xtrain, ytrain)
    fold_scores = []
    pipe.set_params(ridge__alpha=a)
    print('alpha = {}'.format(a))
    for i, (train, test) in enumerate(kfold):
        Xtr, ytr = Xtrain.iloc[train, :], ytrain.iloc[train]
        Xcv, ycv = Xtrain.iloc[test, :], ytrain.iloc[test]
        pipe.fit(Xtr, ytr)
        tr_score = pipe.score(Xtr, ytr)
        cv_score = pipe.score(Xcv, ycv)
        fold_scores.append([i, tr_score, cv_score])
        print('Fold {} done'.format(i+1))

    fold_scores = np.array(fold_scores)
    scores.append([a, fold_scores.mean(axis=0)[1], fold_scores.mean(axis=0)[2],
                   fold_scores.std(axis=0)[1], fold_scores.std(axis=0)[2]])

scores = np.array(scores)

scores

fig, ax = plt.subplots(figsize=(16, 8), subplot_kw=dict(xscale='log'))

ax.plot(scores[:, 0], scores[:, 1], label='Train')
ax.plot(scores[:, 0], scores[:, 2], label='CV')
ax.fill_between(scores[:, 0],
                 scores[:, 1] + scores[:, 3],
                 scores[:, 1] - scores[:, 3],
                 alpha=0.15, color='blue')
ax.fill_between(scores[:, 0],
                 scores[:, 2] + scores[:, 4],
                 scores[:, 2] - scores[:, 4],
                 alpha=0.15, color='orange')
ax.set_title(r'Ridge Regression with different $\alpha$')
ax.set_ylabel('$R^2$')
ax.set_xlabel(r'$\alpha$')
ax.legend();


I am working on the file "Model Evaluation/Model_Evaluation-exercise.ipynb".
The first task is:
Training scores are very similar in each of the plots, there are some differences between the test accuracy curves where the best value for C seems to go towards 1.
Can you write Python code for it?

# solution:
X = wine.drop('class', axis=1)
y = wine['class']

k_vals = np.arange(2, 6, 1)
c_vals = np.arange(0.5, 1.5, 0.05)
err = []
for k in k_vals:
    for c in c_vals:
        lm = LogisticRegression(C=c)
        temp_err = cross_val_score(lm, X=X, y=y, cv=k)

        err_dict = {}
        err_dict['k'] = k
        err_dict['c'] = c
        err_dict['acc'] = temp_err.mean()

        err.append(err_dict)

err = pd.DataFrame(err)

for k in err.k.unique():
    plt.plot(c_vals, err.acc[err.k == k], label='k = {}'.format(k))
plt.title('Accuracy vs Regularization Strength')
plt.xlabel('C')
plt.ylabel('Accuracy')
plt.legend();


I am working on the file "Model Evaluation/Model_Evaluation-exercise.ipynb".
The first task is:
Using LOOCV I get 0.3 as the best choice for C and an accuracy of almost 99%, which is slightly higher than before.
7 - Suppose that the wine data the we received was incomplete, containing, say, only 20% of the full set, but due to a fast approaching deadline, we need to still compute some statistics and fit a model.  Use bootstrapping to compute the mean and variance of the features, and fit the classification model you used in part (4), comparing and commenting on your results with those from the full dataset.
Can you write Python code for it?

# resampling 5 times the data
wine_strapped = resample(wine, n_samples=len(wine)*5, random_state=10)

# solution: leaves 80% of the data out and bootstraps the remaining 20%:

data_sub_test, data_sub = train_test_split(wine, test_size=0.2, random_state=0)
data_boot = data_sub.sample(n=150, replace=True, random_state=0)

# still from solution:
means = {'original': wine.drop('class', axis=1).mean(),
        '20%': data_sub.drop('class', axis=1).mean(),
        'bootstrap%': data_boot.drop('class', axis=1).mean()}

var = {'original': wine.drop('class', axis=1).var(),
        '20%': data_sub.drop('class', axis=1).var(),
        'bootstrap%': data_boot.drop('class', axis=1).var()}

means = pd.DataFrame(means)
var = pd.DataFrame(var)

means

var

# last from solution:

pipe = Pipeline([
        ('scl', StandardScaler()),
        ('lr', LogisticRegression())
    ])

pipe.set_params(lr__C=1)
pipe.fit(wine.drop('class', axis=1), wine['class'])
print(cross_val_score(pipe, wine.drop('class', axis=1), wine['class'], cv=5).mean())

pipe.fit(data_boot.drop('class', axis=1), data_boot['class'])
print(cross_val_score(pipe, data_sub_test.drop('class', axis=1), data_sub_test['class'], cv=5).mean())

wine.describe().T

wine_strapped.describe().T

pipe = Pipeline([
        ('scl', StandardScaler()),
        ('lr', LogisticRegression())
    ])
# using grid search to fit the best logistic regression
param_range = [0.0001, 0.0003, 0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1, 3, 10, 30, 100, 300, 1000, 3000]
param_grid = [{'lr__C': param_range}]
gs = GridSearchCV(estimator=pipe,
                  param_grid=param_grid)

scores = {}
best_scores = {}
# looping through number of folds
for k in [2, 3, 4, 5]:
    gs.set_params(cv=k)
    gs.fit(wine_strapped.drop('class', axis=1), wine_strapped['class'])
    best_scores[k] = [gs.best_params_['lr__C'], gs.best_score_]
    scores[k] = np.array([param_range, gs.cv_results_['mean_train_score'], gs.cv_results_['mean_test_score']]).T

best_scores


I am working on the file "Model Evaluation/Model_Evaluation-exercise.ipynb".
The first task is:
Using a random forest we get better performances but we have a serious problem of high variance in this case.
5 - Which ever of your two algorithms in parts (3) and (4) performed more poorly, perform a variable selection to see if you can improve your results.
I'm going to use PCA to try and improve the R regression, but I don't think there is much to do...
In solution he uses LassoCV with SelectFromModel:
Can you write Python code for it?

scl = StandardScaler()
sfm = SelectFromModel(LassoCV())
Xtrain_sc = scl.fit_transform(Xtrain)
Xtest_sc = scl.transform(Xtest)
Xtrain_lm = sfm.fit_transform(Xtrain_sc, ytrain)
Xtest_lm = sfm.transform(Xtest_sc)

train_scores, test_scores = validation_curve(Ridge(), Xtrain_lm, ytrain,
                                             param_name='alpha',
                                             param_range=[0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1, 3, 10, 30, 100, 300, 1000, 3000])

train_scores_mean = np.mean(train_scores, axis=1)
train_scores_std = np.std(train_scores, axis=1)
test_scores_mean = np.mean(test_scores, axis=1)
test_scores_std = np.std(test_scores, axis=1)

fig, ax = plt.subplots(figsize=(16, 8), subplot_kw=dict(xscale='log'))

ax.plot([0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1, 3, 10, 30, 100, 300, 1000, 3000],
        train_scores_mean, label='Train')
ax.fill_between([0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1, 3, 10, 30, 100, 300, 1000, 3000],
                train_scores_mean - train_scores_std, train_scores_mean + train_scores_std, alpha=0.2)
ax.plot([0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1, 3, 10, 30, 100, 300, 1000, 3000],
        test_scores_mean, label='Test')
ax.fill_between([0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1, 3, 10, 30, 100, 300, 1000, 3000],
                test_scores_mean - test_scores_std, test_scores_mean + test_scores_std, alpha=0.2)
ax.legend();

pca = PCA()
pca.fit(Xtrain)
pca.explained_variance_ratio_


I am working on the file "Model Evaluation/Model_Evaluation-exercise.ipynb".
The first task is:
Using learning curves the accuracy on the training set is always perfect, while the test accuracy increases without getting to 100%: this may be because learning curve uses cross validation and so we are less dependent on the splits.
4 - Use K-Fold cross validation to tune the regularization strength parameter of the logistic regression.  Do this for values of k ranging from 2 to 5, make relevant  plots, and comment on your results.  Do this without using sklearn.model_select.evaluation_curve.
Can you write Python code for it?

# using grid search which uses k-fold cv by default
param_range = [0.0001, 0.0003, 0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1, 3, 10, 30, 100, 300, 1000, 3000]
param_grid = [{'lr__C': param_range}]
gs = GridSearchCV(estimator=pipe,
                  param_grid=param_grid)

scores = {}
best_scores = {}
# looping through number of folds
for k in [2, 3, 4, 5]:
    # set params
    gs.set_params(cv=k)
    # fit
    gs.fit(wine.drop('class', axis=1), wine['class'])
    # saving best scores and mean scores for all params
    best_scores[k] = [gs.best_params_['lr__C'], gs.best_score_]
    scores[k] = np.array([param_range, gs.cv_results_['mean_train_score'], gs.cv_results_['mean_test_score']]).T

best_scores

# creating 4 plots, one for each number of fold
fig, ax = plt.subplots(2, 2, figsize=(16, 8), sharex='col', sharey='row', subplot_kw=dict(xscale='log'))
i, j = 0, 0

for k in scores:
    # plotting train and test scores and the best score
    ax[i, j].plot(scores[k][:, 0], scores[k][:, 1], label='Train')
    ax[i, j].plot(scores[k][:, 0], scores[k][:, 2], label='Test')
    ax[i, j].plot(best_scores[k][0], best_scores[k][1], 'ro', label='Best Score')
    ax[i, j].set_title('{}-fold CV'.format(k))
    if j == 0:
        ax[i, j].set_ylabel('Accuracy')
    if i == 1:
        ax[i, j].set_xlabel('C value')
    ax[i, j].legend()
    j += 1
    if j == 2:
        j = 0
        i += 1


I am working on the file "Model Evaluation/Model_Evaluation-exercise.ipynb".
The first task is:
We get 95% accuracy with simple logistic regression, not bad!
Regression
1 - Head over to the Machine Learning Repository, download the Parkinson's Telemonitoring Data Set, and put it into a dataframe.  Be sure to familiarize yourself with the data before proceeding, removing the columns related to time, age, and sex.  We're going to be predicting motor_UPDRS, so drop the total_UPDRS variable as well.  Break the data into training and testing sets.  Be sure to keep this testing set the same for the duration of this exercise as we will be using to test various algorithms!
Can you write Python code for it?

parkinson = pd.read_csv('http://archive.ics.uci.edu/ml/machine-learning-databases/parkinsons/telemonitoring/parkinsons_updrs.data')
parkinson.drop(['age', 'sex', 'test_time', 'total_UPDRS'], axis=1, inplace=True)


I am working on the file "Model Evaluation/Model_Evaluation-exercise.ipynb".
The first task is:
We get perfect accuracy for a number of neighbors between 2 and 8 this time; we can also see that we have no variance for this range of values.
4 - Repeat part (3) using a different machine learning algorithm.  Comment on your results.
Can you write Python code for it?

# encoder and random forest
pipe = Pipeline([
    ('mle', MultiLabelEncoder()),
    #('ohe', OneHotEncoder()),
    ('forest', RandomForestClassifier())
])

scores = []

for k in np.arange(2, 50, 2):
    kfold = KFold(n_splits=10, random_state=1).split(Xtrain, ytrain)
    fold_scores = []
    pipe.set_params(forest__n_estimators=k)
    print('{} Trees'.format(k))
    for i, (train, test) in enumerate(kfold):
        Xtr, ytr = Xtrain.iloc[train, :], ytrain.iloc[train]
        Xcv, ycv = Xtrain.iloc[test, :], ytrain.iloc[test]
        pipe.fit(Xtr, ytr)
        tr_score = pipe.score(Xtr, ytr)
        cv_score = pipe.score(Xcv, ycv)
        fold_scores.append([i, tr_score, cv_score])
        print('Fold {} done'.format(i+1))

    fold_scores = np.array(fold_scores)
    scores.append([k, fold_scores.mean(axis=0)[1], fold_scores.mean(axis=0)[2],
                   fold_scores.std(axis=0)[1], fold_scores.std(axis=0)[2]])

scores = np.array(scores)

scores

fig, ax = plt.subplots(figsize=(16, 8))

ax.plot(scores[:, 0], scores[:, 1], label='Train')
ax.plot(scores[:, 0], scores[:, 2], label='CV')
ax.fill_between(scores[:, 0],
                 scores[:, 1] + scores[:, 3],
                 scores[:, 1] - scores[:, 3],
                 alpha=0.15, color='blue')
ax.fill_between(scores[:, 0],
                 scores[:, 2] + scores[:, 4],
                 scores[:, 2] - scores[:, 4],
                 alpha=0.15, color='orange')
ax.set_title('Random Forest with different # of Trees'.format(k))
ax.set_ylabel('Accuracy')
ax.set_xlabel('# of Trees')
ax.legend();

pipe.set_params(forest__n_estimators=10)
pipe.fit(Xtrain, ytrain)
pipe.score(Xtest, ytest)


I am working on the file "Model Evaluation/Model_Evaluation-exercise.ipynb".
The first task is:
We get perfect accuracy for about 8 trees or more, so I'm going with the default value of 10.
5 - Which ever of your two algorithms in parts (3) and (4) performed more poorly, perform a variable selection to see if you can improve your results.
Can you write Python code for it?

mle = MultiLabelEncoder()
pca = PCA()
pca.fit(mle.fit_transform(Xtrain), ytrain)

pca.explained_variance_ratio_


I am working on the file "Model an insert shape.ipynb".
The first task is:
Calculate the factor
Can you write Python code for it?

number_of_data = len(input_dataframe)

if number_of_data >= 8:
    ratio_perim_area_data = convert2_ratio_perim_area(width_data, length_data)

    ratio_perim_area = convert2_ratio_perim_area(width, length)
    factor = float(spline_model_with_deformability(width, ratio_perim_area,
        width_data, ratio_perim_area_data, factor_data))
else:
    factor = np.nan

print("Width = {0:0.2f} cm\nLength = {1:0.2f} cm\nFactor = {2:0.4f}".format(
        width, length, factor))


I am working on the file "Model an insert shape.ipynb".
The first task is:
Load model data
Can you write Python code for it?

data_filelist = glob('*_data.csv')
assert len(data_filelist) <= 1, "There appears to be more than 1 data.csv file, please delete the old version(s)"
assert len(data_filelist) == 1, "I need at least one data file"
data_filename = data_filelist[0]
data = pd.read_csv(data_filename)


I am working on the file "Model-Network.ipynb".
The first task is:
Checking model representation
Here we can examine how well the model is able to approximate the true environment after training. The green line indicates the real environment, and the blue indicates model predictions.
Can you write Python code for it?

plt.figure(figsize=(8, 12))
for i in range(6):
    plt.subplot(6, 2, 2*i + 1)
    plt.plot(pState[:,i])
    plt.subplot(6,2,2*i+1)
    plt.plot(state_nextsAll[:,i])
plt.tight_layout()


I am working on the file "Model-Network.ipynb".
The first task is:
Model Network
Here we implement a multi-layer neural network that predicts the next observation, reward, and done state from a current state and action.
Can you write Python code for it?

mH = 256 # model layer size

input_data = tf.placeholder(tf.float32, [None, 5])
with tf.variable_scope('rnnlm'):
    softmax_w = tf.get_variable("softmax_w", [mH, 50])
    softmax_b = tf.get_variable("softmax_b", [50])

previous_state = tf.placeholder(tf.float32, [None,5] , name="previous_state")
W1M = tf.get_variable("W1M", shape=[5, mH],
           initializer=tf.contrib.layers.xavier_initializer())
B1M = tf.Variable(tf.zeros([mH]),name="B1M")
layer1M = tf.nn.relu(tf.matmul(previous_state,W1M) + B1M)
W2M = tf.get_variable("W2M", shape=[mH, mH],
           initializer=tf.contrib.layers.xavier_initializer())
B2M = tf.Variable(tf.zeros([mH]),name="B2M")
layer2M = tf.nn.relu(tf.matmul(layer1M,W2M) + B2M)
wO = tf.get_variable("wO", shape=[mH, 4],
           initializer=tf.contrib.layers.xavier_initializer())
wR = tf.get_variable("wR", shape=[mH, 1],
           initializer=tf.contrib.layers.xavier_initializer())
wD = tf.get_variable("wD", shape=[mH, 1],
           initializer=tf.contrib.layers.xavier_initializer())

bO = tf.Variable(tf.zeros([4]),name="bO")
bR = tf.Variable(tf.zeros([1]),name="bR")
bD = tf.Variable(tf.ones([1]),name="bD")

predicted_observation = tf.matmul(layer2M,wO,name="predicted_observation") + bO
predicted_reward = tf.matmul(layer2M,wR,name="predicted_reward") + bR
predicted_done = tf.sigmoid(tf.matmul(layer2M,wD,name="predicted_done") + bD)

true_observation = tf.placeholder(tf.float32,[None,4],name="true_observation")
true_reward = tf.placeholder(tf.float32,[None,1],name="true_reward")
true_done = tf.placeholder(tf.float32,[None,1],name="true_done")

predicted_state = tf.concat([predicted_observation,predicted_reward,predicted_done],1)

observation_loss = tf.square(true_observation - predicted_observation)

reward_loss = tf.square(true_reward - predicted_reward)

done_loss = tf.multiply(predicted_done, true_done) + tf.multiply(1-predicted_done, 1-true_done)
done_loss = -tf.log(done_loss)

model_loss = tf.reduce_mean(observation_loss + done_loss + reward_loss)

modelAdam =...


I am working on the file "Model-Network.ipynb".
The first task is:
Training the Policy and Model
Can you write Python code for it?

xs,drs,ys,ds = [],[],[],[]
running_reward = None
reward_sum = 0
episode_number = 1
real_episodes = 1
init = tf.global_variables_initializer()
batch_size = real_bs

drawFromModel = False # When set to True, will use model for observations
trainTheModel = True # Whether to train the model
trainThePolicy = False # Whether to train the policy
switch_point = 1

# Launch the graph
with tf.Session() as sess:
    rendering = False
    sess.run(init)
    observation = env.reset()
    x = observation
    gradBuffer = sess.run(tvars)
    gradBuffer = resetGradBuffer(gradBuffer)

    while episode_number <= 5000:
        # Start displaying environment once performance is acceptably high.
        if (reward_sum/batch_size > 150 and drawFromModel == False) or rendering == True :
            env.render()
            rendering = True

        x = np.reshape(observation,[1,4])

        tfprob = sess.run(probability,feed_dict={observations: x})
        action = 1 if np.random.uniform() < tfprob else 0

        # record various intermediates (needed later for backprop)
        xs.append(x)
        y = 1 if action == 0 else 0
        ys.append(y)

        # step the  model or real environment and get new measurements
        if drawFromModel == False:
            observation, reward, done, info = env.step(action)
        else:
            observation, reward, done = stepModel(sess,xs,action)

        reward_sum += reward

        ds.append(done*1)
        drs.append(reward) # record reward (has to be done after we call step() to get reward for previous action)

        if done:

            if drawFromModel == False:
                real_episodes += 1
            episode_number += 1

            # stack together all inputs, hidden states, action gradients, and rewards for this episode
            epx = np.vstack(xs)
            epy = np.vstack(ys)
            epr = np.vstack(drs)
            epd = np.vstack(ds)
            xs,drs,ys,ds = [],[],[],[] # reset array...


I am working on the file "Model/Yahoo/yahoo_SVM.ipynb".
The first task is:
1. Load Data
Can you write Python code for it?

texts = []  # list of text samples
labels_index = {}  # dictionary mapping label name to numeric id
labels = []  # list of label ids
for name in sorted(os.listdir(TEXT_DATA_DIR)):
    path = os.path.join(TEXT_DATA_DIR, name)
    if os.path.isdir(path):
        label_id = len(labels_index)
        labels_index[name] = label_id
        for fname in sorted(os.listdir(path)):
            if fname.isdigit():
                fpath = os.path.join(path, fname)
                if sys.version_info < (3,):
                    f = open(fpath)
                else:
                    f = open(fpath, encoding='latin-1')
                texts.append(f.read())
                f.close()
                labels.append(label_id)
print('Found %s texts.' % len(texts))
texts, labels = np.array(texts), np.array(labels)
print (texts.shape, labels.shape)


I am working on the file "Model/Yahoo/yahoo_SVM.ipynb".
The first task is:
3. Implement  an embedding vector
Can you write Python code for it?

class MeanEmbeddingVectorizer(object):
    def __init__(self, word2vec):
        self.word2vec = word2vec
        self.dim = len(word2vec.itervalues().next())

    def fit(self, X, y):
        return self

    def transform(self, X):
        return np.array([
            np.mean([self.word2vec[w] for w in words if w in self.word2vec]
                    or [np.zeros(self.dim)], axis=0)
            for words in X
        ])

class TfidfEmbeddingVectorizer(object):
    def __init__(self, word2vec):
        self.word2vec = word2vec
        self.word2weight = None
        self.dim = len(word2vec.itervalues().next())

    def fit(self, X, y):
        tfidf = TfidfVectorizer(analyzer=lambda x: x)
        tfidf.fit(X)
        # if a word was never seen - it must be at least as infrequent
        # as any of the known words - so the default idf is the max of
        # known idf's
        max_idf = max(tfidf.idf_)
        self.word2weight = defaultdict(
            lambda: max_idf,
            [(w, tfidf.idf_[i]) for w, i in tfidf.vocabulary_.items()])

        return self

    def transform(self, X):
        return np.array([
                np.mean([self.word2vec[w] * self.word2weight[w]
                         for w in words if w in self.word2vec] or
                        [np.zeros(self.dim)], axis=0)
                for words in X
            ])


I am working on the file "Model01.ipynb".
The first task is:
Avec le flux solaire
Can you write Python code for it?


I am working on the file "Model01.ipynb".
The first task is:
Avec une température fixe
Can you write Python code for it?

def model012_apply(  tau_ext, tau_inf, T_inf, T_ext, sample_freq=15*60 ):
    """ T_ext : np_array
        sample_freq : duré entre les points, en seconde
    """
    # TF (reel)
    TF_ext = np.fft.rfft( T_ext  )
    TF_inf =  np.fft.rfft( np.ones_like(T_ext)*T_inf  )

    freq = np.fft.rfftfreq( T_ext.size, d=sample_freq)

    # modele
    TF_theo = (  tau_ext*TF_ext + tau_inf*TF_inf )/( 2j*np.pi*freq  + tau_ext + tau_inf )

    # TF inverse
    T_theo = np.fft.irfft( TF_theo , n=T_ext.size )

    return T_theo

from scipy.optimize import minimize

def errorfit012( param, T_int, T_ext  ):
    T_theo = model012_apply( *param, T_ext )
    return np.sum( (T_int - T_theo)**2 )

def model012_fit( T_int, T_ext ):

    paramZero = ( 1, 200, 20 )

    res = minimize(errorfit012, paramZero, method='Powell', options={'disp': True}, args=(T_int, T_ext))
    #paramOpt, fopt = res[:2]

    return res

res = model012_fit( T_int, T_ext )

print( res.x )

T_theo = model012_apply( *res.x, T_ext )

df['model012'] = T_theo

df.plot( figsize=(14, 6)  )


I am working on the file "Model01.ipynb".
The first task is:
Chargement des données
Can you write Python code for it?

df_full  = pd.read_pickle( 'weatherdata.pck' )

df = df_full[['T_int', 'temperature']].copy()
df = df.dropna()

df.plot( figsize=(14, 5) )

# To array :
T_ext = df['temperature'].as_matrix().flatten()
T_int = df['T_int'].as_matrix().flatten()

time_sec = df.index.astype(np.int64) // 10e9

print( T_ext.mean() )
print( T_int.mean() )

best_beta = model01_fit( T_int, T_ext )

print('---')
print('beta_opt: %e s-1'%best_beta)

print('1/beta : %.1f h'%(1/best_beta / 60/60))

beta = best_beta  # 2.33e-05
T_theo = model01_apply( beta, T_ext )

df['model01'] = T_theo

df.plot( figsize=(14, 6)  )


I am working on the file "Model01.ipynb".
The first task is:
Fit
Can you write Python code for it?

from scipy.optimize import fmin

def errorfit( param, T_int, T_ext  ):
    T_theo = model01_apply( param, T_ext )
    return np.sum( (T_int - T_theo)**2 )

def model01_fit( T_int, T_ext ):

    betaZero = 3e-05

    res = fmin(errorfit, betaZero, disp=True, full_output = True, args=(T_int, T_ext))

    paramOpt, fopt = res[:2]

    return paramOpt[0]


I am working on the file "Model01.ipynb".
The first task is:
Modèle 01
C'est le modèle le plus simple : une masse thermique ($m.C_p$) reliée à la température extérieure à travers une résistance thermique ($S.h$). Linéaire, 1er ordre, équivalent électrique RC.
Rq.: la température extérieure moyenne doit être égale à celle intérieure moyenne.
$$
m.C_p \, \frac{dT}{dt} = -S.h\,( T - T_{ext} )
$$
$m.Cp$ est la masse thermique, $S$ est la surface avec l'extérieure et $h$ le coefficient d'échange.
On pose:
$$
\beta = \frac{S.h}{m.C_p}
$$
et en pasant par la transformée de Fourier, on obtient:
$$
\tilde T = \frac{\beta}{\beta + 2\pi\, i \, f } \tilde T_{ext}
$$
Can you write Python code for it?

def model01_apply( beta,  T_ext, sample_freq=15*60 ):
    """ T_ext : np_array
        sample_freq : duré entre les points, en seconde
    """
    # TF (reel)
    TF_ext = np.fft.rfft( T_ext  )
    freq = np.fft.rfftfreq( T_ext.size, d=sample_freq)

    # modele
    TF_theo = beta/( beta + 2j*np.pi*freq )*TF_ext

    # TF inverse
    T_theo = np.fft.irfft( TF_theo , n=T_ext.size )

    return T_theo


I am working on the file "Model01.ipynb".
The first task is:
Ode
Can you write Python code for it?

from scipy.integrate import odeint

get_Text = lambda t : np.interp( t, time_sec, T_ext )

flux = df_full['flux_tot'].as_matrix().flatten()

def get_dTdt(  T, t, tau ):
    return ( get_Text(t) - T )/tau

Tzero = 27
tau = 4*3600
res = odeint(get_dTdt, Tzero, time_sec, args=(tau, ) ) #, rtol=1e-2, atol=1e-2, h0=1200)

df['model013'] = res
df.plot( figsize=(14, 6)  )

df = df.drop( ['model012'], axis=1)


I am working on the file "Model02_tuile.ipynb".
The first task is:

Modèlisation de la température des tuiles d'un toit
Mon appartement est sous les toits. Je ressens donc l'effet direct du soleil sur la température chez moi. Je peux aussi facilement mesurer la température des tuiles avec un thermomètre IR, en passant le bras à travers le velux. J'ai donc voulu savoir s'il est possible de prédire correctement la température des tuiles à partir des données météo et de l'irradiation solaire.
Le premier problème est d'obtenir les données météo heure par heure, ainsi que la valeur de l'irradiation solaire.
La question suivante est de correctement décrire les échanges thermiques entre les tuiles et le reste de l'univers ... les tuiles sont chauffées par le soleil, et refroidit par l'air ambiant (convection) et l'échange de rayonnement avec le ciel (corps gris/noir). Aussi, de façon négligeable par la conduction à travers la structure du toit.

Une fois l'ensemble des contributions chiffrées, on effectue la résolution numérique du système, ici assez simple. Et, enfin, on peut valider le modèle en comparant les résultats avec les mesures effectuées.
Can you write Python code for it?

coords_gps = (45.1973288, 5.7139923) # (latitude, longitude), degrée décimaux

# Période d'intérêt :
startday = pd.to_datetime('01/08/2017', format='%d/%m/%Y').tz_localize('Europe/Paris')
lastday  = pd.to_datetime( 'now' ).tz_localize('Europe/Paris')


I am working on the file "Model02_tuile.ipynb".
The first task is:
Echange par radiation
En plus de la radiation direct du soleil, il faut prendre en compte les échanges thermiques par émission du corps noir, qui peuvent être important. Ces échanges ont lieu entre la surface des tuilles et l'atmosphère, qui émet lui aussi un rayonement électromagnétique du fait de sa température. On distingue les short-wave radiation, qui correspondent au rayonnement direct, et diffus, du soleil (UV, visible et proche infrarouge) des long-wave radiation correspondant à l'émission du corps noir de l'atmosphère (T~10°C vs 6000°C du soleil).
L'émission d'un corps à une température $T$ est donné par la loi de Stefan-Boltzmann :
$$
    P = \sigma \, \epsilon  \, \bar T ^4
$$

$P$ est la puissance rayonnée dans le demi-espace, en W/m2
$\sigma$ est la constante de Stefan-Boltzmann, W/m2/K4
$\epsilon$ est l'émissivité de la surface
$\bar T$ est la température de la surface, en Kelvin, c.a.d. $\bar T = T_{Celsius} + 273.15$

 Température du ciel :   La température du ciel est généralement inférieure à la température ambiante, et elle dépend de la météo local, comme la couverture nuageuse et l'humidité. Un modèle simple permet de l'estimer. Voir cette réponse sur Stackexchange par exemple.
Facteur de forme : Comme le toit est incliné, seul une portion du ciel est visible, ~ 75%. Ce facteur doit être pris en compte dans l'échange de chaleur par radiation (voir view factor).
Can you write Python code for it?

""" Rayonement thermique du ciel
    ou autrement dit la température du ciel,
"""

def Goforth_skyrad( cloudCover, Tair_deg, RH ):
    """ Power radiated by the night sky
        see:  https://physics.stackexchange.com/q/153947
        https://physics.stackexchange.com/questions/153839/
        what-is-the-temperature-of-the-clear-night-sky-from-the-surface-of-earth
        "Cloud effects on thermal downwelling sky radiance," AeroSense 2002
    """
    K = 0.2   #  0.34 for very low clouds to 0.06 for very high clouds,
    T = Tair_deg + 273
    RH = RH*100

    Pthermal = ( 1 + K*cloudCover**2 ) * 8.78e-13 * T**5.852 * RH**0.07195

    return Pthermal

sigmaSB = 5.670373e-8 # Constante de Stefan-Boltzmann,   W.m-2.K-4

albedo_tuile = 0.36   # Coefficient de reflexion des tuiles
emissivite_tuile = 0.92

facteur_de_forme = 0.75  # portion du ciel visible par les tuiles (inclinaison du toit)


I am working on the file "Model02_tuile.ipynb".
The first task is:
Effet de la couverture nuageuse ?  L'aspect difficile dans l'estimation de la radiation solaire arrivant au sol est de correctement prendre en compte l'effet des nuages, et de la radiation diffuse. Il ne semble pas avoir de modèle standard, mais différents modèles empiriques dépendant du type de nuages et du climat local...
La formule la plus simple trouvée et qui me semble cohérente est :  $ \Phi =  I0 \, (1 - 0.75 \, C^{3.4}) $, $C$ étant la couverture nuageuse, entre 0 et 1. C'est à dire que même par un ciel entièrement nuageux il y a 25% du rayonement qui arrive au sol, et la relation est non linéaire pour les couvertures nuageuse intermédiaires.
Solar and terrestrial radiation dependent on the amount and type of cloud. Kasten, F., & Czeplak, G. (1980). Solar energy, 24(2), 177-189.

L'effet et la prise en compte de la radiation diffuse n'est pas encore claire pour moi, mais en comparant avec les valeurs de radiation données par Météoblue.com, j'obtiens les mêmes ordres de grandeurs.
Can you write Python code for it?

cloud_attenuation = lambda c: 1 - 0.75* c**(3.4)

# graph
c = np.linspace(0, 1, 50); plt.figure(figsize=(3, 3));
plt.plot( c, cloud_attenuation(c)); plt.axis('equal'); plt.ylim([0, 1]); plt.xlabel('Couverture nuageuse');
plt.ylabel('Atténuation du rayonement');

sundata['I_sol']  = sundata['I0']*cloud_attenuation( data['cloudCover']  )

# graph
sundata['I0'].plot( figsize=(14, 3), style='r', alpha=0.2 );plt.ylabel('W/m2');
sundata['I_sol'].plot( figsize=(14, 3), style='r' );plt.ylabel('W/m2');
plt.title('Flux direct dans la direction du soleil avec et sans nuages');


I am working on the file "Model02_tuile.ipynb".
The first task is:
Estimation des flux de chaleur sur les tuiles
<img src="images/sch_model02.jpg" width="400px" alt='schema mod02' />
L'équilibre thermique des tuiles du toit est calculé en prennant en compte :
    - Le rayonnement solaire direct sur le toit (phi(t))
    - Le rayonnement propre des tuiles (corps noir)
    - Le rayonnement thermique du ciel (corps noir)
    - La convection avec l'air extérieur (forcée par le vent, ou bien naturelle)
En revanche, sont considérés comme négligeable :
    - Le flux de chaleur à travers l'isolation du toit (~ 0.17 W/m2/K ).
    - L'inertie thermique des tuiles.
    - La pluie, et l'humidité de l'air.
Refroidissement par convection avec l'air ambiant
La convection correspond aux échanges de chaleur avec l'air (un fluide) en mouvement. Ce mouvement peut être forcée par une source extérieure (ventilateur), ou dans notre cas le vent. Le mouvement de l'air peut aussi être induit par les variations de température suivant l'altitude, et donc de densité. L'air chaude monte parce que moins dense que l'air froid. On parle alors de convection naturelle.
Le flux de chaleur par convection s'écrit sous la forme générale: $ h\, (T - T_{air}) $, où $h$ est le coefficient d'échange en W/m2/K. Sa valeur est habituellement entre 5 et 25 W/m2/K pour une surface en contact avec l'air ambiant.
Une valeur plus précisse de ce coefficient peut être obtenue en considérant les propriétés de l'écoulement (laminaire vs turbulent, forcée vs naturelle, interne vs externe) et la vitesse du vent. On trouve, par exemple, dans ce cours un resumé des modèles physique permetttant d'estimer $h$ dans différentes situations.
Á basse vitesse la convection naturelle prédomine, et $h$ est de l'ordre de 5 W/m2/K. Aux vitesses plus importantes, l'écoulement devient turbulent, et une loi en puissance donne une estimation de $h$ en fonction de la vitesse du vent.
Il semble que sur une structure réelle les coefficients mesurées sont plus important. Voir par exemple l'article A thermal model for photovoltaic panels under...
Can you write Python code for it?

def coef_echange_old( vitesse_vent ):
    """ Fonction du coefficient d'échange pour la convection thermique
        vitesse_vent: m/s
        h : W/m2/K
    """

    k = 0.0262 # W/m/K, conductivité thermique
    nu = 1.57e-5 #  m2.s−1 , viscosité cinématique air
    Pr = 0.708 # Nombre de Prandl

    L = 4 # m, taille caractéristique, ( toit ou tuile ? )

    Reynolds = vitesse_vent*L/nu

    if Reynolds > 3e5:
        # convection forcée, turbulent, Pr < 50  (cf. page 15)
        h = k/L * 0.036 *  Reynolds**(4/5) * Pr**(1/3)
    else:
        # convection naturelle
        h = 5

def coef_echange( vitesse_vent ):
    """ Fonction du coefficient d'échange pour la convection thermique
        - empirique -
        vitesse_vent: m/s
        h : W/m2/K
    """

    # h = 5.7*vitesse_vent + 11.14 # Sturrock, Wind faced
    h = 2.56*vitesse_vent + 8.55 # Test et al. (outdoor)
    return h

v_span = np.linspace( 0, 3, 100 )
h_span = [ coef_echange( v ) for v in v_span ]

plt.plot( v_span, h_span ); plt.xlabel('vitesse vent (m/s)');plt.ylabel("coeff. d'échange (W/m²/K)");


I am working on the file "Model02_tuile.ipynb".
The first task is:
Et il peut être intéressant aussi de voir l'effet de la pluie:
Can you write Python code for it?

plt.figure()
data['precipIntensity'].plot(figsize=(14, 2), color='k' ); plt.ylabel('mm/heure'); plt.title('pluie [mm/heure]');


I am working on the file "Model02_tuile.ipynb".
The first task is:
Irradiation solaire
Le calcul de l'irradiation solaire est avant tout un problème géométrique avec l'angle de l'axe de rotation de la terre, la position de la terre sur son orbite et notre position sur le globe. Il existe une libraire python réalisant ce calcul: pysolar.org. L'atténuation du flux solaire par l'épaisseur de la couche d'atmosphère traversée est aussi prise en compte.
Un affinement possible du modèle prend en compte la hauteur de l'horizon. Ceci est surtout utile en région de montagne (horizon à ~ 15° le matin chez moi) ce qui recule l'heure du levée de soleil. Voir le notebook ombres_montagnes.
De même, j'ai écris un module avec les quelques fonctions utiles me permettant d'obtenir un Dataframe unique.
Can you write Python code for it?

import sunradiation as sun

sundata = sun.buildmultidayDF( coords_gps, data.index  )

sundata['I0'].plot( figsize=(14, 3), style='r' );plt.ylabel('W/m2'); plt.title('Flux direct dans la direction du soleil');

plt.figure();
sundata['sunAzimuth'].plot( figsize=(14, 3), style='deeppink' );plt.ylabel('deg');
sundata['sunAltitude'].plot( figsize=(14, 3), style='seagreen' );
plt.title('Position du soleil'); plt.legend();


I am working on the file "Model02_tuile.ipynb".
The first task is:
L'angle du toit de mon appartement est d'environ 37° et l'orientation est de 50° par rapport au sud (50°+180° pour l'autre pan).
Can you write Python code for it?

theta = 37   # deg
phi_bastille = 50  # deg
phi_vercors = 50+180  # deg

data['flux_bastille'] = sundata['I_sol'] * sundata.apply( get_cosGamma, axis=1, args=(theta, phi_bastille) )
data['flux_vercors'] =  sundata['I_sol'] * sundata.apply( get_cosGamma, axis=1, args=(theta, phi_vercors) )

data[['flux_bastille', 'flux_vercors']].plot(figsize=(14, 5) )
plt.ylabel('W/m2');  plt.title('Flux solaire pour les deux pans de toit [W/m²]');


I am working on the file "Model02_tuile.ipynb".
The first task is:
Obtention des données météo
Le site web DarkSky.net fourni une API gratuite pour avoir les historiques de données météo à une certaine position GPS. Il existe aussi une libraire python darkskylib simplifiant les échanges avec l'API.
Par dessus, j'ai un module perso (weatherfeed) avec les fonctions de téléchargement jour par jour et de construction d'un DataFrame pandas unique.
Can you write Python code for it?

import weatherfeed as wf

# téléchargement des données:
data = wf.buildmultidayDF(startday, lastday, coords_gps )

data['temperature'].plot(figsize=(14, 3), color='r' ); plt.ylabel('°C'); plt.title('Température extérieure [°C]');


I am working on the file "Model02_tuile.ipynb".
The first task is:
Projection sur une surface inclinée
I0 est le flux dans la direction des rayons du soleil. Il faut donc projeter ce flux sur la surface receptrice qui est vue sous un certain angle, noté $\Gamma$ ici. Un peu de trigonométrie permet de calculer cet angle en fonction de l'orientation de la surface ($\theta$, $\varphi$) et de la position du soleil ($\theta_S$, $\varphi_S$) :
<img src="images/sch_angleprojection.jpg" width="400px" alt='schema mod02' />
Comme indiqué, on note $\Gamma$ l'angle entre le rayon du soleil $\vec r$ et la normal à la surface receptrice $\vec n$. Ces deux vecteurs sont unitaires. On obtient $cos\,\Gamma$ avec le produit scalaire de ces deux vecteurs :
$$
cos\,\Gamma = \vec r \bullet \vec n =
\begin{bmatrix} sin\, \varphi_S \,\, cos\, \theta_S  \ cos\, \varphi_S \,\, cos\, \theta_S  \ sin\, \theta_S \end{bmatrix} \bullet \begin{bmatrix} sin\, \varphi \,\, sin\, \theta  \ cos\, \varphi \,\, sin\, \theta  \ cos\, \theta \end{bmatrix}
$$
qui se simplifie :
$$
cos\,\Gamma = cos( \varphi - \varphi_S )\, . \,sin\, \theta \,.\, cos \, \theta_S + cos\,\theta \,.\, sin \, \theta_S
$$
Une surface d'aire $A$ vue sous l'angle $\Gamma$ a une surface projetée $A . cos\, \Gamma$. Le flux reçu par la surface est donc $I. cos\, \Gamma$.
Can you write Python code for it?

rad = lambda x: x*np.pi/180

def get_cosGamma(sundata_row, theta, phi):
    theta, phi = rad(theta), rad(phi)
    theta_S = rad( sundata_row['sunAltitude']  )
    phi_S = rad( sundata_row['sunAzimuth']  )

    cosGamma = np.cos( phi - phi_S )*np.sin( theta )*np.cos( theta_S ) \
            + np.cos( theta )*np.sin( theta_S )
    if  cosGamma<0 :  cosGamma = 0
    return cosGamma


I am working on the file "Model02_tuile.ipynb".
The first task is:
Sont disponible aussi pleins d'autres mesures, et en particulier la vitesse du vent et la couverture nuageuse.
Can you write Python code for it?

data['windSpeed'].plot(figsize=(14, 3), color='g' ); plt.ylabel('m/s'); plt.title('Vitesse du vent [m/s]');

data['cloudCover'].plot(figsize=(14, 3), color='r' ); plt.ylabel('%'); plt.title('Couverture nuageuse [%]');


I am working on the file "ModelResults_Final.ipynb".
The first task is:
Modeling
Windowed Leave-one-out cross validation
Though the temporal information is redundantly encoded so we can use a model with independent observations, the observations are still not actually independent. Therefore, we do not want to evaluate the model on a test set that is dependent on the training set. To do this, we use a variation of leave-one-out cross validation that excludes a window around the test observation from use in the training set (reference). The window extends forward and backward from the test observation by the number of correlated lags derived from ARIMA.
In practice, this amounts to doing LOOCV but also leaving out a larger window from the training set.
<img src="WLOOCV.png">
Can you write Python code for it?

def evaluateModel_LOOCV(model, X, y, n_lags=14):
    n = len(y)
    RSS = 0
    TSS = np.sum(np.square(y-np.mean(y)))
    pred = np.zeros(n)

    for i in range(n):

        X_test = X.iloc[i]
        y_test = y.iloc[i]

        train_indexes = list(range(i-n_lags))
        train_indexes.extend(range(i+n_lags,n))

        X_train = X.iloc[train_indexes]
        y_train = y.iloc[train_indexes]
        scaler = StandardScaler()

        X_train_scaled = scaler.fit_transform(X_train)
        X_test_scaled = scaler.transform(X_test)

        model.fit(X_train_scaled, y_train)
        pred[i] = model.predict(X_test_scaled)
        RSS += np.square(pred[i]-y_test)
    R2 = (1-RSS/TSS)
    MSE = RSS/n
    return(R2, MSE, pred)


I am working on the file "ModelResults_Final.ipynb".
The first task is:
SVM
Can you write Python code for it?

gammas = [.00001,.0001]
R2 = np.zeros(len(gammas))
for i, gamma in enumerate(gammas):
    model = SVR(C = 100000, kernel = 'rbf', gamma=gamma)
    R2[i] = evaluateModel_LOOCV(model, X_subset, y, n_lags=3)[0]

pd.DataFrame(R2.reshape((1,len(R2))), columns = gammas, index=['R-squared'])

model = SVR(C = 100000, kernel = 'rbf', gamma=0.00001)
R2_SVR, MSE_SVR, pred_SVR = evaluateModel_LOOCV(model, X_subset, y, n_lags=3)

print 'All Predictor SVR Forecasting MSE = {:.3f}'.format(MSE_SVR)


I am working on the file "ModelResults_Final.ipynb".
The first task is:
Then, we can put all of the columns into one dataframe.
Can you write Python code for it?

topicsSentiCCI = pd.concat([CCIlimited, topicsDF, sentiDF.full_avg], axis=1)


I am working on the file "ModelResults_Final.ipynb".
The first task is:
Trying various models
With the features selected via Lasso, we tried linear regression, SVM, random forest, and gradient boosting. Linear regression and a tuned SVM were both very successful.
Can you write Python code for it?

lr = LinearRegression()
R2_LR, MSE_LR, pred_LR = evaluateModel_LOOCV(lr, X_subset, y, n_lags=3)

print 'All Predictor Forecasting MSE = {:.3f}'.format(MSE_LR)

lr = LinearRegression()
R2_CCI, MSE_CCI, predCCI = evaluateModel_LOOCV(lr, X_justCCI, y, n_lags=3)

print 'Just CCI Forecasting MSE = {:.3f}'.format(MSE_CCI)


I am working on the file "ModelResults_Final.ipynb".
The first task is:
We can look at the coefficients from the Lasso model and drop the features that have coefficients of zero.
Can you write Python code for it?

pd.DataFrame(lasso.coef_.reshape((1,len(lasso.coef_))), columns = X.columns)

X_subset = X.drop(X.columns[np.where(lasso.coef_==0)], axis=1)


I am working on the file "ModelSampler-Copy1.ipynb".
The first task is:
Do Geweke Test to see if trace has converged. Burn-in looks to be around 2000.
Can you write Python code for it?

# autocorrelation with lag t
def rhot(x, t):
    n = len(x)
    return np.corrcoef(x[0:(n-t)], x[t:n])[0,1]

# Geweke function
def Geweke(trace, intervals, length):
    # take two parts of the chain.
    # subsample lenght
    nsl=length
    jump = int(0.9*len(trace)/(2*intervals))
    first = 0.1*len(trace)
    z =np.empty(intervals)
    for k in np.arange(0, intervals):
        # beg of each sub samples
        bega=first+k*jump
        begb = len(trace)/2 + k*jump

        sub_trace_a = trace[bega:bega+nsl]
        sub_trace_b = trace[begb:begb+nsl]

        theta_a = np.mean(sub_trace_a)
        theta_b = np.mean(sub_trace_b)
        rho_a, rho_b = 1.0, 1.0
        # only compute autocorrelation at lag 1-0.1*nsl.
        for i in xrange(int(0.1*nsl)):
            rho_a += 2*rhot(sub_trace_a, i+1)
            rho_b += 2*rhot(sub_trace_b, i+1)

        # estimate the variance roughly
        var_a  = np.var(sub_trace_a)*rho_a/length
        var_b  = np.var(sub_trace_b)*rho_b/length

        z[k] = (theta_a-theta_b)/np.sqrt( var_a + var_b)

    return z

burnin = 2000
gewekeB0 = Geweke(B[burnin:,0], 10, 5000)
gewekeB1 = Geweke(B[burnin:,1], 10, 5000)
gewekeB2 = Geweke(B[burnin:,2], 10, 5000)
gewekeB3 = Geweke(B[burnin:,3], 10, 5000)
gewekesig2 = Geweke(sig2[3000:], 10, 300)

plt.figure(1)
plt.subplot(211)
plt.plot(gewekeB0)
plt.plot(gewekeB1)
plt.plot(gewekeB2)
plt.plot(gewekeB3)
plt.ylim((-2,2))

plt.subplot(212)
plt.plot(gewekesig2)
plt.ylim((-5,10))
plt.show()


I am working on the file "ModelSampler-Copy1.ipynb".
The first task is:
I would want to get rid of all rows missing any of the data before running the sampler.
Can you write Python code for it?

train = train[train['longitude'] > 1]
train = train[train['latitude'] < 0]
train = train[train['construction_year'] != 0]
train = train[train['gps_height'] != 0]

features = ['longitude','latitude']
trainLoc = train[features]
#hasLocIdx = train['longitude']>1
#trainLoc = trainLoc[hasLocIdx] #remove rows with empty location data
#trainLoc.head()
#hasLocIdx.head()

trainID = train['id']
trainLabelsLoc = trainLabels[trainLabels['id'].isin(trainID)] #only keep labels corresponding to rows with non-empty location data

nNeighbors = 60
clf = KNeighborsClassifier(n_neighbors=nNeighbors,weights='distance',algorithm='auto')
clf.fit(trainLoc[features], trainLabelsLoc['status_group'])
#preds = clf.predict(test[features])

features=['construction_year', 'gps_height']
trainFeatures = train[features]
trainFeatures['age'] = 2015 - trainFeatures['construction_year']
trainFeatures = trainFeatures.drop('construction_year', 1)
#trainFeatures.head()

kNNFunctional = clf.predict_proba(trainFeatures)[:,0]
#TODO: get more meaningful data out of kNN algorithm?
x = np.vstack((trainFeatures['age'],trainFeatures['gps_height'],kNNFunctional))

trainLabels = trainLabels[trainLabels['id'].isin(trainID)] #get rid of Labels for which there is no corresponding data

numFeatures = 3
numBeta = numFeatures + 1  #1 more for the constant
#TODO: Convert labels into numbers
trainLabelsVect = pd.get_dummies(trainLabels['status_group'])
trainLabelsVect['functionality'] = trainLabelsVect['functional'] + 0.5*trainLabelsVect['functional needs repair']

#B, sig2= mcmc(np.zeros(numBeta), 0.2, x, trainLabelsVect['functionality'], 100, 0, 0.5, 0.5)

B, sig2= mcmc(np.zeros(numBeta), 0.2, x, trainLabelsVect['functionality'], 10000, 0, 0.1, 0.1)

plt.plot(B[:,0])
plt.plot(B[:,1])
plt.plot(B[:,2])
plt.plot(B[:,3])
print np.mean(B[:,1])

plt.plot(sig2)


I am working on the file "ModelSampler.ipynb".
The first task is:
Below are functions to import the data and process them for use in our MH sampler.
Can you write Python code for it?

def import_data():
    """Import data and labels and separate into training and testing data for cross-validation."""
    #import data and labels
    train_data = pd.read_csv('Waterpump-training-values.csv')
    train_labels = pd.read_csv('Waterpump-training-labels.csv')

    #separating dataset into training and testing for cross-validation - 90% into training
    test_idx = np.random.uniform(0, 1, len(train_data)) <= 0.9
    train = train_data[test_idx==True]
    trainLabels = train_labels[test_idx==True]
    test = train_data[test_idx==False]
    testLabels = train_labels[test_idx==False]

    return train, trainLabels, test, testLabels

def processAllData(train, trainLabels, test, testLabels):
    """Process data for use in training and testing."""
    train, nFeatures = processData(train)
    test, _ = processData(test)

    trainLabelsVect = pd.get_dummies(trainLabels['status_group'])
    trainLabelsVect['functionality'] = trainLabelsVect['functional'] + 0.5*trainLabelsVect['functional needs repair']

    return train, trainLabelsVect, test, testLabels, nFeatures

def processData(data):
    """Pick out features we want to use in our model."""
    features = ['longitude', 'latitude', 'age', 'gps_height', 'dry', 'population']
    nFeatures = len(features)
    data['age'] = 2015 - data['construction_year']
    data['dry'] = data['quantity'] == 'dry'
    return np.transpose(data[features].values), nFeatures


I am working on the file "ModelSampler.ipynb".
The first task is:
The blockwise update has an extremely low acceptance rate of 0.039, even when we start the parameters near where the componentwise updating converges. This is because we have 8 parameters!
Can you write Python code for it?

plt.plot(B[:,0], label='b0')
plt.plot(B[:,1], label='b1')
plt.plot(B[:,2], label='b2')
plt.plot(B[:,3], label='b3')
plt.plot(B[:,4], label='b4')
plt.plot(B[:,5], label='b5')
plt.plot(B[:,6], label='b6')
plt.legend()


I am working on the file "ModelSampler.ipynb".
The first task is:
We implement the MH algorithm using both blockwise updating and componentwise updating.
Can you write Python code for it?

# mcmc algorithm
def mcmc(b_init, sig2, x, y, N, burnin, be, sig2e, model):
    #MH with blockwise updating.
    nFeat = len(b_init) - 1

    B = np.zeros((N,nFeat + 1))
    Sig2 = np.zeros(N)

    b_prev = b_init
    sig2_prev = sig2
    count = 0
    r = np.random.random(N)

    for i in xrange(N):
        b_star = np.random.normal(b_prev,be)

        sig2_star = abs(np.random.normal(sig2_prev, sig2e))

        p = logpost(b_star, sig2_star, x, y, model) - logpost(b_prev, sig2_prev, x, y, model)
        if np.log(r[i]) < p:
            b_prev = b_star
            sig2_prev = sig2_star
            count += 1

        B[i] = b_prev
        Sig2[i] = sig2_prev
    print "The acceptance rate is "+ str(float(count)/N)+"."
    return B, Sig2

def mcmcComp(b_init, sig2, x, y, N, burnin, be, sig2e, model):
    #MCMC with componentwise updating instead of block updating
    nParam = len(b_init)

    B = np.zeros((N,nParam))
    Sig2 = np.zeros(N)

    b_prev = b_init
    sig2_prev = sig2
    count = 0
    r = np.random.random((N,nParam+1))

    for i in xrange(N):
        #updating all the beta parameters
        for j in xrange(nParam):
            b_star = np.copy(b_prev)
            b_star[j] = np.random.normal(b_prev[j],be[j])
            p = logpost(b_star, sig2_prev, x, y, model) - logpost(b_prev, sig2_prev, x, y, model)
            if np.log(r[i,j]) < p:
                b_prev = b_star
                count += 1

        #updating sig2
        sig2_star = abs(np.random.normal(sig2_prev, sig2e))
        p = logpost(b_prev, sig2_star, x, y, model) - logpost(b_prev, sig2_prev, x, y, model)
        if np.log(r[i,-1]) < p:
            sig2_prev = sig2_star
            count += 1
        B[i] = b_prev
        Sig2[i] = sig2_prev

    print "The acceptance rate is "+ str(float(count)/(N*(nParam+1)))+"."
    return B, Sig2


I am working on the file "ModelSelection.ipynb".
The first task is:
Find most appropriate number of states using cross validation
Here we split the data into training, validation, and test sets. We monitor the average log probability per sequence (normalized by length) for each of these sets, and we use the validation set to choose the number of model states $m$.
Note to self: I should re-write my data splitting routines to allow me to extract as many subsets as I want, so that I can do k-fold cross validation.
Can you write Python code for it?

## bin ALL spikes
ds = 0.125 # bin spikes into 125 ms bins (theta-cycle inspired)
binned_spikes_all = klab.bin_spikes(spikes.data, ds=ds, fs=spikes.samprate, verbose=True)

## identify boundaries for running (active) epochs and then bin those observations into separate sequences:
runbdries = klab.get_boundaries_from_bins(eeg.samprate,bins=speed.active_bins,bins_fs=60)
binned_spikes_bvr = klab.bin_spikes(spikes.data, fs=spikes.samprate, boundaries=runbdries, boundaries_fs=eeg.samprate, ds=ds)

## stack data for hmmlearn:
seq_stk_bvr = sq.data_stack(binned_spikes_bvr, verbose=True)
seq_stk_all = sq.data_stack(binned_spikes_all, verbose=True)

## split data into train, test, and validation sets:
tr_b,vl_b,ts_b = sq.data_split(seq_stk_bvr, tr=60, vl=20, ts=20, randomseed = 0, verbose=False)

Smax = 40
S = np.arange(start=5,step=1,stop=Smax+1)

tr_ll = []
vl_ll = []
ts_ll = []

for num_states in S:
    clear_output(wait=True)
    print('Training and evaluating {}-state hmm'.format(num_states))
    sys.stdout.flush()
    myhmm = sq.hmm_train(tr_b, num_states=num_states, n_iter=30, verbose=False)
    tr_ll.append( (np.array(list(sq.hmm_eval(myhmm, tr_b)))/tr_b.sequence_lengths ).mean())
    vl_ll.append( (np.array(list(sq.hmm_eval(myhmm, vl_b)))/vl_b.sequence_lengths ).mean())
    ts_ll.append( (np.array(list(sq.hmm_eval(myhmm, ts_b)))/ts_b.sequence_lengths ).mean())

clear_output(wait=True)
print('Done!')
sys.stdout.flush()

num_states = 35

fig = plt.figure(1, figsize=(12, 4))
ax = fig.add_subplot(111)

ax.annotate('plateau at approx ' + str(num_states), xy=(num_states, -38.5), xycoords='data',
                xytext=(-140, -30), textcoords='offset points',
                arrowprops=dict(arrowstyle="->",
                                connectionstyle="angle3,angleA=0,angleB=-90"),
                )

ax.plot(S, tr_ll, lw=1.5, label='train')
ax.plot(S, vl_ll, lw=1.5, label='validation')
ax.plot(S, ts_ll, lw=1.5, label='test')
ax.legend(loc=2)
ax.set_xlabel('number of states')
ax.set_ylabel('normalized (to...


I am working on the file "ModelSelection.ipynb".
The first task is:
Load data
Here we consider lin2 data for gor01 on the first recording day (6-7-2006), since this session had the most units (91) of all the gor01 sessions, and lin2 has position data, whereas lin1 only has partial position data.
Can you write Python code for it?

datadirs = ['/home/etienne/Dropbox/neoReader/Data',
            'C:/etienne/Dropbox/neoReader/Data',
            '/Users/etienne/Dropbox/neoReader/Data']

fileroot = next( (dir for dir in datadirs if os.path.isdir(dir)), None)

animal = 'gor01'; month,day = (6,7); session = '16-40-19' # 91 units

spikes = load_data(fileroot=fileroot, datatype='spikes',animal=animal, session=session, month=month, day=day, fs=32552, verbose=False)
eeg = load_data(fileroot=fileroot, datatype='eeg', animal=animal, session=session, month=month, day=day,channels=[0,1,2], fs=1252, starttime=0, verbose=False)
posdf = load_data(fileroot=fileroot, datatype='pos',animal=animal, session=session, month=month, day=day, verbose=False)
speed = klab.get_smooth_speed(posdf,fs=60,th=8,cutoff=0.5,showfig=False,verbose=False)