I am working on the file "Learning on marked data/Week 3/Preprocessing_LR.ipynb".
The first task is:
Теория по логистической регрессии
После осознания того, какую именно задачу требуется решить на этих данных, следующим шагом при реальном анализе был бы подбор подходящего метода. В данном задании выбор метода было произведён за вас, это логистическая регрессия. Кратко напомним вам используемую модель.
Логистическая регрессия предсказывает вероятности принадлежности объекта к каждому классу. Сумма ответов логистической регрессии на одном объекте для всех классов равна единице.
$$ \sum_{k=1}^K \pi_{ik} = 1, \quad \pi_k \equiv P\,(y_i = k \mid x_i, \theta), $$
где:
- $\pi_{ik}$ - вероятность принадлежности объекта $x_i$ из выборки $X$ к классу $k$
- $\theta$ - внутренние параметры алгоритма, которые настраиваются в процессе обучения, в случае логистической регрессии - $w, b$
Из этого свойства модели в случае бинарной классификации требуется вычислить лишь вероятность принадлежности объекта к одному из классов (вторая вычисляется из условия нормировки вероятностей). Эта вероятность вычисляется, используя логистическую функцию:
$$ P\,(y_i = 1 \mid x_i, \theta) = \frac{1}{1 + \exp(-w^T x_i-b)} $$
Параметры $w$ и $b$ находятся, как решения следующей задачи оптимизации (указаны функционалы с L1 и L2 регуляризацией, с которыми вы познакомились в предыдущих заданиях):
L2-regularization:
$$ Q(X, y, \theta) = \frac{1}{2} w^T w + C \sum_{i=1}^l \log ( 1 + \exp(-y_i (w^T x_i + b ) ) ) \longrightarrow \min\limits_{w,b} $$
L1-regularization:
$$ Q(X, y, \theta) = \sum_{d=1}^D |w_d| + C \sum_{i=1}^l \log ( 1 + \exp(-y_i (w^T x_i + b ) ) ) \longrightarrow \min\limits_{w,b} $$
$C$ - это стандартный гиперпараметр модели, который регулирует то, насколько сильно мы позволяем модели подстраиваться под данные.
Предобработка данных
Из свойств данной модели следует, что:
- все $X$ должны быть числовыми данными (в случае наличия среди них категорий, их требуется некоторым способом преобразовать в вещественные числа)
- среди $X$ не должно быть пропущенных значений (т.е. все пропущенные значения перед применением модели следует каким-то...
Can you write Python code for it?


I am working on the file "Learning on marked data/Week 4/grad_boosting.ipynb".
The first task is:
Задание 1
Как вы уже знаете из лекций, бустинг - это метод построения композиций базовых алгоритмов с помощью последовательного добавления к текущей композиции нового алгоритма с некоторым коэффициентом.
Градиентный бустинг обучает каждый новый алгоритм так, чтобы он приближал антиградиент ошибки по ответам композиции на обучающей выборке. Аналогично минимизации функций методом градиентного спуска, в градиентном бустинге мы подправляем композицию, изменяя алгоритм в направлении антиградиента ошибки.
Воспользуйтесь формулой из лекций, задающей ответы на обучающей выборке, на которые нужно обучать новый алгоритм (фактически это лишь чуть более подробно расписанный градиент от ошибки), и получите частный ее случай, если функция потерь L - квадрат отклонения ответа композиции a(x) от правильного ответа y на данном x.
Если вы давно не считали производную самостоятельно, вам поможет таблица производных элементарных функций (которую несложно найти в интернете) и правило дифференцирования сложной функции. После дифференцирования квадрата у вас возникнет множитель 2 — т.к. нам все равно предстоит выбирать коэффициент, с которым будет добавлен новый базовый алгоритм, проигноируйте этот множитель при дальнейшем построении алгоритма.
Can you write Python code for it?


I am working on the file "Learning on marked data/Week 4/grad_boosting.ipynb".
The first task is:
Задание 3
Вас может также беспокоить, что двигаясь с постоянным шагом, вблизи минимума ошибки ответы на обучающей выборке меняются слишком резко, перескакивая через минимум.
Попробуйте уменьшать вес перед каждым алгоритмом с каждой следующей итерацией по формуле 0.9 / (1.0 + i), где i - номер итерации (от 0 до 49). Используйте качество работы алгоритма как ответ в пункте 3.
В реальности часто применяется следующая стратегия выбора шага: как только выбран алгоритм, подберем коэффициент перед ним численным методом оптимизации таким образом, чтобы отклонение от правильных ответов было минимальным. Мы не будем предлагать вам реализовать это для выполнения задания, но рекомендуем попробовать разобраться с такой стратегией и реализовать ее при случае для себя.
Can you write Python code for it?

base_algorithms_list = []
coefficients_list = []
error_3 = []

estimator = tree.DecisionTreeRegressor(max_depth=5, random_state = 42)
estimator.fit(X_train, y_train)
base_algorithms_list.append(estimator)
coefficients_list.append(0.9)
err = np.sqrt(metrics.mean_squared_error(y_test, gbm_predict(X_test)))
error_3.append(err)

for i in range(1, 50):
    estimator = tree.DecisionTreeRegressor(max_depth=5,random_state = 42)

    y_pred = gbm_predict(X_train)

    estimator.fit(X_train, grad(y_train, y_pred))
    base_algorithms_list.append(estimator)
    coefficients_list.append(0.9/(1. + i))

    err = np.sqrt(metrics.mean_squared_error(y_test, gbm_predict(X_test)))
    error_3.append(err)
    print 'error № ',i,' = ', err,'\n'

ans2 = error_3[49]
print ans2
with open('grad_boost_ans2.txt', 'w') as file_out:
        file_out.write(str(ans2))


I am working on the file "Learning on marked data/Week 4/grad_boosting.ipynb".
The first task is:
Задание 4
Реализованный вами метод - градиентный бустинг над деревьями - очень популярен в машинном обучении. Он представлен как в самой библиотеке sklearn, так и в сторонней библиотеке XGBoost, которая имеет свой питоновский интерфейс. На практике XGBoost работает заметно лучше GradientBoostingRegressor из sklearn, но для этого задания вы можете использовать любую реализацию.
Исследуйте, переобучается ли градиентный бустинг с ростом числа итераций (и подумайте, почему), а также с ростом глубины деревьев. На основе наблюдений выпишите через пробел номера правильных из приведенных ниже утверждений в порядке возрастания номера (это будет ответ в п.4):
1. С увеличением числа деревьев, начиная с некоторого момента, качество работы градиентного бустинга не меняется существенно.

2. С увеличением числа деревьев, начиная с некоторого момента, градиентный бустинг начинает переобучаться.

3. С ростом глубины деревьев, начиная с некоторого момента, качество работы градиентного бустинга на тестовой выборке начинает ухудшаться.

4. С ростом глубины деревьев, начиная с некоторого момента, качество работы градиентного бустинга перестает существенно изменяться
Can you write Python code for it?

# iteration
err_list = []
coefficients_list = []
base_algorithms_list = []

estimator = tree.DecisionTreeRegressor(max_depth=5, random_state = 42)
estimator.fit(X_train, y_train)
base_algorithms_list.append(estimator)
coefficients_list.append(0.9)
err = np.sqrt(metrics.mean_squared_error(y_test, gbm_predict(X_test)))
err_list.append(err)

for i in range(1, 100):
    estimator = tree.DecisionTreeRegressor(max_depth=5,random_state = 42)

    y_pred = gbm_predict(X_train)

    estimator.fit(X_train, grad(y_train, y_pred))
    base_algorithms_list.append(estimator)
    coefficients_list.append(0.9/(1. + i))

    err = np.sqrt(metrics.mean_squared_error(y_test, gbm_predict(X_test)))
    err_list.append(err)
    print i,

plt.plot(range(0, 100), err_list)

# max_depth
err_list = []
coefficients_list = []
base_algorithms_list = []
depths = range(2, 2*50 + 2, 2)

estimator = tree.DecisionTreeRegressor(max_depth=depths[0], random_state = 42)
estimator.fit(X_train, y_train)
base_algorithms_list.append(estimator)
coefficients_list.append(0.9)
err = np.sqrt(metrics.mean_squared_error(y_test, gbm_predict(X_test)))
err_list.append(err)

for i in range(1, 50):
    estimator = tree.DecisionTreeRegressor(max_depth=depths[i],random_state = 42)

    y_pred = gbm_predict(X_train)

    estimator.fit(X_train, grad(y_train, y_pred))
    base_algorithms_list.append(estimator)
    coefficients_list.append(0.9/(1.0 + i))

    err = np.sqrt(metrics.mean_squared_error(y_test, gbm_predict(X_test)))
    err_list.append(err)
    print i,

plt.plot(depths, err_list)

ans3 = '2 3'
with open('grad_boost_ans3.txt', 'w') as file_out:
        file_out.write(ans3)


I am working on the file "Learning on marked data/Week 5/task_nn.ipynb".
The first task is:
В качестве альтернативного варианта, можно выполнить загрузку данных напрямую из репозитория UCI, воспользовавшись библиотекой urllib.
Can you write Python code for it?

import urllib
# URL for the Wine Quality Data Set (UCI Machine Learning Repository)
url = "https://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-red.csv"
# загрузка файла
f = urllib.urlopen(url)
f.readline()  # пропуск заголовочной строки
data = np.loadtxt(f, delimiter=';')


I am working on the file "Learning on marked data/Week 5/task_nn.ipynb".
The first task is:
Выделим из данных целевую переменную. Классы в задаче являются несбалинсированными: основной доле объектов поставлена оценка качества от 5 до 7. Приведем задачу к трехклассовой: объектам с оценкой качества меньше пяти поставим оценку 5, а объектам с оценкой качества больше семи поставим 7.
Can you write Python code for it?

TRAIN_SIZE = 0.7 # Разделение данных на обучающую и контрольную части в пропорции 70/30%
from sklearn.cross_validation import train_test_split

y = data[:, -1]
np.place(y, y < 5, 5)
np.place(y, y > 7, 7)
y -= min(y)
X = data[:, :-1]
X = normalize(X)

X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=TRAIN_SIZE, random_state=0)


I am working on the file "Learning on marked data/Week 5/task_nn.ipynb".
The first task is:
Выполним загрузку данных
Can you write Python code for it?

with open('winequality-red.csv') as f:
    f.readline()  # пропуск заголовочной строки
    data = np.loadtxt(f, delimiter=';')


I am working on the file "Learning on marked data/Week 5/task_nn.ipynb".
The first task is:
Выполним оптимизацию параметров сети. График ниже показывает сходимость функции ошибки на обучающей/контрольной части.
Can you write Python code for it?

random.seed(0)
# Модуль настройки параметров pybrain использует модуль random; зафиксируем seed для получения воспроизводимого результата
trainer = BackpropTrainer(net, dataset=ds_train) # Инициализируем модуль оптимизации
err_train, err_val = trainer.trainUntilConvergence(maxEpochs=MAX_EPOCHS)
line_train = plt.plot(err_train, 'b', err_val, 'r') # Построение графика
xlab = plt.xlabel('Iterations')
ylab = plt.ylabel('Error')


I am working on the file "Learning on marked data/Week 5/task_nn.ipynb".
The first task is:
Двуслойная нейронная сеть
Двуслойная нейронная сеть представляет собой функцию распознавания, которая може быть записана в виде следующей суперпозиции:
$f(x,W)=h^{(2)}\left(\sum\limits_{i=1}^D w_i^{(2)}h^{(1)}\left(\sum\limits_{j=1}^n w_{ji}^{(1)}x_j+b_i^{(1)}\right)+b^{(2)}\right)$, где
$x$ -- исходный объект (сорт вина, описанный 11 признаками), $x_j$ -- соответствующий признак,
$n$ --  количество нейронов во входном слое сети, совпадающее с количеством признаков,
$D$ --  количество нейронов в скрытом слое сети,
$w_i^{(2)}, w_{ji}^{(1)}, b_i^{(1)}, b^{(2)}$ --  параметры сети, соответствующие весам нейронов,
$h^{(1)}, h^{(2)}$ -- функции активации.
В качестве функции активации на скрытом слое сети используется линейная функция. На выходном слое сети используется функция активации softmax, являющаяся обобщением сигмоидной функции на многоклассовый случай:
$y_k=\text{softmax}k(a_1,...,a_k)=\frac{\exp(a_k)}{\sum{k=1}^K\exp(a_k)}.$
Настройка параметров сети
Оптимальные параметры сети $W_{opt}$ определяются путем минимизации функции ошибки:
$W_{opt}=\arg\min\limits_{W}L(W)+\lambda\|W\|^2$.
Здесь $L(W)$ является функцией ошибки многоклассовой классификации,
$L(W)=- \sum^N_{n=1}\sum^K_{k=1} t_{kn} log(y_{kn}),$
$t_{kn}$ -- бинарно закодированные метки классов, $K$ -- количество меток, $N$ -- количество объектов,
а $\lambda\|W\|^2$ является регуляризующим слагаемым, контролирующим суммарный вес параметров сети и предотвращающий эффект переобучения.
Оптимизация параметров выполняется методом обратного распространения ошибки (backpropagation).
Выполним загрузку основных модулей: ClassificationDataSet -- структура данных pybrain, buildNetwork -- инициализация нейронной сети, BackpropTrainer -- оптимизация параметров сети методом backpropagation, SoftmaxLayer -- функция softmax, соответствующая выходному слою сети, percentError -- функцию подсчета ошибки классификации (доля неправильных ответов).
Can you write Python code for it?

from pybrain.datasets import ClassificationDataSet # Структура данных pybrain
from pybrain.tools.shortcuts import buildNetwork
from pybrain.supervised.trainers import BackpropTrainer
from pybrain.structure.modules import SoftmaxLayer
from pybrain.utilities import percentError


I am working on the file "Learning on marked data/Week 5/task_nn.ipynb".
The first task is:
Задание. Определение оптимального числа нейронов.
В задании требуется исследовать зависимость ошибки на контрольной выборке в зависимости от числа нейронов в скрытом слое сети. Количество нейронов, по которому предполагается провести перебор, записано в векторе
hidden_neurons_num = [50, 100, 200, 500, 700, 1000]

Для фиксированного разбиения на обучающую и контрольную части подсчитайте долю неправильных ответов (ошибок) классификации на обучении/контроле в зависимости от количества нейронов в скрытом слое сети. Запишите результаты в массивы res_train_vec и res_test_vec, соответственно. С помощью функции plot_classification_error постройте график зависимости ошибок на обучении/контроле от количества нейронов. Являются ли графики ошибок возрастающими/убывающими? При каком количестве нейронов достигается минимум ошибок классификации?

С помощью функции write_answer_nn запишите в выходной файл число: количество нейронов в скрытом слое сети, для которого достигается минимум ошибки классификации на контрольной выборке.
Can you write Python code for it?

random.seed(0) # Зафиксируем seed для получния воспроизводимого результата
np.random.seed(0)

def plot_classification_error(hidden_neurons_num, res_train_vec, res_test_vec):
# hidden_neurons_num -- массив размера h, содержащий количество нейронов, по которому предполагается провести перебор,
#   hidden_neurons_num = [50, 100, 200, 500, 700, 1000];
# res_train_vec -- массив размера h, содержащий значения доли неправильных ответов классификации на обучении;
# res_train_vec -- массив размера h, содержащий значения доли неправильных ответов классификации на контроле
    plt.figure()
    plt.plot(hidden_neurons_num, res_train_vec)
    plt.plot(hidden_neurons_num, res_test_vec, '-r')

def write_answer_nn(optimal_neurons_num):
    with open("nnets_answer1.txt", "w") as fout:
        fout.write(str(optimal_neurons_num))

hidden_neurons_num = [50, 100, 200, 500, 700, 1000]
res_train_vec = list()
res_test_vec = list()

for nnum in hidden_neurons_num:
    # Put your code here
    # Не забудьте про инициализацию весов командой np.random.random((len(net.params)))
    net = buildNetwork(ds_train.indim, nnum, ds_train.outdim, outclass=SoftmaxLayer)

    init_params = np.random.random((len(net.params))) # Инициализируем веса сети для получения воспроизводимого результата
    net._setParameters(init_params)

    trainer = BackpropTrainer(net, dataset=ds_train) # Инициализируем модуль оптимизации
    err_train, err_val = trainer.trainUntilConvergence(maxEpochs=MAX_EPOCHS)

    res_train = net.activateOnDataset(ds_train).argmax(axis=1) # Подсчет результата на обучающей выборке
    res_test = net.activateOnDataset(ds_test).argmax(axis=1) # Подсчет результата на тестовой выборке

    res_train_vec.append(percentError(res_train, ds_train['target'].argmax(axis=1)))
    res_test_vec.append(percentError(res_test, ds_test['target'].argmax(axis=1)))
# Постройте график зависимости ошибок на обучении и контроле в зависимости от количества нейронов
plot_classification_error(hidden_neurons_num, res_train_vec, res_test_vec) ...


I am working on the file "Learning on marked data/Week 5/task_nn.ipynb".
The first task is:
Инициализируем структуру данных ClassificationDataSet, используемую библиотекой pybrain. Для инициализации структура принимает два аргумента: количество признаков np.shape(X)[1] и количество различных меток классов len(np.unique(y)).
Кроме того, произведем бинаризацию целевой переменной с помощью функции _convertToOneOfMany( ) и разбиение данных на обучающую и контрольную части.
Can you write Python code for it?

# Конвертация данных в структуру ClassificationDataSet
# Обучающая часть
ds_train = ClassificationDataSet(np.shape(X)[1], nb_classes=len(np.unique(y_train)))
# Первый аргумент -- количество признаков np.shape(X)[1], второй аргумент -- количество меток классов len(np.unique(y_train)))
ds_train.setField('input', X_train) # Инициализация объектов
ds_train.setField('target', y_train[:, np.newaxis]) # Инициализация ответов; np.newaxis создает вектор-столбец
ds_train._convertToOneOfMany( ) # Бинаризация вектора ответов
# Контрольная часть
ds_test = ClassificationDataSet(np.shape(X)[1], nb_classes=len(np.unique(y_train)))
ds_test.setField('input', X_test)
ds_test.setField('target', y_test[:, np.newaxis])
ds_test._convertToOneOfMany( )


I am working on the file "Learning on marked data/Week 5/task_nn.ipynb".
The first task is:
Рассчитаем значение доли неправильных ответов на обучающей и контрольной выборке.
Can you write Python code for it?

res_train = net.activateOnDataset(ds_train).argmax(axis=1) # Подсчет результата на обучающей выборке
print 'Error on train: ', percentError(res_train, ds_train['target'].argmax(axis=1)), '%' # Подсчет ошибки
res_test = net.activateOnDataset(ds_test).argmax(axis=1) # Подсчет результата на тестовой выборке
print 'Error on test: ', percentError(res_test, ds_test['target'].argmax(axis=1)), '%' # Подсчет ошибки


I am working on the file "Learning/1DLBP with CUDA.ipynb".
The first task is:
A good thing, too, because this is actually a way to debug a CUDA kernel in Python

Use numba to compile this simple code to something non-Python. For this we just decorate the definition like so:
Can you write Python code for it?

@jit("int32[:](uint8[:], int64, int32[:], int32[:])", nopython=True)
def extract_1dlbp_cpu_jit(input, neighborhood, powers, res):
    maxThread = 512
    blockDim = maxThread
    gridDim = (len(input) - 2 * neighborhood + blockDim) / blockDim

    for block in range(0, gridDim):
        for thread in range(0, blockDim):
            r = 0
            i = blockDim * block + thread
            if i < input.shape[0] - 2 * neighborhood:
                i += neighborhood
                for j in range(i - neighborhood, i):
                    if input[j] >= input[i]:
                        r += powers[j - i + neighborhood]

                for j in range(i + 1, i + neighborhood + 1):
                    if input[j] >= input[i]:
                        r += powers[j - i + neighborhood - 1]

                res[r] += 1
    return res


I am working on the file "Learning/1DLBP with CUDA.ipynb".
The first task is:
Algorithm in Brief
Since the paper (above) does a great job explaining the algorithm, I will only describe it very briefly.
For every byte (b) in the image, we look at 4 (or any neighborhood) of bytes immediately to the "left" and at 4 bytes immediately to the "right" of the current byte.
We now have 8 selected bytes, call them c[], where size(c) = 8. Each of the elements of array c is converted into a binary digit, based on the following rule:
Can you write Python code for it?

c = np.random.randint(0, 256, size=8) #neighborhood bytes to the left and to the right contain these values
b = np.random.randint(256) # center of the neighborhood
digits = np.array([0 if d < b else 1 for d in c]) # this is the number that represents the pattern
print "Center: ", b
print "Bytes around it: ", c
print "Pattern: ", digits

# In order to compute the actual pattern, we just mulitply each of the digits by a power of two and add them up
powers = 1 << np.arange(0, 8)
print "As number base 10: ", np.dot(powers, digits.T)


I am working on the file "Learning/1DLBP with CUDA.ipynb".
The first task is:
Improvements
The above results are a bit extreme and very boring. GPU is faster than CPU, however, 4 orders of magnitue?! Can we do better on CPU? Obviously the slowdown is due to Python complexity: array manipulations we are using for once...
Here are things to try:
- Mimic CUDA kernel on CPU:
Can you write Python code for it?

def extract_1dlbp_gpu_debug(input, neighborhood, powers, res):
    maxThread = 512
    blockDim = maxThread
    gridDim = (len(input) - 2 * neighborhood + blockDim) / blockDim

    for block in range(0, gridDim):
        for thread in range(0, blockDim):
            r = 0
            i = blockDim * block + thread
            if i < input.shape[0] - 2 * neighborhood:
                i += neighborhood
                for j in range(i - neighborhood, i):
                    if input[j] >= input[i]:
                        r += powers[j - i + neighborhood]

                for j in range(i + 1, i + neighborhood + 1):
                    if input[j] >= input[i]:
                        r += powers[j - i + neighborhood - 1]

                res[r] += 1
    return res


I am working on the file "Learning/1DLBP with CUDA.ipynb".
The first task is:
Now we modify the test:
Can you write Python code for it?

X = np.arange(3, 7)
X = 10 ** X
neighborhood = 4

cpu_times = np.zeros(X.shape[0])
cpu_times_simple = cpu_times.copy()
cpu_times_jit = cpu_times.copy()
gpu_times = np.zeros(X.shape[0])

p = 1 << np.array(range(0, 2 * neighborhood), dtype='int32')
d_powers = cuda.to_device(p)

for i, x in enumerate(X):
    input = np.random.randint(0, 256, size = x).astype(np.uint8)

    print "Length: {0}".format(x)
    print "--------------"

    start = timer()
    h_cpu = extract_1dlbp_cpu(input, neighborhood, p)
    cpu_times[i] = timer() - start
    print "Finished on CPU: time: {0:3.5f}s".format(cpu_times[i])

    res = np.zeros(1 << (2 * neighborhood), dtype='int32')
    start = timer()
    h_cpu_simple = extract_1dlbp_gpu_debug(input, neighborhood, p, res)
    cpu_times_simple[i] = timer() - start
    print "Finished on CPU (simple): time: {0:3.5f}s".format(cpu_times_simple[i])

    res = np.zeros(1 << (2 * neighborhood), dtype='int32')
    start = timer()
    h_cpu_jit = extract_1dlbp_cpu_jit(input, neighborhood, p, res)
    cpu_times_jit[i] = timer() - start
    print "Finished on CPU (numba: jit): time: {0:3.5f}s".format(cpu_times_jit[i])

    start = timer()
    h_gpu = extract_1dlbp_gpu(input, neighborhood, d_powers)
    gpu_times[i] = timer() - start
    print "Finished on GPU: time: {0:3.5f}s".format(gpu_times[i])
    print "All h_cpu == h_gpu: ", (h_cpu_jit == h_gpu).all() and (h_cpu_simple == h_cpu_jit).all() and (h_cpu == h_cpu_jit).all()
    print ''

f = plt.figure(figsize=(10, 5))

plt.plot(X, cpu_times, label = "CPU")
plt.plot(X, cpu_times_simple, label = "CPU non-vectorized")
plt.plot(X, cpu_times_jit, label = "CPU jit")
plt.plot(X, gpu_times, label = "GPU")
plt.yscale('log')
plt.xscale('log')
plt.xlabel('input length')
plt.ylabel('time, sec')
plt.legend()
plt.show()


I am working on the file "Learning/1DLBP with CUDA.ipynb".
The first task is:
Testing
Running comparison for arrays of several lengths.
Can you write Python code for it?

%matplotlib inline
X = np.arange(3, 7)
X = 10 ** X
neighborhood = 4

cpu_times = np.zeros(X.shape[0])
gpu_times = np.zeros(X.shape[0])

p = 1 << np.array(range(0, 2 * neighborhood), dtype='int32')
d_powers = cuda.to_device(p)

for i, x in enumerate(X):
    input = np.random.randint(0, 256, size = x).astype(np.uint8)
    start = timer()
    h_cpu = extract_1dlbp_cpu(input, neighborhood, p)
    cpu_times[i] = timer() - start
    print "Finished on CPU: length: {0}, time: {1:3.4f}s".format(x, cpu_times[i])

    start = timer()
    h_gpu = extract_1dlbp_gpu(input, neighborhood, d_powers)
    gpu_times[i] = timer() - start
    print "Finished on GPU: length: {0}, time: {1:3.4f}s".format(x, gpu_times[i])
    print "All h_cpu == h_gpu: ", (h_cpu == h_gpu).all()

# Here both axes are log-scaled.
f = plt.figure(figsize=(10, 5))

plt.plot(X, cpu_times, label = "CPU")
plt.plot(X, gpu_times, label = "GPU")
plt.xlabel('input length')
plt.ylabel('time, sec')
plt.yscale('log')
plt.xscale('log')
plt.legend()
plt.show()


I am working on the file "Learning/1DLBP with CUDA.ipynb".
The first task is:
The Kernel
Can you write Python code for it?

@cuda.jit('void(uint8[:], int32, int32[:], int32[:])')
def lbp_kernel(input, neighborhood, powers, h):
    i = cuda.grid(1)
    r = 0
    if i < input.shape[0] - 2 * neighborhood:
        i += neighborhood
        for j in range(i - neighborhood, i):
            if input[j] >= input[i]:
                r += powers[j - i + neighborhood]

        for j in range(i + 1, i + neighborhood + 1):
            if input[j] >= input[i]:
                r += powers[j - i + neighborhood - 1]

        cuda.atomic.add(h, r, 1)


I am working on the file "Learning/1DLBP with CUDA.ipynb".
The first task is:
This looks like a regular CUDA kernel. It almost looksy "C++y". Each thread is tasked with computing the pattern ("digits" above) around one single element of the original input array. We compute the pattern, then wait for all of the threads in the block to finish and update the histogram with their results.
Note: implementation of histogramming using atomics is the most straightforward, but not the most efficient one. For our purposes it will do just fine.
Calling the Kernel
That is pretty standard: allocate memory and move it to the GPU.
Can you write Python code for it?

def extract_1dlbp_gpu(input, neighborhood, d_powers):
    '''
    input - the input array
    neighborhood - size of the neighborhood
    d_powers - device address of the powers of two constants used to compute the final pattern
    '''
    maxThread = 512
    blockDim = maxThread
    d_input = cuda.to_device(input)

    hist = np.zeros(2 ** (2 * neighborhood), dtype='int32')
    gridDim = (len(input) - 2 * neighborhood + blockDim) / blockDim

    d_hist = cuda.to_device(hist)

    lbp_kernel[gridDim, blockDim](d_input, neighborhood, d_powers, d_hist)
    d_hist.to_host()
    return hist


I am working on the file "Learning/1DLBP with CUDA.ipynb".
The first task is:
Which is actually a non-negative integer less than $2^8$.
Then a histogram of these values is built. It has the size of 256 ints and in training experiments proves to be much more effective than a simple byte histogram.
The functions below translate this description into Python:
Can you write Python code for it?

def extract_1dlbp_cpu(input, neighborhood, p):
    """
    Extract the 1d lbp pattern on CPU
    """
    res = np.zeros(1 << (2 * neighborhood))
    for i in range(neighborhood, len(input) - neighborhood):
        left = input[i - neighborhood : i]
        right = input[i + 1 : i + neighborhood + 1]
        both = np.r_[left, right]
        res[np.sum(p [both >= input[i]])] += 1
    return res


I am working on the file "Learning/1DLBP with CUDA.ipynb".
The first task is:
With numba, this can be expressed on the GPU using CUDA.
If you have numbapro, CUDA support can be validated:
Can you write Python code for it?


I am working on the file "Learning/CountingStats1.ipynb".
The first task is:
I am looking to work out how to handle Bayesian estimation in a rate counting system.
Model

Foreground data we want is Binomial with trials 100 and probability 0.5, the number of samples will be changed to see how that impacts the results
Background is taken to be Binomial with trails 1e3 and probability 0.01, number of samples will be the same as foreground
Add a nuisiance parameter, $\alpha$, to represent systematic uncertantity.
Is this U[0,100] or just a number?
Can you write Python code for it?

samples = tb.logspace(1, 10000, 10)

fore = [np.random.binomial(100, 0.50, size=v) for v in samples]
print(tb.logspace(1, 1000, 10))
for v in fore[::-1]:
    h = np.histogram(v, 20)
    plt.hist(v, 20, label='{0}'.format(np.floor(len(v))))
plt.yscale('log')
plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
plt.title('Foreground')

back = [np.random.binomial(1e3, 0.01, size=v) for v in samples]
for v in back[::-1]:
    h = np.histogram(v, 20)
    plt.hist(v, 20, label='{0}'.format(np.floor(len(v))))
plt.yscale('log')
plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
plt.title('Background')

alpha = 10.0 # make the same as the cetner of the Background, no real reason

syn = []
for i in range(len(fore)):
    syn.append(fore[i] + back[i] + alpha)
syn[5]


I am working on the file "Learning/MS Malware Experiments.ipynb".
The first task is:
SKSupervisedLearning wraps the sklearn grid search technique for searching for optimal parameters in one call. You can take a look at the implementation details on GitHub.
Let's plot the confusion matrix to see how well we are doing (values inside squares are %'s):
Can you write Python code for it?

%matplotlib inline
conf_svm = plot_confusion(sl)


I am working on the file "Learning/MS Malware Experiments.ipynb".
The first task is:
An auxilary function
Let's define a function that plots the confusion matrix to see how accurate our predictions really are
Can you write Python code for it?

def plot_confusion(sl):
    conf_mat = confusion_matrix(sl.Y_test, sl.clf.predict(sl.X_test_scaled)).astype(dtype='float')
    norm_conf_mat = conf_mat / conf_mat.sum(axis = 1)[:, None]

    fig = plt.figure()
    plt.clf()
    ax = fig.add_subplot(111)
    ax.set_aspect(1)
    res = ax.imshow(norm_conf_mat, cmap=plt.cm.jet,
                    interpolation='nearest')
    cb = fig.colorbar(res)
    labs = np.unique(Y_test)
    x = labs - 1

    plt.xticks(x, labs)
    plt.yticks(x, labs)

    for i in x:
        for j in x:
            ax.text(i - 0.2, j + 0.2, "{:3.0f}".format(norm_conf_mat[j, i] * 100.))
    return conf_mat


I am working on the file "Learning/MS Malware Experiments.ipynb".
The first task is:
As expected, we are not doing so well in class 5 where there are very few samples.
Train Neural Net`
This is a fun one, I promise. :)
The neural net is built by PyBrain, has just one hidden layer which is equal to $\frac{1}{4}$th of the input layer. The hidden layer activation is sigmoid, the output - softmax (since this is a multi-class neural net), and has bias units for the hidden and the output layers. We use the PyBrain $\color{green}{buildNetwork()}$ function that builds the network in one call.
<p>__NOTE:__ We are still using all the scaled features to train the neural net

On the left - average error, on the right - log loss.

This starts to overfit pretty fast, though.
Can you write Python code for it?

%matplotlib inline
trndata, tstdata = createDataSets(sl.X_train_scaled, Y_train, sl.X_test_scaled, Y_test)
fnn = train(trndata, tstdata, epochs = 10, test_error = 0.07, momentum = 0.15, weight_decay = 0.0001)


I am working on the file "Learning/MS Malware Experiments.ipynb".
The first task is:
As you can see, we are simply wrapping the $\color{green}{RandomForestClassifier}$ in the $\color{green}{CalibratedClassifier}$. Plot the matrix (after a couple of hours):
Can you write Python code for it?

%matplotlib inline
conf_ccrf = plot_confusion(sl_ccrf)


I am working on the file "Learning/MS Malware Experiments.ipynb".
The first task is:
Load data from the text file
Loaded data contains all of the training examples.
<p>__NOTE:__ Actually _almost_ all. 8 are missing, because binary features could not be extracted from them.
Can you write Python code for it?

train_path_mix = "./mix_lbp.csv"
labels_file = "./trainLabels.csv"
X, Y_train, Xt, Y_test = TrainFiles.from_csv(train_path_mix, test_size = 0.1)


I am working on the file "Learning/MS Malware Experiments.ipynb".
The first task is:
Train Random Forest with Calibration
Finally, we train the random forest (which happens to train in seconds) with the calibration classifier (which takes 2 hours or so)
Random forests are very accurate, the problem is that they maker over-confident predictions (or at least that is what the predict_proba function, which is supposed to return probabilities of each class gives us). So, god forbid we are ever wrong! Since it predicts the probability of 0 on the correct class, log loss goes to infinity. Calibration classifier makes predict_proba return something sane.
Can you write Python code for it?

sl_ccrf = SKSupervisedLearning(CalibratedClassifierCV, X, Y_train, Xt, Y_test)
sl_ccrf.train_params = \
    {'base_estimator': RandomForestClassifier(**{'n_estimators' : 7500, 'max_depth' : 200}), 'cv': 10}
sl_ccrf.fit_standard_scaler()
ll_ccrf_trn, ll_ccrf_tst = sl_ccrf.fit_and_validate()

print "Calibrated log loss: ", ll_ccrf_tst


I am working on the file "Learning/SABRS_demo_stats1.ipynb".
The first task is:
Create simulated data that can be used in this proof of concept
Can you write Python code for it?

# observed data

from scipy.signal import savgol_filter
# make a time dependent x
t_x = np.random.uniform(0, 1, 500)
t_x = savgol_filter(t_x, 95, 2)[95//2:-95//2]
t_x -= t_x.min()
t_x /= (t_x.max() - t_x.min())
plt.plot(t_x)
plt.xlabel('time')
plt.ylabel('SWx data value')

a = 6
b = 2
sigma = 2.0
y_obs = a*t_x + b + np.random.normal(0, sigma, len(t_x))
data = pd.DataFrame(np.array([t_x, y_obs]).T, columns=['x', 'y'])

x = t_x

data.plot(x='x', y='y', kind='scatter', s=50)
plt.xlabel('SWx inst')
plt.ylabel('Mission Inst')

# define priors
a = pymc.Normal('slope', mu=0, tau=1.0/10**2)
b = pymc.Normal('intercept', mu=0, tau=1.0/10**2)
tau = pymc.Gamma("tau", alpha=0.1, beta=0.1)

# define likelihood
@pymc.deterministic
def mu(a=a, b=b, x=x):
    return a*x + b

y = pymc.Normal('y', mu=mu, tau=tau, value=y_obs, observed=True)

# inference
m = pymc.Model([a, b, tau, x, y])
mc = pymc.MCMC(m)
# run 6 chains
for i in range(6):
    mc.sample(iter=90000, burn=10000)

# plot up the data and overplot the possible fit lines

data.plot(x='x', y='y', kind='scatter', s=50)

xx = np.linspace(data.x.min(), data.x.max(), 10)
for ii in range(0, len(mc.trace('slope', chain=None)[:]),
                len(mc.trace('slope', chain=None)[:])//400):
    yy = (xx*mc.trace('slope', chain=None)[:][ii] +
          mc.trace('intercept', chain=None)[:][ii])
    plt.plot(xx,yy, c='r')

pymc.Matplot.plot(mc)
pymc.Matplot.summary_plot(mc)


I am working on the file "Learning/SABRS_demo_stats1.ipynb".
The first task is:
Now based on the results above we can use this as a prediction of the Y-data from the X-data into the future
Now that we have time dependent data use the results from the above correlation to predict what the Y-instrument would have seen
Can you write Python code for it?

int_vals = mc.stats()['intercept']['95% HPD interval']
slope_vals = mc.stats()['slope']['95% HPD interval']
print(int_vals, slope_vals)

y_inst =  np.tile(t_x, (2,1)).T * slope_vals + int_vals

plt.plot(y_inst, c='r')
plt.xlabel('time')
plt.ylabel('Mission inst value')
plt.title('Major upper limit of spread')


I am working on the file "Learning/SABRS_demo_stats1.ipynb".
The first task is:
Or do this smarter by sampling the posterior
Can you write Python code for it?

pred = []
for v in t_x:
    pred.append(np.percentile(v * mc.trace('slope',
                                           chain=None)[:] +
                              mc.trace('intercept',
                                       chain=None)[:],
                              [2.5, 97.5]))
plt.plot(pred, c='r')
plt.xlabel('time')
plt.ylabel('Predicted Mission inst value')


I am working on the file "Learning_Curve_and_Other_Visualizations.ipynb".
The first task is:
Contour Plots
First, let's define a new training function which also returns the theta values of every optimization step.
Can you write Python code for it?

def train_with_history(X, y):
    theta = np.random.rand(X.shape[1])
    history = [theta]
    optimal_theta = scipy.optimize.minimize(
        fun=cost_function,
        x0=theta,
        args=(X, y),
        method='bfgs',
        jac=gradient,
        options={'maxiter': 200, 'disp': False},
        callback=lambda x: history.append(x),
    ).x
    history.append(optimal_theta)
    return optimal_theta, np.vstack(history)


I am working on the file "Learning_Curve_and_Other_Visualizations.ipynb".
The first task is:
Define a grid of potentially relevant theta values and compute J for all of them.
Can you write Python code for it?

theta0 = np.linspace(optimal_theta[0] - 5, optimal_theta[0] + 5, 100)
theta1 = np.linspace(optimal_theta[1] - 5, optimal_theta[1] + 5, 100)

J_values = np.zeros(shape=(theta0.size, theta1.size))

for i, theta0_val in enumerate(theta0):
    for j, theta1_val in enumerate(theta1):
        J_values[i, j] = cost_function(np.array([theta0_val, theta1_val]), contour_train_X, train_y)


I am working on the file "Learning_Curve_and_Other_Visualizations.ipynb".
The first task is:
Define functions for Linear Regression (using Gradient Descent).
Can you write Python code for it?

def cost_function(theta, X, y):
    squared_errors = (X.dot(theta) - y) ** 2
    J = 0.5 * squared_errors.sum()
    return J

def gradient(theta, X, y):
    errors = X.dot(theta) - y
    return errors.dot(X)

def train(X, y):
    return scipy.optimize.minimize(
        fun=cost_function,
        x0=np.random.rand(n),
        args=(X, y),
        method='bfgs',
        jac=gradient,
        options={'maxiter': 200, 'disp': False},
    ).x


I am working on the file "Learning_Curve_and_Other_Visualizations.ipynb".
The first task is:
Finally, reward ourselves with some high class contour plots.
Can you write Python code for it?

fig = plt.figure(figsize=(14, 6))

ax = fig.add_subplot(121)
ax.contour(theta0, theta1, J_values, cmap=cm.coolwarm)
ax.scatter(optimal_theta[0], optimal_theta[1])
ax.plot(history[:, 0], history[:, 1], 'k-', marker='o')

ax = fig.add_subplot(122, projection='3d')
ax.contourf(theta0, theta1, J_values, cmap=cm.coolwarm)
ax.scatter(optimal_theta[0], optimal_theta[1])
ax.plot(history[:, 0], history[:, 1], 'k-', marker='o')
ax.view_init(30, 150)


I am working on the file "Learning_Curve_and_Other_Visualizations.ipynb".
The first task is:
In every iteration:
1. Add one fraction of the full train set to the current train set.
2. Train a model.
3. Compute and save its Root Mean Square Error for both the current train set and the test set.
Can you write Python code for it?

rms_train = np.zeros(fractions)
rms_test = np.zeros(fractions)
for i, fraction in enumerate(np.linspace(m / fractions, m, fractions)):
    optimal_theta = train(train_X[:fraction], train_y[:fraction])
    rms_train[i] = np.sqrt(np.mean((train_X[:fraction].dot(optimal_theta) - train_y[:fraction]) ** 2))
    rms_test[i] = np.sqrt(np.mean((test_X.dot(optimal_theta) - test_y) ** 2))


I am working on the file "Learning_Curve_and_Other_Visualizations.ipynb".
The first task is:
Learning Curve
Define the number of equally sized parts the training data should be split up into. This equals the number of models that will be created.
Can you write Python code for it?


I am working on the file "Learning_Cython_video/Chapter05/memview/loops.ipynb".
The first task is:
Application: custom random number generator
Numpy only generates 32-bit random integers.  Let's make a 64-bit random integer generator!
Can you write Python code for it?

%%cython
cimport cython

# This code was translated to Cython from a
# wikipedia article at:
#
# https://en.wikipedia.org/wiki/Xorshift
#
#         "Xorshift random number generators are a
#         class of pseudorandom number generators that
#         was discovered by George Marsaglia.[1] They
#         generate the next number in their sequence
#         by repeatedly taking the exclusive or of a
#         number with a bit shifted version of itself.
#         This makes them extremely fast on modern
#         computer architectures"
#
#         "A naive C implementation of a xorshift+ generator
#         that passes all tests from the BigCrush suite
#         (with an order of magnitude fewer failures than
#         Mersenne Twister or WELL) typically takes fewer
#         than 10 clock cycles on x86 to generate a random
#         number, thanks to instruction pipelining."

cdef unsigned long long s[2]  # Seed: initialize to nonzero
cdef inline unsigned long long xorshiftplus():
    """ Direct translation from Wikipedia page """
    cdef unsigned long long x = s[0]
    cdef unsigned long long y = s[1]
    s[0] = y
    x ^= x << 23  # a
    x ^= x >> 17  # b
    x ^= y^(y>>26) # c
    s[1] = x
    return x + y

@cython.boundscheck(False)
def random_array(unsigned long long[:] output):
    """ Array must be already be sized """
    s[0] = 1  # Set the seed
    s[1] = 2
    cdef int i, n = output.shape[0]
    for i in range(n):
        output[i] = xorshiftplus()


I am working on the file "Learning_Cython_video/Chapter05/memview/loops.ipynb".
The first task is:
Cython: give types to all variables
Can you write Python code for it?

%%cython

def c(int n):
    cdef int i, output
    for i in range(n):
        output += i % 42
    return output

%timeit c(1000000)


I am working on the file "Learning_Cython_video/Chapter05/memview/loops.ipynb".
The first task is:
Plain Python: modulo (%) in a loop
Can you write Python code for it?

def p(n, m):
    output = 0
    for i in range(n):
        output += i % m
    return output

%timeit p(1000000, 42)


I am working on the file "Learning_Cython_video/Chapter05/memview/loops.ipynb".
The first task is:
Quick demo
Can you write Python code for it?

# Create storage for our 8-byte random numbers
import numpy
output = numpy.zeros(10, dtype=numpy.uint64)
random_array(output)
print(output)


I am working on the file "Learning_Cython_video/Chapter05/memview/loops.ipynb".
The first task is:
Speed test - compare Cython and Numpy
Note: Numpy generates only 32-bit integers, but uses the Mersenne Twister algorithm.  Our Cython function generates 64-bit integers, but uses the Xorshift+ algorithm.
Can you write Python code for it?

n = int(1e8)  # 100 million random numbers
output = numpy.zeros(n, dtype=numpy.uint64)
%timeit random_array(output)
%timeit y = numpy.random.randint(low=0, high=2**31 - 1, size=n)


I am working on the file "Learning_Cython_video/Chapter05/memview/loops.ipynb".
The first task is:
Still Python, but inside a Cython cell
There are still no types declared, but you get a surprising speedup.
Can you write Python code for it?

%%cython

def f(n, m):
    output = 0
    for i in range(n):
        output += i % m
    return output

%timeit f(1000000, 42)


I am working on the file "Learning_Cython_video/Chapter05/pynativetypes/pynativetypes.ipynb".
The first task is:
(How to calculate max integers)
Can you write Python code for it?

# Biggest unsigned short, 2 bytes => 16 bits
print(2**16 - 1)
# Biggest unsigned int, 4 bytes => 32 bits
print(2**32 - 1)
# Biggest unsigned long long, 8 bytes => 64 bits
print(2**64 - 1)


I am working on the file "Learning_Cython_video/Chapter05/pynativetypes/pynativetypes.ipynb".
The first task is:
Getting integers from Python (ouch!)
Python can handle very big numbers.  Be careful passing big numbers into Cython functions.
Can you write Python code for it?

%%cython

cpdef can_handle_ushort(unsigned short x):
    return x

cpdef can_handle_uint(unsigned int x):
    return x

cpdef can_handle_ulong_long(unsigned long long x):
    return x

can_handle_ushort(2**16-1)

can_handle_ushort(2**16)

can_handle_uint(2**32-1)

can_handle_uint(2**32)

can_handle_ulong_long(2**64-1)

can_handle_ulong_long(2**64)


I am working on the file "Learning_Cython_video/Chapter05/pynativetypes/pynativetypes.ipynb".
The first task is:
Return integers to Python (no problems)
Can you write Python code for it?

%%cython

def f1():
    # 2 bytes
    cdef short x = 1
    return x

def f2():
    # 4 bytes
    cdef int x = 1
    return x

def f3():
    # 4 or 8 bytes, depends on your OS
    cdef long x = 1
    return x

def f4():
    # 8 bytes
    cdef long long x = 1
    return x

print(type(f1()))
print(type(f2()))
print(type(f3()))
print(type(f4()))


I am working on the file "Learning_Cython_video/Chapter06/opoverload/opoverload.ipynb".
The first task is:
2. Interval Arithmetic (found online)
In interval arithmetic, each number is really an interval with an upper and lower bound.  When calculations with such numbers are performed, we always make sure the answer retains the correct upper and lower bound.
For example, if
$$
x = \left[ x_{min} \;\;, x_{max} \;\; \right]
$$
And similar for $y$, then
$$
x + y = [ x_{min} \;\; + y_{min} \;\;, x_{max} \;\; + y_{max} \;\;]
$$
Can you write Python code for it?

%%cython
cimport cython

@cython.freelist(10)
cdef class Interval:
    cdef double vmin, vmax
    def __init__(self, double vmin, double vmax):
        self.vmin = vmin
        self.vmax = vmax

    def __add__(x, y):
        if not isinstance(x, Interval) or not isinstance(y, Interval):
            raise Exception('Bad Types')

        cdef Interval px = <Interval>x
        cdef Interval py = <Interval>y

        cdef Interval result = Interval(0, 0)
        result.vmin = px.vmin + py.vmin
        result.vmax = px.vmax + py.vmax
        return result

    def __repr__(self):
        return '[{:.6g}, {:.6g}]'.format(self.vmin, self.vmax)

x = Interval(21.0, 22.0)
print(x)

y = Interval(47.0, 48.0)
print(y)

def my_calculation(a, b):
    return a + b

print(my_calculation(x, y))
print(my_calculation(10, 20))


I am working on the file "Learning_Cython_video/Chapter06/opoverload/opoverload.ipynb".
The first task is:
3. Dependency Tracking
It would be very cool to have a bunch of calculations where afterwards you can tell which values depend on which other values.
Can you write Python code for it?

%%cython
import numbers
from weakref import WeakSet  # WeakSet, so intermediates get deleted

cdef class Tracked:
    cdef object __weakref__
    cdef double value
    cdef public str name
    cdef public object depends_on  # This value depends on others
    def __init__(self, double value, name=None):
        self.value = value
        self.name = name
        self.depends_on = WeakSet()

    @staticmethod
    def handle_type(x):
        if isinstance(x, numbers.Number):
            return Tracked(x)
        elif isinstance(x, Tracked):
            return x
        else:
            raise Exception('Bad Types')

    def __mul__(x, y):
        cdef Tracked px = <Tracked>Tracked.handle_type(x)
        cdef Tracked py = <Tracked>Tracked.handle_type(y)

        cdef Tracked result = Tracked(0)

        result.value = px.value * py.value
        result.name = '(calculated)'.format(px.name, py.name)

        result.depends_on.add(px)
        result.depends_on.add(py)
        result.depends_on |= px.depends_on | py.depends_on

        return result

    def __repr__(self):
        depon = [' '*8+x.name for x in self.depends_on]
        s = '{:.6g} name="{}"'.format(self.value, self.name)
        if depon:
            s += '\n    I depend on: \n{}'.format('\n'.join(depon))
        return s

ww = Tracked(0.2, name='wall width, metres, measured 5 May 1995 by CJRH')
print(ww)

wh = Tracked(2.0, name='wall height, metres, measured 12 Jan 1998 by XYZ')
print(wh)

wl = Tracked(5.0, name='wall length, metres, ESTIMATE sometime by ABC')
print(wl)

def volume(width, height, length):
    return width * height * length

print(volume(ww, wh, wl))

def mass(volume, density):
    return volume * density

d = Tracked(2000.0, 'brick (kg/m3), https://en.wikipedia.org/wiki/Brick')
print(
    mass(volume(ww, wh, wl), d)
)


I am working on the file "Learning_Cython_video/Chapter06/opoverload/opoverload.ipynb".
The first task is:
Units of measurement
See https://github.com/cjrh/misu
Can you write Python code for it?

!pip install misu

from misu import *

x = 100*kg
y = 300*lb

z = x + y
print(z)

z >> lb

z + 100*stone

z + 200*feet


I am working on the file "Learning_Cython_video/Chapter07/cpp-support/cpp-support.ipynb".
The first task is:
Auto-converting TO Python objects
Can you write Python code for it?

%%cython
# cython: language_level = 3
# distutils: language = c++

from libcpp.vector cimport vector
from libcpp.list cimport list
from libcpp.map cimport map
from libcpp.set cimport set
from libcpp.pair cimport pair
from libcpp.deque cimport deque
from libcpp.queue cimport queue

def try_vector():
    cdef vector[int] x
    cdef int i, n = 5
    for i in range(n):
        x.push_back(i)
    cdef object o = x
    print(type(o))

def try_list():
    cdef list[int] x
    cdef int i, n = 5
    for i in range(n):
        x.push_back(i)
    cdef object o = x
    print(type(o))

def try_map():
    cdef map[int, int] x
    cdef int i, n = 5
    for i in range(n):
        x.insert([i, i])
    cdef object o = x
    print(type(o))

def try_set():
    cdef set[int] x
    cdef int i, n = 5
    for i in range(n):
        x.insert(i)
    cdef object o = x
    print(type(o))

def try_pair():
    cdef pair[int, int] x
    x.first = 1
    x.second = 2
    cdef object o = x
    print(type(o))

# def try_deque():
#     cdef deque[int] x
#     cdef int i, n = 5
#     for i in range(n):
#         x.push_front(i)
#     cdef object o = x
#     print(type(o))

# def try_queue():
#     cdef queue[int] x
#     cdef int i, n = 5
#     for i in range(n):
#         x.push(i)
#     cdef object o = x
#     print(type(o))

try_vector()
try_list()
try_map()
try_set()
try_pair()


I am working on the file "Learning_Cython_video/Chapter07/cpp-support/cpp-support.ipynb".
The first task is:
Must return a modified dictionary
Can you write Python code for it?

%%cython
# cython: language_level = 3
# distutils: language = c++
from libcpp.map cimport map
from libcpp.pair cimport pair
from libcpp.string cimport string

def erase_debt_but_only_for_me(map[string, double] debtors):
    debtors[b'Caleb'] = 0.0
    return debtors

new_debtors = erase_debt_but_only_for_me(debtors)
print(new_debtors)

print(debtors)


I am working on the file "Learning_Cython_video/Chapter07/pxd-stdlib/pxd-stdlib.ipynb".
The first task is:
10 Secret trigonometry functions you never heard of!
Are you intrigued? Was the heading sufficiently click-baity?  See:
https://en.wikipedia.org/wiki/Versine and this blog
Let's make a Cython library for them!

Versine: versin(θ)=1-cos(θ)
Vercosine: vercosin(θ)=1+cos(θ)
Coversine: coversin(θ)=1-sin(θ)
Covercosine: covercosine(θ)=1+sin(θ)
Haversine: haversin(θ)=versin(θ)/2
Havercosine: havercosin(θ)=vercosin(θ)/2
Hacoversine: hacoversin(θ)=coversin(θ)/2
Hacovercosine: hacovercosin(θ)=covercosin(θ)/2
Exsecant: exsec(θ)=sec(θ)-1
Excosecant: excsc(θ)=csc(θ)-1

Approach 1: Using math from Python
Note: this is a Cython cell, and all variables are typed. This should be very fast
Can you write Python code for it?

%%cython -a
# cython: boundscheck=False

from math import sin, cos

cdef inline double versine(double x):
    return 1.0 - cos(x)

def versine_array_py(double[:] x):
    cdef int i, n = x.shape[0]
    for i in range(n):
        x[i] = versine(x[i])


I am working on the file "Learning_Cython_video/Chapter07/pxd-stdlib/pxd-stdlib.ipynb".
The first task is:
Approach 2: Using math from the C Standard Library
 This code is <u>exactly the same</u> as method1, all that's different is the call to a sin() function.
Can you write Python code for it?

%%cython -a
# cython: boundscheck=False

from libc.math cimport sin, cos

cdef inline double versine(double x):
    return 1.0 - cos(x)

def versine_array_cy(double[:] x):
    cdef int i, n = x.shape[0]
    for i in range(n):
        x[i] = versine(x[i])


I am working on the file "Learning_Cython_video/Chapter07/pxd-stdlib/pxd-stdlib.ipynb".
The first task is:
Roughly 13 X slower than using C math directly
Remember, we used types for everything.  What about just plain Python?
Can you write Python code for it?

from math import cos
def versine_array_pyonly(x):
    for i in range(len(x)):
        x[i] = 1 - cos(x[i])

data = numpy.random.rand(10000)
%timeit versine_array_pyonly(data)


I am working on the file "Learning_Cython_video/Chapter07/pxd-stdlib/pxd-stdlib.ipynb".
The first task is:
Speed test
Can you write Python code for it?

import numpy

data = numpy.random.rand(10000)
%timeit versine_array_py(data)

data = numpy.random.rand(10000)
%timeit versine_array_cy(data)


I am working on the file "Learning_Cython_video/Chapter09/openmp/openmp.ipynb".
The first task is:

Compare range() and prange()
Can you write Python code for it?

%%cython -f
# distutils: extra_compile_args = -fopenmp
# distutils: extra_link_args = -fopenmp
# cython: boundscheck = False
from libc.math cimport log
from cython.parallel cimport prange

def f1(double[:] x, double[:] out):
    cdef int i, n = x.shape[0]
    for i in range(n):
        out[i] = log(x[i])

def f2(double[:] x, double[:] out):
    cdef int i, n = x.shape[0]
    for i in prange(n, nogil=True):
        out[i] = log(x[i])


I am working on the file "Learning_Cython_video/Chapter09/openmp/openmp.ipynb".
The first task is:
Confusing explanation from the Cython docs:

If you assign to a variable in a prange block, it becomes lastprivate, meaning that the variable will contain the value from the last iteration. <u>If you use an inplace operator on a variable, it becomes a reduction</u>, meaning that the values from the thread-local copies of the variable will be reduced with the operator and assigned to the original variable after the loop. The index variable is always lastprivate. Variables assigned to in a parallel with block will be private and unusable after the block, as there is no concept of a sequentially last value.

Solution: need a temporary variable
Can you write Python code for it?

%%cython
# distutils: extra_compile_args = -fopenmp
# distutils: extra_link_args = -fopenmp
# cython: boundscheck = False
from libc.math cimport log
from cython.parallel cimport prange

def g_single(double[:] x):
    cdef int i, n = x.shape[0]
    cdef double result = 0
    for i in range(n):
        if x[i] > 0.5:
            result += log(x[i])
        else:
            result += 1.0
    return result

def g_parallel(double[:] x):
    cdef int i, n = x.shape[0]
    cdef double tmp, result = 0
    for i in prange(n, nogil=True):
        if x[i] > 0.5:
            tmp = log(x[i])
        else:
            tmp = 1.0
        result += tmp
    return result

print(g_single(data[:3]))
print(g_parallel(data[:3]))

%timeit g_single(data)
%timeit g_parallel(data)


I am working on the file "Learning_Cython_video/Chapter09/openmp/openmp.ipynb".
The first task is:
Make some data
Can you write Python code for it?

data = np.random.rand(10000000)
out = np.zeros_like(data)


I am working on the file "Learning_Cython_video/Chapter09/openmp/openmp.ipynb".
The first task is:
The gcc command maps back to clang. The "real" GCC is different:
Let's find all executables beginning with "gcc":
Can you write Python code for it?


I am working on the file "Learning_Cython_video/Chapter11/dates/dateobject-withC.ipynb".
The first task is:
Case Study: Slow Pandas dates
Batches of data are collected from field instruments. These instruments capture the date in three separate columns: day, month and year.
Data is processed in Pandas, but currently it is <u>slow to convert the three columns into datetimes</u>.
Example (randomised) data
Can you write Python code for it?

import numpy as np
import pandas as pd

def make_sample_data(size):
    d = dict(
        # Years: 1980 - 2015
        year=np.random.randint(1980, 2016, int(size)),
        # Months 1 - 12
        month=np.random.randint(1, 13, int(size)),
        # Day number: 1 - 28
        day=np.random.randint(1, 28, int(size)),
        )
    return pd.DataFrame(d)


I am working on the file "Learning_Cython_video/Chapter11/dates/dateobject-withC.ipynb".
The first task is:
Goal: make single datetime column
Let's see the Python code first:
Can you write Python code for it?

import datetime

def create_datetime_py(year, month, day):
    """ Take year, month, day and return a datetime """
    return datetime.datetime(year, month, day, 0, 0, 0, 0, None)


I am working on the file "Learning_Cython_video/Chapter11/dates/dateobject-withC.ipynb".
The first task is:
Note: the type is "datetime64[ns]".
Awkward to type that all out each time.  Let's make a convenient function.
Can you write Python code for it?

def make_datetime_py(df):
    return df.apply(lambda x : create_datetime_py(
        x['year'], x['month'], x['day']), axis=1)


I am working on the file "Learning_Cython_video/Chapter11/dates/dateobject-withC.ipynb".
The first task is:
Problem: this is slow
With lots of data, the conversion to a datetime column takes a very long time!  Let's try a bunch of data:
Can you write Python code for it?

df_big = make_sample_data(100000)

%timeit make_datetime_py(df_big)


I am working on the file "Learning_Cython_video/Chapter11/dates/dateobject-withC.ipynb".
The first task is:
Speed Test
Can you write Python code for it?

df_big = make_sample_data(100000)

%timeit make_datetime_py(df_big)
%timeit make_datetime_cy(df_big, convert_arrays_ts)


df_big = make_sample_data(100000)

%timeit make_datetime_py(df_big)
%timeit make_datetime_cy(df_big, convert_arrays_ts)
%timeit make_datetime_cy(df_big, convert_arrays_dt)


I am working on the file "Learning_Cython_video/Chapter11/dates/dateobject-withC.ipynb".
The first task is:
Test it out
Can you write Python code for it?


I am working on the file "Learning_Cython_video/Chapter11/dates/dateobject-withC.ipynb".
The first task is:
Then we can just call it like so:
Can you write Python code for it?


I am working on the file "Learning_Cython_video/Chapter11/dates/dateobject-withC.ipynb".
The first task is:
Use the Python conversion function
Pandas has an apply() method that runs your function on a bunch of columns.
You must provide a function that receives a row, and your function must return a value.  All the output values get put into a new Pandas series.
Can you write Python code for it?

# Refer to fields by name! Very cool 👍
df.apply(lambda x : create_datetime_py(
        x['year'], x['month'], x['day']), axis=1)


I am working on the file "Learning_Cython_video/Chapter11/dates/dateobject-withC.ipynb".
The first task is:
Utility function for applying our conversion
Can you write Python code for it?

def make_datetime_cy(df, method):
    s = pd.Series(np.zeros(len(df), dtype='datetime64[ns]'))
    method(df['year'].values, df['month'].values, df['day'].values,
               s.values.view('int64'))
    return s

# Test it out
make_datetime_cy(df, convert_arrays_ts)


I am working on the file "Learning_Cython_video/Chapter11/dates/dateobject-withC.ipynb".
The first task is:
What to do?
The first thing is to check whether there is a low-level PXD interface file for the Python datetime object.

Let's use Cython!
Can you write Python code for it?

%%cython
# cython: boundscheck = False
# cython: wraparound = False
from cpython.datetime cimport (
    import_datetime, datetime_new, datetime, timedelta)
from pandas import Timestamp

import_datetime()

cpdef convert_arrays_ts(
        long[:] year, long[:] month, long[:] day,
        long long[:] out):
    """ Result goes into `out`  """
    cdef int i, n = year.shape[0]
    cdef datetime dt
    for i in range(n):
        dt = <datetime>datetime_new(
                year[i], month[i], day[i], 0, 0, 0, 0, None)
        out[i] = Timestamp(dt).value


I am working on the file "Learning_Cython_video/Chapter11/dates/dateobject-withC.ipynb".
The first task is:
XX / XX
Using C standard library
Can you write Python code for it?

%%cython -a
# cython: boundscheck = False
# cython: wraparound = False
from libc.time cimport mktime, tm, timezone

cdef inline long to_unix(long year, long month, long day):
    """ month: 1 - 12, day: 1 - 31
        Result is in UTC. """
    cdef tm tms
    tms.tm_year = year - 1900  # years since 1900 !!
    tms.tm_mon = month - 1     # 0 to 11 !!
    tms.tm_mday = day          # 1 - 31
    tms.tm_hour, tms.tm_min, tms.tm_sec  = 0, 0, 0
    return mktime(&tms) - timezone

cpdef convert_arrays_libc(
        long[:] year, long[:] month, long[:] day,
        long long[:] out):
    """ Result goes into `out`  """
    cdef int i, n = year.shape[0]
    cdef long unix
    for i in range(n):
        unix = to_unix(year[i], month[i], day[i])
        #print(unix)
        #out[i] = to_unix(year[i], month[i], day[i]) * 1000000000
        out[i] = unix * 1000000000

make_datetime_cy(df, convert_arrays_libc)

df_big = make_sample_data(100000)

%timeit make_datetime_py(df_big)
%timeit make_datetime_cy(df_big, convert_arrays_dt)
%timeit make_datetime_cy(df_big, convert_arrays_ts)
%timeit make_datetime_cy(df_big, convert_arrays_libc)


I am working on the file "Learning_Cython_video/Chapter11/dates/dateobject.ipynb".
The first task is:
Case Study: Slow Pandas dates
Batches of data are collected from field instruments. These instruments capture the date in three separate columns: day, month and year.
Data is processed in Pandas, but currently it is <u>slow to convert the three columns into datetimes</u>.
Example (randomised) data
Can you write Python code for it?


I am working on the file "Learning_Cython_video/Chapter11/dates/dateobject.ipynb".
The first task is:
Note: the type is "datetime64[ns]".
Awkward to type that all out each time.  Let's make a convenient function.
Can you write Python code for it?


I am working on the file "Learning_Cython_video/Chapter11/dates/dateobject.ipynb".
The first task is:
Problem: this is slow
With lots of data, the conversion to a datetime column takes a very long time!  Let's try a bunch of data:
Can you write Python code for it?


I am working on the file "Learning_Cython_video/Chapter11/dates/dateobject.ipynb".
The first task is:
Speed Test
Can you write Python code for it?


I am working on the file "Learning_Cython_video/Chapter11/dates/dateobject.ipynb".
The first task is:
Start with a small dataset
Can you write Python code for it?


I am working on the file "Learning_Cython_video/Chapter11/dates/dateobject.ipynb".
The first task is:
Test it out
Can you write Python code for it?


I am working on the file "Learning_Cython_video/Chapter11/dates/dateobject.ipynb".
The first task is:
Then we can just call it like so:
Can you write Python code for it?


I am working on the file "Learning_Cython_video/Chapter11/dates/dateobject.ipynb".
The first task is:
Use the Python conversion function
Pandas has an apply() method that runs your function on a bunch of columns.
You must provide a function that receives a row, and your function must return a value.  All the output values get put into a new Pandas series.
Can you write Python code for it?


I am working on the file "Learning_Cython_video/Chapter11/dates/dateobject.ipynb".
The first task is:
Utility function for applying our conversion
Can you write Python code for it?


I am working on the file "LeastSqSol.ipynb".
The first task is:
Atat eroarea medie, cat si figura de mai sus ilustreaza ca modelul patratic (parabola celor mai mici patrate) este mai adecvat pentru datele considerate.
In semestrul doi metoda cel mai mici patrate va fi rafinata prin analiza  regresiei.
In secventele de cod de mai sus am urmat metoda manuala de calcul a dreptei, respectiv parabolei celor mai mici patrate,
adica am constituit matricea sistemului supradeterminat asociat si am apelat functia lstsq, pentru a ilustra
legatura cu solutia celor mai mici patrate.
Setului de date
$(x_i, y_i)$, $i=\overline{0,n-1}$, $n>k$,  i se poate asocia polinomul de grad $k$ al celor mai mici patrate,
$y=p[0]x^k+p[1]x^{k-1}+\cdots+p[k-1]x+p[k]$, apeland direct functia np.polyfit(x,y,k).
Array-ul x, ca argument al functiei, contine abscisele punctelor $(x_i, y_i)$, iar y, ordonatele lor.
Functia returneaza coeficientii polinomului si optional reziduul si alte informatii:
    numpy.polyfit
Can you write Python code for it?

p=np.polyfit(x,y,2)
print p

from IPython.core.display import HTML
def  css_styling():
    styles = open("./custom.css", "r").read()
    return HTML(styles)
css_styling()


I am working on the file "LeastSqSol.ipynb".
The first task is:
Daca $m=n$ matricea $A$ este patratica. Daca $A$ este  nesingulara, atunci solutia celor mai mici patrate este unica solutie pe care sistemul $Ax=b$ o admite:
Can you write Python code for it?

A=np.array([[1,0,-1], [2,3,-5], [4,-2, 1]],float)
b=np.array([3, -1, 7], float)
[xstar, rez2, rang, sing]=np.linalg.lstsq(A,b)

print 'Rangul matricii A este:', rang
print 'Solutia celor mai mici patrate este:', xstar
print 'Norma reziduului la patrat:', rez2


I am working on the file "LeastSqSol.ipynb".
The first task is:
Din punct de vedere algebric dreapta celor mai mici patrate exista si este unica pentru orice set de date
$(x_i, y_i)$, $i=\overline{1,n}$, $n>2$, care nu apartin unei drepte verticale $y=c$. Nu intotdeauna insa dreapta celor mai mici patrate este modelul adecvat pentru date, asa cum se vede din exemplul urmator:
Can you write Python code for it?

x=[ 0.63, 1.19, 1.44, 2.13,  2.88, 3, 3.46, 3.66, 3.71, 4.19,  4.29, 4.46, 4.67,  4.83]
y=[ 3.63, 1.42, 0.81, 0.43, -1.02, -0.22, 0.13, 0.93, 1.48,  2.97,  2.52, 3.41, 4.54,  5.07]
x=np.array(x)
y=np.array(y)
A=np.vstack((x, np.ones(x.size))).transpose()
plt.plot(x,y,  color="green", lw=2, ls='*', marker='o')
[[a, b], Er]=np.linalg.lstsq(A,y)[:2]
xx=[0, 5]
yy=[]
for elem in xx:
    yy.append(a*elem+b)
plt.plot(xx, yy, 'r')
print 'Dreapta celor mai mici patrate are parametrii a=', a, 'b=', b
print 'Eroarea la patrat a celor mai mici patrate este', Er
print 'Media erorii la patrat', Er/len(x)


I am working on the file "LeastSqSol.ipynb".
The first task is:
Forma scara redusa a matricii prelungite, $\overline{M}=[M|c]$, a sistemului $Mx=c$ este:
$$S_{\overline{M}}^0=\left[\begin{array}{rrrrr} 1&0&0&-1/3& -1\0&1&0&-1/9&-1\0&0&1&-3&1\0&0&0&0&0\end{array}\right]$$
Prin urmare sistemul $Mx=c$ este compatibil nedeterminat si daca rezolvam sistemul echivalent definit de  forma scara redusa,  de mai sus, avem solutiile:
$$\begin{array}{ll}\begin{array}{lll}
x_1^&=& \displaystyle\frac{x_4^}{3}-1\
x_2^&=&\displaystyle\frac{x_4^}{9}-1\
x_3^&=&3x_4^+1\end{array}\quad x_4^*\in\mathbb{R}\end{array}$$
Astfel, pentru fiecare alegere particulara a lui $x_4^$ (necunoscuta secundara a sistemului) obtinem o alta solutie, $x^=(x_4^/3-1, x_4^/9-1, 3x_4^+1, x_4^)^T$, a celor mai mici patrate. Toate aceste solutii au particularitatea ca $Ax^=b'$, unde $b'$ este proiectia ortogonala a lui $b$ pe subspatiul coloanelor, col(A). $b'$ este unic, in timp ce $x^$ nu este unic. Cu alte cuvinte aplicatia liniara $L:\mathbb{R}^4\to col(A)$, $L(x)=Ax$ nu este injectiva, pentru ca exista o infinitate de vectori $x^$ care sunt aplicati in acelasi vector $b'\in col(A)$.
Solutia numerica a celor mai mici patrate pentru sistemul $Ax=b$, unde $A$ si $b$ sunt matricile de mai sus este:
Can you write Python code for it?

[xstar, rez2, rang, sing]=np.linalg.lstsq(A,b)

print 'Rangul matricii A este:', rang
print 'Solutia celor mai mici patrate este:', xstar
print 'Norma reziduului la patrat:', rez2


I am working on the file "LeastSqSol.ipynb".
The first task is:
Matricea sistemului  de ecuatii  $ax_i+b=y_i$, $i=\overline{1,8}$, o constituim apeland functia
np.vstack (tuple)
care genereaza  un array avand drept linii, array-urile 1D din tuple. Prin transpunerea array-ului generat avem matricea sistemului:
Can you write Python code for it?

A=np.vstack((x, np.ones(x.size))).transpose()
print A
[[a, b], Er]=np.linalg.lstsq(A,y)[:2]# cerem returnarea primelor doua obiecte din lista globala
print 'dreapta celor mai mici patrate are parametrii a=', a, 'b=', b
print 'Media erorilor la patrat  este', Er/len(x)
print 'Media la bacalaureat predictionata de model pentru anul 2014 este', a*2014+b

xx=[1965, 2015]
# dreapta este perfect determinata de doua puncte (1965, a*1965+b), (2015, a*2015+b)
yy=[]
for an in xx:
    yy.append(a*an+b)
plt.plot(xx, yy, 'r')# este trasata dreapta ce uneste punctele alese
plt.plot(x,y, 'go')# marcam datele din nou pentru a vedea pozitia fata de dreapta model
plt.plot(2014, a*2014+b, 'ro')# marcam pe dreapta punctul de coordonate (2014,a*2014+b)


I am working on the file "LeastSqSol.ipynb".
The first task is:
Remarcam ca din infinitatea de solutii analitice, numeric este calculata una singura, corespunzatoare lui
$x_4^=xstar[3]$  (Atentie!  indexarea coordonatelor lui $x^$ este tipica pentru matematica, iar ale lui xstar
                   este cea din Python).
Daca incercam sa transmitem ca argument $A$ al functiei np.linalg.lstsq(A,b) o matrice avand numarul de linii mai mic decat numarul de coloane,
este afisat un mesaj de eroare, pentru ca solutia celor mai mici patrate exista doar pentru sistemele $Ax=b$, unde numarul liniilor lui $A$ este mai mare sau
egal cu numarul coloanelor:
Can you write Python code for it?

A=np.array([[2, 3, -4], [1,5,-2]], float)
b=np.array([-1,3,7], float)
[xstar, rez2, rang, sing]=np.linalg.lstsq(A,b)


I am working on the file "LeastSqSol.ipynb".
The first task is:
Rezolvand uzual sistemul normal $A^TAx=A^Tb$ obtinem solutia celor mai mici patrate ca mai sus:
Can you write Python code for it?

M=np.dot(A.transpose(), A)#
c=np.dot(A.transpose(), b)# Mx=c -->  x=inv(M)c
xx=np.dot(np.linalg.inv(M), c)
print xx


I am working on the file "LeastSqSol.ipynb".
The first task is:
Sa ilustram solutiile analitice ale sistemului $A^TAx=A^Tb$, in cazul in care rangul matricii $A$,
deci si al matricii $A^TA$ este mai mic decat $n$.
Apoi invocam functia np.linalg.lstsq(A,b) si identificam printre solutiile analitice, pe cea numerica returnata de functia np.linalg.leastsq.
Pentru aceasta, construim dintr-o lista o matrice din $\mathbb{R}^{6\times 4}$:
Can you write Python code for it?

L=2*[1, -3, 0, 0, -3,0,0,1, 0,0,1,-3]
A=np.asarray(L, float).reshape((6,4))#convertim lista la un array pe care il  redimensionam
b=np.array([2,8,-3,2,-2,5], float) # vectorul termenilor liberi ai sistemului Ax=b
print A
M=np.dot(A.transpose(), A)#M=A^TA
c=np.dot(A.transpose(), b)# c=A^T b
print M
print c


I am working on the file "LeastSqSol.ipynb".
The first task is:
Solutia celor mai mici patrate a sistemului  Ax=b si caracteristici ale solutiei si problemei celor mai mici patrate se obtin apeland functia numpy.linalg.lstsq:
[xstar, rez2, rangA, singA]=np.linalg.lstsq(A,b)
Functia np.linalg.lstsq returneaza o lista  de obiecte:

xstar este array-ul care contine solutia, $x^*$;

rez2 este norma la patrat a reziduului: $\mbox{rez2}=\|b-Ax^*\|^2$

rangA este rangul matricii A

singA sunt valorile singulare ale lui A (se vor studia in Cursul 12)

Solutia celor mai mici patrate este solutie a sistemului normal $A^TAx=A^Tb$.

Deoarece rang($A^TA$)=rang($A$) (vezi Cursul 7), rezulta ca daca coloanele $c_1, c_2, \ldots, c_n$ ale matricii $A$ sunt liniar independente,
 atunci $A$ are rangul $n$, iar sistemul normal are o unica solutie,
$x^*$.

Daca rangul lui $A$ este mai mic decat $n$ sistemul normal este compatibil nedeterminat si admite o infinitate de solutii $x^*$, astfel incat pentru fiecare dintre ele eroarea de aproximare este minima:

$||b-Ax^*||^2\leq ||b-Ax||^2$, $\forall x\in\mathbb{R}^n$.
Dam cateva exemple de calcul a solutiei celor mai mici patrate:
Can you write Python code for it?

import numpy as np
A=np.array([[1,2], [-3,1], [5,-1.2], [-2.14, -3.4]], float)
b=np.array([-5, 2,1,3], float)
[xstar, rez2, rang, sing]=np.linalg.lstsq(A,b)
print A.shape
print 'Rangul matricii A este:', rang
print 'Solutia celor mai mici patrate este:', xstar
print 'Norma reziduului la patrat:', rez2


I am working on the file "Lecture 11/.ipynb_checkpoints/Satoshi Nakamoto Lecture 10 Jupyter Notebook-checkpoint.ipynb".
The first task is:
Import non-standard libraries (install as needed)
Can you write Python code for it?

from osgeo import ogr,osr
import folium
import simplekml


I am working on the file "Lecture 11/.ipynb_checkpoints/Satoshi Nakamoto Lecture 10 Jupyter Notebook-checkpoint.ipynb".
The first task is:
Is the ESRI Shapefile driver available?
Can you write Python code for it?

driverName = "ESRI Shapefile"
drv = ogr.GetDriverByName( driverName )
if drv is None:
    print "%s driver not available.\n" % driverName
else:
    print  "%s driver IS available.\n" % driverName


I am working on the file "Lecture 11/.ipynb_checkpoints/Satoshi Nakamoto Lecture 10 Jupyter Notebook-checkpoint.ipynb".
The first task is:
Optional directory creation
Can you write Python code for it?

if not exists('./data'):
    makedirs('./data')

#chdir("./data")


I am working on the file "Lecture 11/.ipynb_checkpoints/Satoshi Nakamoto Lecture 11 Jupyter Notebook-checkpoint.ipynb".
The first task is:
Define a function which will create a shapefile from the points input and export it as kml if the option is set to True.
Can you write Python code for it?

def shpFromPoints(filename, layername, points, save_kml = True):
    spatialReference = osr.SpatialReference()
    spatialReference.ImportFromProj4('+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs')
    ds = drv.CreateDataSource(filename)
    layer=ds.CreateLayer(layername, spatialReference, ogr.wkbPoint)
    layerDefinition = layer.GetLayerDefn()

    point = ogr.Geometry(ogr.wkbPoint)
    feature = ogr.Feature(layerDefinition)

    kml = simplekml.Kml()
    for i, value in enumerate(points):
        point.SetPoint(0,value[0], value[1])
        feature.SetGeometry(point)
        layer.CreateFeature(feature)
        kml.newpoint(name=str(i), coords = [(value[0],value[1])])
    ds.Destroy()
    if save_kml == True:
        kml.save("my_points2.kml")


I am working on the file "Lecture 11/.ipynb_checkpoints/Satoshi Nakamoto Lecture 11 Jupyter Notebook-checkpoint.ipynb".
The first task is:
Is the ESRI Shapefile driver available?
Can you write Python code for it?


I am working on the file "Lecture 11/Satoshi Nakamoto Lecture 11 Jupyter Notebook.ipynb".
The first task is:
Call the function specifying the list of points, the output map name and its zoom level. If not False, the map is saved as an html
Can you write Python code for it?

mapFromPoints(pts, "SatoshiNakamotoMap", zoom_level = 15)


I am working on the file "Lecture 11/Satoshi Nakamoto Lecture 11 Jupyter Notebook.ipynb".
The first task is:
Define a function which will create a shapefile from the points input and export it as kml if the option is set to True.
Can you write Python code for it?


I am working on the file "Lecture 11/Satoshi Nakamoto Lecture 11 Jupyter Notebook.ipynb".
The first task is:
Define the file and layer name as well as the points to be mapped.
Can you write Python code for it?

filename = "wageningenpoints.shp"
layername = "wagpoints"
pts = [(5.665777,51.987398),
        (5.663133,51.978434)]
shpFromPoints(filename, layername, pts)


I am working on the file "Lecture 11/Satoshi Nakamoto Lecture 11 Jupyter Notebook.ipynb".
The first task is:
Import non-standard libraries (install as needed)
Can you write Python code for it?


I am working on the file "Lecture 11/Satoshi Nakamoto Lecture 11 Jupyter Notebook.ipynb".
The first task is:
Is the ESRI Shapefile driver available?
Can you write Python code for it?


I am working on the file "Lecture 11/Satoshi Nakamoto Lecture 11 Jupyter Notebook.ipynb".
The first task is:
Optional directory creation
Can you write Python code for it?

if not exists('./data'):
    makedirs('./data')

chdir("./data")


I am working on the file "Lecture Notebooks/Getting Started.ipynb".
The first task is:
Filtering data
Filtering parts of the data in pandas is really easy.
If you want to filter data for editing it then you need to make a copy of the filtered data.
Can you write Python code for it?

print "Show data containing with petalwidth > 2.0"
df_iris[df_iris["petalwidth"] > 2.0]


I am working on the file "Lecture Notebooks/Getting Started.ipynb".
The first task is:
Lists
A list is a data structure which can hold a list of items of different types. Think of a shopping list. Items in the list can be accessed using zero based index. You will use these when you want to add more data and access the data based on the position in index.
Can you write Python code for it?

list_items = ["milk", "cereal", "banana", 22.5, [1,2,3]]  ## A list can contain another list and items of different types
print list_items
print "3rd item in the list: ", list_items[2] # Zero based index starts from 0 so 3rd item will have index 2


I am working on the file "Lecture Notebooks/Getting Started.ipynb".
The first task is:
Pandas
Pandas is a wonderful library for working with tabular data in python. It can read csv files easily and represents them as dataframes. Think of it like excel but faster and without a GUI.
Can you write Python code for it?

import pandas as pd

df_iris = pd.DataFrame(data, columns=meta.names())
df_iris.head()

print "The shape of iris data is: ", df_iris.shape

print "Show how many instances are of each class: "
df_iris["class"].value_counts()

df_iris["sepallength"].hist(bins=10)


I am working on the file "Lecture Notebooks/Getting Started.ipynb".
The first task is:
Plotting data
Great for visual inspection.
Matplotlib and Seaborn
Matplotlib is a low level python library which gives you complete control over your plots.
Seaborn is a library made on top of matplotlib and which adds functionality to create certain types of plots easily. Works great with pandas.
Can you write Python code for it?

# We need the line below to show plots directly in the notebook.
%matplotlib inline

import matplotlib.pyplot as plt
import seaborn as sns

sns.set_style("ticks")
sns.set_context("paper")

colors = {
    "Iris-setosa": "red",
    "Iris-versicolor": "green",
    "Iris-virginica": "blue",
}
plt.scatter(df_iris.petallength, df_iris.petalwidth, c=map(lambda x: colors[x], df_iris["class"]))
plt.xlabel("petallength")
plt.ylabel("petalwidth")

sns.lmplot(x="petallength", y="petalwidth", hue="class", data=df_iris, fit_reg=False)

sns.pairplot(df_iris, hue="class")

sns.countplot(x="sex", data=df)

sns.countplot(x="class", data=df)

sns.countplot(x="embark_town", data=df)

sns.countplot(x="alive", data=df)

sns.countplot(x="alone", data=df)

sns.lmplot(x="age", y="survived", hue="sex", data=df, fit_reg=True, logistic=True)

sns.barplot(x="sex", y="survived", hue="embark_town", data=df)

sns.barplot(x="sex", y="survived", hue="class", data=df)

sns.barplot(x="sex", y="survived", hue=pd.cut(df.age, bins=[0,18,30,100]), data=df)

sns.barplot(x="sex", y="survived", hue="alone", data=df)

sns.barplot(x="sex", y="survived", hue=pd.cut(df.sibsp, bins=[0,1,2,3,10]), data=df)

sns.barplot(x="sex", y="survived", hue=pd.cut(df.parch, bins=[0,1,2,3,10]), data=df)

sns.barplot(x="sex", y="age", hue=pd.cut(df.parch, bins=[0,1,2,3,10]), data=df)

sns.barplot(x="sex", y="age", hue=pd.cut(df.sibsp, bins=[0,1,2,3,10]), data=df)

sns.barplot(x="sex", y="age", hue="embark_town", data=df)

sns.barplot(x="sex", y="age", hue="class", data=df)


I am working on the file "Lecture Notebooks/Getting Started.ipynb".
The first task is:
Question: Draw the plot of mean petalwidth of the various categories of Iris-classes. It should show the mean petalwidth for each petallengths in buckets [0, 2.5, 4.5, 6.5, 10]
ANSWER BELOW
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Can you write Python code for it?

sns.barplot(x="class", y="petalwidth", hue=pd.cut(df_iris.petallength, bins=[0, 2.5, 4.5, 6.5, 10]), data=df_iris)


I am working on the file "Lecture Notebooks/Getting Started.ipynb".
The first task is:
Sets
Like list but only store unique items which are hashable (think basic data types like string, ints and not lists, will explain later). Super useful for checking if an item is already in the list. Items are not indexed. So items can only be added or removed. You will use these when you want to keep track of unique items e.g. feature names in the data.
Can you write Python code for it?

set_items = set([1,2,3, 1])
print set_items
print "Is 1 in set_items: ", 1 in set_items
print "Is 10 in set_items: ", 10 in set_items


I am working on the file "Lecture Notebooks/Getting Started.ipynb".
The first task is:
Titanic data
```
VARIABLE DESCRIPTIONS:
survival        Survival
                (0 = No; 1 = Yes)
pclass          Passenger Class
                (1 = 1st; 2 = 2nd; 3 = 3rd)
name            Name
sex             Sex
age             Age
sibsp           Number of Siblings/Spouses Aboard
parch           Number of Parents/Children Aboard
ticket          Ticket Number
fare            Passenger Fare
cabin           Cabin
embarked        Port of Embarkation
                (C = Cherbourg; Q = Queenstown; S = Southampton)
SPECIAL NOTES:
Pclass is a proxy for socio-economic status (SES)
 1st ~ Upper; 2nd ~ Middle; 3rd ~ Lower
Age is in Years; Fractional if Age less than One (1)
 If the Age is Estimated, it is in the form xx.5
With respect to the family relation variables (i.e. sibsp and parch)
some relations were ignored.  The following are the definitions used
for sibsp and parch.
Sibling:  Brother, Sister, Stepbrother, or Stepsister of Passenger Aboard Titanic
Spouse:   Husband or Wife of Passenger Aboard Titanic (Mistresses and Fiances Ignored)
Parent:   Mother or Father of Passenger Aboard Titanic
Child:    Son, Daughter, Stepson, or Stepdaughter of Passenger Aboard Titanic
Other family relatives excluded from this study include cousins,
nephews/nieces, aunts/uncles, and in-laws.  Some children travelled
only with a nanny, therefore parch=0 for them.  As well, some
travelled with very close friends or neighbors in a village, however,
the definitions do not support such relations.
```
Can you write Python code for it?

df = pd.read_csv("../data/titanic.csv")
df.shape

df.head()


I am working on the file "Lecture Notebooks/Redoing Weka stuff.ipynb".
The first task is:
Creating many features at once using patsy
Can you write Python code for it?

df_t = df_t.rename(columns={"class": "label"})
df_t.head()

y, X = patsy.dmatrices("label ~ petalwidth + petallength:petalwidth + I(sepallength**2)-1", data=df_t, return_type="dataframe")
print y.shape, X.shape

y.head()

X.head()

model = sm.MNLogit(y, X)
res = model.fit()
res.summary()

model_sk = linear_model.LogisticRegression(multi_class="multinomial", solver="lbfgs")
model_sk.fit(X, df_t["label"])

y_pred = model_sk.predict(X)

y_pred[:10]

print metrics.classification_report(df_t["label"], y_pred)

model_sk_t = tree.DecisionTreeClassifier()

model_sk_t.fit(X, df_t["label"])

show_decision_tree(model_sk_t, X, df_t["label"])

model_0r = dummy.DummyClassifier(strategy="most_frequent")
model_0r.fit(X, df_t["label"])
y_pred = model_0r.predict(X)
print metrics.classification_report(df_t["label"], y_pred)

cm = metrics.confusion_matrix(y_true=df_t["label"], y_pred=y_pred)

df_cm = pd.DataFrame(cm, columns=model_0r.classes_, index=model_0r.classes_)

df_cm

_ = get_confusion_matrix(model_0r, X, df_t["label"])

_ = get_confusion_matrix(model_sk_t, X, df_t["label"])

_ = get_confusion_matrix(model_sk, X, df_t["label"])


I am working on the file "Lecture Notebooks/Redoing Weka stuff.ipynb".
The first task is:
Decision surface of Naive Bayes classifier will not have overlapping colors because of the basic code I am using to show decision boundaries. A better code can show the mixing of colors properly
Can you write Python code for it?

plot_decision_regions(clf, X, y, col_x="petalwidth", col_y="petallength")


I am working on the file "Lecture Notebooks/Redoing Weka stuff.ipynb".
The first task is:
IBk of K-nearest neighbors classifier
Can you write Python code for it?

clf = neighbors.KNeighborsClassifier(n_neighbors=1)
clf.fit(X, y)
_ = get_confusion_matrix(clf, X, y)

plot_decision_regions(clf, X, y, col_x="petalwidth", col_y="petallength")


I am working on the file "Lecture Notebooks/Redoing Weka stuff.ipynb".
The first task is:
Logistic regression
Can you write Python code for it?

clf = linear_model.LogisticRegression(multi_class="multinomial", solver="lbfgs")
clf.fit(X, y)
_ = get_confusion_matrix(clf, X, y)

plot_decision_regions(clf, X, y, col_x="petalwidth", col_y="petallength")


I am working on the file "Lecture Notebooks/Redoing Weka stuff.ipynb".
The first task is:
Naive Bayes classifier
Can you write Python code for it?

clf = naive_bayes.GaussianNB()
clf.fit(X, y)
_ = get_confusion_matrix(clf, X, y)


I am working on the file "Lecture Notes/03-InformationBased/03-InformationBased.ipynb".
The first task is:
After being fitted, the model can then be used to predict the class of samples:
Can you write Python code for it?


I am working on the file "Lecture Notes/03-InformationBased/03-InformationBased.ipynb".
The first task is:
Information-base Learning
Let's review the basic idea of decision trees.
Shannon's model of entropy is a weighted sum of the logs of the probabilities of each of the possible outcomes when we make a random selection from a set.
$$
H(t) = - \sum_{i=1}^l \left( P(t=i)\times log_s(P(t=i)) \right)
$$
The information gain of a descriptive feature can be understood as a measure of the reduction in the overall entropy of a prediction task by testing on that feature.
Computing information gain involves the following 3 equations:
$$
H\left(t, \mathcal{D}\right) = - \sum_{l \in levels(t)} \left( P(t=l) \times log_2(P(t=l)) \right)
$$
$$
rem\left(d,\mathcal{D}\right) =
\sum_{l \in levels\left(d\right)}
\underbrace{
\frac{|\mathcal{D}{d=l}|}{|\mathcal{D}|}}{\text{weighting}} \times
 \underbrace{H\left(t, \mathcal{D}{d=l}\right)}{
    \substack{
        \text{entropy of}\
        \text{partition }\mathcal{D}_{d=l}
    }
}
$$
$$
IG\left(d,\mathcal{D}\right) = H\left(t,\mathcal{D}\right) - rem\left(d, \mathcal{D}\right)
$$
Implementing our own ID3
Let's implement a simplistic ID3 algorithm. We assume that each feature can only have two values. Hence we have two branches per tree: the left branch for the lower value or false, the right branch for higher or true.
There is no consideration of efficiency or even avoiding redundant computations. The algorithm also won't return the actual decision tree. We're just going to use print statement to indicate what the algorithm does.
However, this could be build out to something more useful...
Can you write Python code for it?

### define function information gain IG(d_feature, D_set)
def IG(d_feature, D_set):
    # compute the information gain by dividing with feature d
    return 0.0

def splitDataSet(d_feature, D_set):
    Left_set = []
    Right_set = []
    return (Left_set, Right_set)

def myID3(list_of_features, D_set):
    if len(list_of_features)==0:
        # we're done
    elif len(D_set)<=1:
        # we're done
    else:
        for f in list_of_features:
            # which feature has greatest information gain on D_set?
            (L_set, R_set) = splitDataSet(fmax, D_set)
            remfeatures = list_of_feature.remove(fmax)
            myID3(remfeatures, L_set)
            myID3(remfeatures, R_set)


I am working on the file "Lecture Notes/03-InformationBased/03-InformationBased.ipynb".
The first task is:
Once trained, we can export the tree in Graphviz format using the export_graphviz exporter. Below is an example export of a tree trained on the entire iris dataset.
In order to compute inline images the package pydot is required.
Can you write Python code for it?

from sklearn.externals.six import StringIO
import pydot
from IPython.display import Image

dot_data = StringIO()
tree.export_graphviz(clf, out_file=dot_data,
    feature_names=iris.feature_names)
graph = pydot.graph_from_dot_data(dot_data.getvalue())
graph.write_pdf("iris.pdf")
Image(graph.create_png())

import numpy as np
import matplotlib.pyplot as plt

from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier

# Parameters
n_classes = 3
plot_colors = "bry"
plot_step = 0.02

# Load data
iris = load_iris()

for pairidx, pair in enumerate([[0, 1], [0, 2], [0, 3],
                                [1, 2], [1, 3], [2, 3]]):
    # We only take the two corresponding features
    X = iris.data[:, pair]
    y = iris.target

    # Shuffle
    idx = np.arange(X.shape[0])
    np.random.seed(13)
    np.random.shuffle(idx)
    X = X[idx]
    y = y[idx]

    # Standardize
    mean = X.mean(axis=0)
    std = X.std(axis=0)
    X = (X - mean) / std

    # Train
    clf = DecisionTreeClassifier().fit(X, y)

    # Plot the decision boundary
    plt.subplot(2, 3, pairidx + 1)

    x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
    y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
    xx, yy = np.meshgrid(np.arange(x_min, x_max, plot_step),
                         np.arange(y_min, y_max, plot_step))

    Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])
    Z = Z.reshape(xx.shape)
    cs = plt.contourf(xx, yy, Z, cmap=plt.cm.Paired)

    plt.xlabel(iris.feature_names[pair[0]])
    plt.ylabel(iris.feature_names[pair[1]])
    plt.axis("tight")

    # Plot the training points
    for i, color in zip(range(n_classes), plot_colors):
        idx = np.where(y == i)
        plt.scatter(X[idx, 0], X[idx, 1], c=color, label=iris.target_names[i],
                    cmap=plt.cm.Paired)

    plt.axis("tight")

plt.suptitle("Decision surface of a decision tree using paired features")
plt.legend()
plt.show()


I am working on the file "Lecture Notes/03-InformationBased/03-InformationBased.ipynb".
The first task is:
Regression with Decision Trees in SciKit-learn
A 1D regression with decision tree.
The decision trees is used to fit a sine curve with addition noisy observation. As a result, it learns local linear regressions approximating the sine curve.
We can see that if the maximum depth of the tree (controlled by the max_depth parameter) is set too high, the decision trees learn too fine details of the training data and learn from the noise, i.e. they overfit.
Can you write Python code for it?

# Import the necessary modules and libraries
import numpy as np
from sklearn.tree import DecisionTreeRegressor
import matplotlib.pyplot as plt
%matplotlib inline

# Create a random dataset
rng = np.random.RandomState(1)
X = np.sort(5 * rng.rand(80, 1), axis=0)
y = np.sin(X).ravel()
y[::5] += 3 * (0.5 - rng.rand(16))

# Fit regression model
regr_1 = DecisionTreeRegressor(max_depth=2)
regr_2 = DecisionTreeRegressor(max_depth=5)
regr_1.fit(X, y)
regr_2.fit(X, y)

# Predict
X_test = np.arange(0.0, 5.0, 0.01)[:, np.newaxis]
y_1 = regr_1.predict(X_test)
y_2 = regr_2.predict(X_test)

# Plot the results
plt.figure()
plt.scatter(X, y, c="k", label="data")
plt.plot(X_test, y_1, c="g", label="max_depth=2", linewidth=2)
plt.plot(X_test, y_2, c="r", label="max_depth=5", linewidth=2)
plt.xlabel("data")
plt.ylabel("target")
plt.title("Decision Tree Regression")
plt.legend()
plt.show()

## Another Canned Example

print(__doc__)

import numpy as np
import matplotlib.pyplot as plt
from sklearn.tree import DecisionTreeRegressor

# Create a random dataset
rng = np.random.RandomState(1)
X = np.sort(200 * rng.rand(100, 1) - 100, axis=0)
y = np.array([np.pi * np.sin(X).ravel(), np.pi * np.cos(X).ravel()]).T
y[::5, :] += (0.5 - rng.rand(20, 2))

# Fit regression model
regr_1 = DecisionTreeRegressor(max_depth=2)
regr_2 = DecisionTreeRegressor(max_depth=5)
regr_3 = DecisionTreeRegressor(max_depth=8)
regr_1.fit(X, y)
regr_2.fit(X, y)
regr_3.fit(X, y)

# Predict
X_test = np.arange(-100.0, 100.0, 0.01)[:, np.newaxis]
y_1 = regr_1.predict(X_test)
y_2 = regr_2.predict(X_test)
y_3 = regr_3.predict(X_test)

# Plot the results
plt.figure()
plt.scatter(y[:, 0], y[:, 1], c="k", label="data")
plt.scatter(y_1[:, 0], y_1[:, 1], c="g", label="max_depth=2")
plt.scatter(y_2[:, 0], y_2[:, 1], c="r", label="max_depth=5")
plt.scatter(y_3[:, 0], y_3[:, 1], c="b", label="max_depth=8")
plt.xlim([-6, 6])
plt.ylim([-6, 6])
plt.xlabel("data")
plt.ylabel("target")
plt.title("Multi-output Decision Tree Regression")
plt.legend()
plt.show()


I am working on the file "Lecture Notes/03-InformationBased/03-InformationBased.ipynb".
The first task is:
Verify what we have
Let's find out how well the classifier predicts it's own training data. We use the method predict.
Can you write Python code for it?

Ncategory = len(levdict[6])
CM = [ [0 for c in range(Ncategory)] for r in range(Ncategory) ]
CM

Yhat = clf.predict(X)
CM = [ [0 for c in range(Ncategory)] for r in range(Ncategory) ]
for k in range(Nsample):
    CM[Yhat[k]][Y[k]] +=1

CM

for c in CM:
    print(c)


I am working on the file "Lecture Notes/Binning.ipynb".
The first task is:
Binning - Basic Ideas
Can you write Python code for it?


I am working on the file "Lecture Notes/Binning.ipynb".
The first task is:
Binning - Practicalities
For all of the plots we made so far, we allowed python to define the boundaries of the bins, but this "automatic binning" can be very problematic, as several of the previous plots will reveal if we look at them closely.
Can you write Python code for it?

#auto binning (max - min)/nbins
quarcs_data["PRE_SCORE"].hist(bins=25, figsize=(10,5))
ticks=np.arange(0,26,1)
plt.xticks(ticks)

#ovserplot hard-coded bins to check
bin_def = np.arange(1,25.9,24/25)
quarcs_data["PRE_SCORE"].hist(bins=bin_def, figsize=(10,5),alpha=0.5)


I am working on the file "Lecture Notes/Binning.ipynb".
The first task is:
Case Study - Continuous Variable
Can you write Python code for it?

#print(*quarcs_data.columns)

quarcs_data["CF_Mean"].hist(bins=10)

quarcs_data["CF_Mean"].hist(bins=5)

quarcs_data["CF_Mean"].hist(bins=25)

quarcs_data["CF_Mean"].hist(bins=50)


I am working on the file "Lecture Notes/Binning.ipynb".
The first task is:
For continuous variables, we don't have to be quite as careful, but there is a limit in that not all continuous variables are equally continuous. Many "continuous" variables were at some point calculated from discrete variables, which means there's a limitation to how many values they can take on. You can see this when you get high enough in bins for the average confidence rankings
Can you write Python code for it?


I am working on the file "Lecture Notes/Binning.ipynb".
The first task is:
Side Note: Binning in 2-Dimensions
Can you write Python code for it?

from astropy.io import fits

def rebin( a, newshape ):
        '''Rebin an array to an arbitrary new shape.
        '''
        assert len(a.shape) == len(newshape)

        slices = [ slice(0,old, float(old)/new) for old,new in zip(a.shape,newshape) ]
        coordinates = np.mgrid[slices]
        indices = coordinates.astype('i')   #choose the biggest smaller integer index
        return a[tuple(indices)]

def rebin_factor( a, newshape ):
        '''Rebin an array to a new shape.
        newshape must be a factor of a.shape.
        '''
        assert len(a.shape) == len(newshape)
        assert not sometrue(mod( a.shape, newshape ))

        slices = [ slice(None,None, old/new) for old,new in zip(a.shape,newshape) ]
        return a[slices]

def rebin_simple(a, newshape):
        assert len(a.shape) == len(newshape)
        assert not sometrue(mod( a.shape, newshape ))

im = fits.getdata('DSS_M51_15arcminsq.fits')

plt.imshow(im, cmap='gray')
print(im.shape)

im_rebin=rebin(im, (13.2,13.2))

plt.imshow(im_rebin, cmap='gray')


I am working on the file "Lecture Notes/Binning.ipynb".
The first task is:
So what if we want to downsample effectively?
Again, define bins that make sense by hand, where each includes the same integer number of possible scores (of course, this gets complicated when the number of possibilities/n bins is not itself an integer value. For example, binning total score by 2 leaves one of the 25 possible scores an "odd man out" since 25/2 is not an integer. In this case, we'll define the lowest bin as a score of 0 or 1, since 0 was technically allowed
Can you write Python code for it?

bin_def2=np.arange(0,28,2)-0.5
print(bin_def2)
quarcs_data["PRE_SCORE"].hist(bins=bin_def2)
plt.xticks(bin_def2+1)
plt.xlim(-0.5,25.5)

#bin by five
#another trick = offset upper bound by a tiny amount so that it includes the upper score in the bin (5,10,15,20,25)
bin_def3=[0,5.001,10.001,15.001,20.001,25.001]
ticks=[2.5,7.5,12.5,17.5,22.5]

quarcs_data["PRE_SCORE"].hist(bins=bin_def3)
plt.xticks(ticks)
plt.xlim(0,25)


I am working on the file "Lecture Notes/Binning.ipynb".
The first task is:
Tip 1 - when setting number of bins, make sure you understand where boundaries between them end up (e.g. by carefully examining graph)
Here we've naively specified 25 bins since there are 25 possible points. Since the minimum score is 1 for this dataset (it didn't have to be - 0 is actually the minimum possible score) and the maximum is 25, each bin has a width of (25 - 1)/25 or 24/25 = 0.96.
This causes some weirdness at the upper end of the plot, which you can see when you compare it to the same plot with bins separated by 1 point instead of 0.96 points.
Can you write Python code for it?

#side by side
plt.subplots(2,1, figsize=(10,5))
plt.subplot(211)
quarcs_data["PRE_SCORE"].hist(bins=24)
plt.subplot(212)
quarcs_data["PRE_SCORE"].hist(bins=25)
ticks=np.arange(0,26,1)
plt.xticks(ticks)


I am working on the file "Lecture Notes/ModelFitting.ipynb".
The first task is:
Chi Squared Test for Goodness of Fit
An alternate "quality of fit" statistic is the so-called chi-squared statistic. It's based on an underlying assumption that the distribution from which each of your measurements (each data point) is drawn is normal, which is usually a reasonable assumption.
$\chi^2 = \dfrac{(data-model)^2}{model}$
Can you write Python code for it?

chisq1 = sum((model1-data)**2/model1)
chisq1


I am working on the file "Lecture Notes/ModelFitting.ipynb".
The first task is:
General rule of thumb
Number of parameters in your model should be <<< number of data points
Fitting with error bars
Often we know enough about how our measurements are taken that we can assign "error bars" or "uncertainties" to our measurements
Can you write Python code for it?

# equal errors (homoschedastic)
errors_uniform = np.ones(11)

#errors that vary (heteroschedastic)
errors_poisson = np.sqrt(data)

#visualize this
plt.errorbar(x,data,yerr=errors_uniform, fmt='go')
plt.xlim(0,10)
plt.xlabel("independent variable")
plt.ylabel("dependent variable")
plt.title("homoschedastic error bars")

plt.errorbar(x,data,yerr=errors_uniform, fmt='go')
plt.xlim(0,10)
plt.plot(x, slopeintfunc(x,fit[0][0],fit[0][1]))
plt.xlabel("independent variable")
plt.ylabel("dependent variable")
plt.title("underestimated error bars (or bad model)")

plt.errorbar(x,data,yerr=errors_uniform*2.5, fmt='go')
plt.xlim(0,10)
plt.plot(x, slopeintfunc(x,fit[0][0],fit[0][1]))
plt.xlabel("independent variable")
plt.ylabel("dependent variable")
plt.title("error bars consistent with model")

plt.errorbar(x,data,yerr=errors_poisson, fmt='go')
plt.xlim(-1,11)
plt.xlabel("independent variable")
plt.ylabel("dependent variable")
plt.title("heteroschedastic error bars")