Some minor but life-saving tips in Python
By: A Ydobon
Tags: Numpy, Python, Classification, Confusion Matrix, Lstm

Hi, I hope everyone is still ok from this tedious 2020! I was gone for a few months, and I came back with some tips that I found life saving while preparing for my master’s degree paper. (can’t believe how time flies)

So, this post will not be consistent nor explanatory but it will be a list of problems that I faced while coding and their solutions to help those of you who might have the similar problems. I was able to get the solutions from searches after searches, and it is my turn to repay back for their help. Without the help of the people answering at stackoverflow.com or the anonymous bloggers who did such a work without any ‘economical’ interest, I think I would have given up at the very first stage.

Also, one more thing is that, in my case, regardless how much I knew about the algorithms and their mechanisms, ironically the very first codings were more challenging.

CAUTION! Unlike my previous posts that were mostly very detailed with step by step explanations, this one would be less like that.

1. arma_generate_sample

First, what I wanted to do was to generate, for example, AR(1) processes with distinct coefficients.

Before moving on, let’s have do a simple practice with AR(1) process. (which is very recommended to check your minor problems)

from statsmodels.tsa.arima_process import arma_generate_sample rho=0.5

nsample=200

ar_coefs = [1,-rho]

ma_coefs = [1,0] ar1=arma_generate_sample(ar_coefs, ma_coefs, nsample=100, sigma=1)

So in this coding I generated a sample from AR(1) process with its coefficient 0.5 with sample size=100. Which would look like following;

array([-0.43327517, 0.3061977 , 0.80159066, 1.49798894, 2.58501507, 1.67462174, 1.25875958, -1.42044711, -0.68007658, 0.37558947, …

…-0.26342701, -0.53688217, 1.93173622, 1.10047058, 0.6058739 ])

ar1.shape #(100,)

Tip: notice that ar_coefs=[1,-rho] “- rho”

Then, we would like to check if the rho=0.5 is well estimated by Python.

from statsmodels.tsa.stattools import acf

modela = sm.tsa.ARMA(ar1, (1, 0)).fit(trend='nc', disp=0) print('estimated rho=%f, actual rho= %.3f' % (modela.params, rho ))

This will result in:

estimated rho=0.559307, actual rho= 0.500

Fair enough! Now let’s try generating say, ten AR(1) samples with 100 time steps each.

Here my strategy was to define a function based on the previous practice and stack the 10 samples for later use.

np.random.seed(12345) def AR1(timestep):

rho = np.random.uniform(-1,1,1)+1e-6

ar_coefs = [1,-rho]

ma_coefs = [1,0]

ar1=arma_generate_sample(ar_coefs, ma_coefs, nsample=timestep, sigma=1)

ar1=np.array(ar1)

return ar1

Then from my expectation, AR1(100) should have the same shape as our previous exercise, however, things were a bit messy.

tmp=AR1(100)

tmp.shape #(100,)

The shape were the same but in depth, they were 100% different.

array([-0.9678381828411187, array([-1.39385533]), array([-0.88148675]), …, array([1.82240015]), array([3.56664748]), array([3.20947843]), array([3.55423975]), array([3.03438549]), array([1.81995384]), array([1.59298593])], dtype=object)

This is not how we want it.

Further more I had to notice that the dtype were also different.

ar1.dtype #dtype('float64')

tmp.dtype #dtype('O')

This had to be solved before being stacked!

I could not notice such problem until I actually commanded AR1(100) so to say until I saw how it looked like. I made the plot which worked fine and then the shape was the same, so it took me a while to even notice the problem. My tip is to at least see by your own eyes if it really works!

The solution was very simple! rho=float(rho)

def AR1(timestep):

rho = np.random.uniform(-1,1,1)+1e-6

rho=float(rho)

ar_coefs = [1,-rho]

ma_coefs = [1,0]

ar1=arma_generate_sample(ar_coefs, ma_coefs, nsample=timestep, sigma=1)

ar1=np.array(ar1)

return ar1

At last, it looked liked this;

ar1=AR1(100)

ar1.shape #(100,)

ar1.dtype #dtype('float64')

array([ 0.10050124, 0.64788036, -0.9043851 , 1.62197309, -1.11857796, 0.7803612 , -0.0176377 , 0.24278273, -0.74016969, 1.49048946, …,1.54750935, 2.40578187, -0.15412767])

2. Generate 10 AR(1) processes with T=100

Then, what I wanted to do was to stack 10 AR(1) processes which would look like following;

numpy stack or other things were also possible but I found the following way more intuitive; make an empty np array of intended shape and then, fill out the empty array.

rs=10

timestep=100 ar1 = np.empty((rs,timestep),dtype=float) for i in range(rs):

ar1[i]=AR1(timestep)

Let’s check the shape of the ar1.

ar1.shape #(10,100)

3. Save numpy array into npz file and then load

This is simple and easy to search, but anyway.

from numpy import save

from numpy import load from numpy import savez_compressed np.savez_compressed('file name here', ar1=ar1, ma1=ma1)

So, let’s say we made two arrays; an array with ten AR(1) processes with timesteps 100 (of shape (10,100)) and then same for MA(1) processes, with names ‘ar1’ and ‘ma1’ respectively.

Then the arrays will be saved in npz file for later use.

Now load the data

data = load('drive/My Drive/file name here.npz')

So, I mounted Google drive in my colab note book, which is very convenient, you should take care with the file path and name!

Then from the data you will be able to retrieve the ar1 and ma1 like following.

ar1 = data['ar1'] ma1 =data['ma1']

4. Data preprocess for LSTM

In this simple coding, I prepared two (10,100) datasets meaning that the data is 2D tensor. However, LSTM takes 3D tensor as an input with shape; (sample, time steps, features). Questions like how to reshape the data? or what is feature? might arise.

First, in our example, sample=10, time steps=100 and feature=1. Feature is the most tricky part, but it can be viewed as number of ‘features’ that characterize a sequence. So, in my example, a single AR(1) process is defined by a single row vector — feature=1

However, take example of the Human Activity classification problems. Your smartphone/Apple watch collects your health data; say heart beat, blood pressure and body movement. (don’t know how nor what kind of data it collects in reality) I mean, there exist detailed explanations about the dataset, but the medical terminologies complicate the problem.

Then by analyzing the three data at each time step the application will analyze whether you’re sleeping or walking or doing sports, etc. So, in this case feature=3.

ar1.shape #(10,100) rs=10

timestep=100

feature=1 ar1=ar1.reshape(rs,timestep,feature)

ma1=ma1.reshape(rs,timestep,feature)

Here I did the reshape for ar1 and ma1 respectively and then concatenated to make a train and test dataset. However the process can also be reversed. The motive for my procedure was for later use. Actually what I did was to make so called ‘prediction_dataset’ in my own words to see the accuracy for ar1 and ma1 separately.

Anyway,

X=np.concatenate((ar1,ma1),axis=0) X= np.asarray(X).astype(np.float32)

5. Binarize and categorize

So, my problem was to classify AR process vs MA process, and labeling can be done in 2 ways.

A. Binary labeling.

ar_label=np.zeros((rs,1))#(rs,1) is our desired shape ma_label=np.ones((rs,1))

ar_label and ma_label would look like this with shape (10,1) respectively;

array([[0.], [0.], [0.], [0.], [0.], [0.], [0.], [0.], [0.], [0.]]) and array([[1.], [1.], [1.], [1.], [1.], [1.], [1.], [1.], [1.], [1.]])

Then label for X can be made by concatenating the ar_label and ma_label.

label=np.concatenate([ar_label,ma_label],axis=0)

Here, label would look like this with shape (20,1);

array([[0.], [0.], [0.], [0.], [0.], [0.], [0.], [0.], [0.], [0.], [1.], [1.], [1.], [1.], [1.], [1.], [1.], [1.], [1.], [1.]])

B. Categorical labeling

One hot encoding is often preferred, so we can do that by doing the following.

y=tf.keras.utils.to_categorical(label,num_classes=2)

Then, y will look like with shape (20,2);

array([[1., 0.], [1., 0.], [1., 0.], [1., 0.], [1., 0.], [1., 0.], [1., 0.], [1., 0.], [1., 0.], [1., 0.], [0., 1.], [0., 1.], [0., 1.], [0., 1.], [0., 1.], [0., 1.], [0., 1.], [0., 1.], [0., 1.], [0., 1.]], dtype=float32)

C. Multi-label classification

So, my final goal was to do a multiple classification so, the label was made by using the following lines of code.

Similar to np.zeros or np.ones integers 2, 3, … can also fill a numpy array with np.full.

ar1_label=np.zeros((rs,1))

ar2_label=np.ones((rs,1))

ma1_label=np.full((rs,1),2)

ma2_label=np.full((rs,1),3)

For example, ma1_label would look like this:

array([[2], [2], [2], [2], [2], [2], [2], [2], [2], [2]])

Then, we want to do one-hot encoding. Actually, to_categorical is useful in this situation; it is time consuming to type [1,0,0,0] to [0,0,0,1]. Imagine if you are doing a classification with 10 categories!

label=np.concatenate([ar1_label,ar2_label,ma1_label,ma2_label],axis=0) y=tf.keras.utils.to_categorical(label,num_classes=4)

Here y will have a shape of (40,4) :

array([[1., 0., 0., 0.], …, [1., 0., 0., 0.], [0., 1., 0., 0.], …, [0., 1., 0., 0.], [0., 0., 1., 0.], … , [0., 0., 1., 0.], [0., 0., 0., 1.], …, [0., 0., 0., 1.]], dtype=float32)

D. Category to binary?

In some cases, you want to return from the one-hot encoded label to the labels 0 to 3. np.argmax can be used to do such job.

i) np.argmax

It returns the index of the maximum value.

From the numpy.org example, let x look like this with shape (2,3);

array([[10, 11, 12], [13, 14, 15]])

np.argmax(x) #5

When axis is not defined, the array will be flattened like [10,11,12,13,14,15]. Also, notice that index starts from 0, 5 implies that the 6th element has the maximum value.

np.argmax(x, axis=0) #array([1, 1, 1])

When axis=0 is defined, it will explore column-wise and then return in which row the largest value exists. So, to put it more specifically, for the first column the largest value is in the 2nd row, etc.

np.argmax(x, axis=1) #array([2, 2])

When axis=1 is defined, it will look through each row and tell in which column the largest value situates.

ii) Back to our example

So, what we want to do is to retrieve the numerical labels from the one-hot encoded labels.

So, from our practice with np.argmax, we can see that we need to look through row-wise and then get the index for the columns that have the maximum value (or which row has the value 1) — axis=1!!

y_non_cat=np.argmax(y,axis=1) y_non_cat.shape #(40,)

So, the numerical label can be returned like this; array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3]) with shape (40,).

5. Confusion Matrix

The numerical label was necessary to make the confusion matrix.

y_classes = np.argmax(ar12_ma12_model.predict(X),axis=1)

So, here y_classes will have the predicted labels in numerical version, i.e.,

array([0, 0, 0, …, 3, 3, 1])

By comparing it with the true labels, we can make a confusion matrix.

from sklearn.metrics import confusion_matrix cm =confusion_matrix(y_classes,y_non_cat)#(predicted result, truth)

Confusion matrix can later be used to calculate the precision, recall or to draw roc (receiver operating characteristic) curve or calculate AUC (area under roc curve) and PRC (precision-recall) curve.

This is all for this post, and I hope this list of problem shooting experience might help your struggles.

Thank you for reading and have a nice day!

Cecilia Kim.