I am working on the file "Single Wave- Interactive.ipynb".
The first task is:
STOP HERE
You can run another file be going back the the Begin User Input section and chose another file path.
What Should I do now?
Advanced user options
Approximate entropy
this only runs if you have pyeeg.py in the same folder as this notebook and bass.py. WARNING: THIS FUNCTION RUNS SLOWLY
run the below code to get the approximate entropy of any measurement or raw signal. Returns the entropy of the entire results array (no windowing). I am using the following M and R values:
M = 2
R = 0.2*std(measurement)

these values can be modified in the source code. alternatively, you can call ap_entropy directly. supports 'all'
Interpretation: A time series containing many repetitive patterns has a relatively small ApEn; a less predictable process has a higher ApEn.
Approximate Entropy in BASS
Aproximate Entropy Source
Events
Can you write Python code for it?

#Approximate Entropy
event_type = 'Peaks'
meas = 'all'
Results = ap_entropy_wrapper(event_type, meas, Data, Settings, Results)
Results['Approximate Entropy']


I am working on the file "Single Wave- Interactive.ipynb".
The first task is:
Sample Entropy
this only runs if you have pyeeg.py in the same folder as this notebook and bass.py. WARNING: THIS FUNCTION RUNS SLOWLY
run the below code to get the sample entropy of any measurement. Returns the entropy of the entire results array (no windowing). I am using the following M and R values:
M = 2
R = 0.2*std(measurement)

these values can be modified in the source code. alternatively, you can call samp_entropy directly.
Supports 'all'
Sample Entropy in BASS
Sample Entropy Source
Events
Can you write Python code for it?

#Sample Entropy
event_type = 'Bursts'
meas = 'all'
Results = samp_entropy_wrapper(event_type, meas, Data, Settings, Results)
Results['Sample Entropy']


I am working on the file "Single Wave- Interactive.ipynb".
The first task is:
Set Baseline for Thresholding
WARNING If you do not load a settings file OR enter your own settings, the analysis will not run. There are no defaults. This section is not optional.
Baseline
Choose either linear or rolling baseline.
Linear - takes a user specified time segment as a good representation of baseline. If the superstructure is linear but has a slope, use linear fit in the transformation to still use linear. Linear automatically shifts your data by the ammount of your baseline normalize the baseline to zero.
Rolling - rolling mean of the data is generated based on a moving window. User provides the window size in miliseconds. there is no shift in the data with this method.
Static - Skips baseline generatoin and allows you to choose an arbitrary y value for threshold. No Shift in the data.
Can you write Python code for it?


I am working on the file "Single Wave- Interactive.ipynb".
The first task is:
Spectrogram (Optional)
Use the block below to get the spectrogram of the signal. The frequency (y-axis) scales automatically to only show 'active' frequencies. This can take some time to run.
version = 'original'
key = 'Mean1'

After transformation is run, you can call version = 'trans'. This graph is not automatically saved.
Spectrogram
Can you write Python code for it?

version = 'original'
key = 'Mean1'
spectogram(version, key, Data, Settings, Results)


I am working on the file "Single Wave- Interactive.ipynb".
The first task is:
Time Series
Can you write Python code for it?

#on raw signal
#takes a VERY long time
version = 'original' #original, trans, shift, or rolling
key = 'Mean1' #Mean1 default key for one time series
start = 0 #seconds, where you want the slice to begin
end = 1 #seconds, where you want the slice to end. The absolute end is -1

samp_entropy(Data[version][key][start:end].tolist(), 2, (0.2*np.std(Data[version][key][start:end])))


I am working on the file "Single Wave- Interactive.ipynb".
The first task is:
Use the block below to generate the PSD graph and power in bands results (if selected). scale toggles which units to use for the graph:
raw = s^2/Hz
db = dB/Hz = 10*log10(s^2/Hz)

Graph and table are automatically saved in the PSD-Signal subfolder.
Can you write Python code for it?

scale = 'raw' #raw or db
Results = psd_signal(version = 'original', key = 'Mean1', scale = scale,
                     Data = Data, Settings = Settings, Results = Results)
Results['PSD-Signal']


I am working on the file "Singular Spectrum Analysis for NH Monthly Land Temperature.ipynb".
The first task is:
2. Read data
Can you write Python code for it?

df = pd.read_csv(r'data\NH.Ts.csv', parse_dates=True, index_col='Year', skiprows=1)
df = df['1881':'2016']
df.drop(['SON', 'MAM', 'DJF', 'JJA', 'J-D', 'D-N'], axis=1, inplace=True)
df1 = df.apply(pd.to_numeric, errors='coerce')
df1.tail()


I am working on the file "Singular Spectrum Analysis for NH Monthly Land Temperature.ipynb".
The first task is:
3. Carry out SSA analysis
The general procedure in SSA is as follows:
1. Embed the time series by forming a Hankel matrix of lagged window (length K) vectors.
2. Decompose the embedded time series via Singular Value Decomposition
3. Eigentripple Grouping is the process of identifying eigenvalue-eigenvector pairs as trend, seasonal and noise
4. Reconstruct the time series from the eigenvalue-eigenvector pairs identified as trend and seasonal.  This is done through a process called diagonal averaging.
Instantiate a ssa object with the time series
Can you write Python code for it?

ssa = mySSA(ts)

K = 40
suspected_seasonality = 12

ssa.embed(embedding_dimension=40, suspected_frequency=suspected_seasonality, verbose=True)

ssa.decompose(verbose=True)


I am working on the file "Singular Spectrum Analysis for NH Monthly Land Temperature.ipynb".
The first task is:
4. Forecast
To forecast using only some of the singular values, pass the list of singular value indices to be used to the forecast_recurrent method.
Take a forecast with 10 years ahead (i.e., 120 months) as an instance.
Can you write Python code for it?


I am working on the file "Singular Spectrum Analysis for NH Monthly Land Temperature.ipynb".
The first task is:
Convert data to monthly time series
Can you write Python code for it?

tmp = df1.values.astype(np.float)
tmp = tmp.reshape(tmp.shape[0]*12,1)
print(tmp.shape)

ds = pd.date_range(start='1/1/1881', end='12/1/2016', freq='MS')
ts = pd.DataFrame(tmp, columns=['TEMP'])
ts = ts.set_index(ds)


I am working on the file "Singular Spectrum Analysis for NH Monthly Land Temperature.ipynb".
The first task is:
Have a quick plot
Can you write Python code for it?


I am working on the file "Singular Spectrum Analysis for NH Monthly Land Temperature.ipynb".
The first task is:
Let's see the result if we reconstruct the signal from these 5 streams.  We add the first 5 signals and pass the result to object.view_reconstruction.
Can you write Python code for it?

streams5 = [i for i in range(5)]
reconstructed5 = ssa.view_reconstruction(*[ssa.Xs[i] for i in streams5], names=streams5, return_df=True, plot=False)

ts_copy5 = ssa.ts.copy()
ts_copy5['Reconstruction'] = reconstructed5.Reconstruction.values
ts_copy5.plot(title='Original vs. Reconstructed Time Series[First 5 RCs]');


I am working on the file "Singular Spectrum Analysis for NH Monthly Land Temperature.ipynb".
The first task is:
Now that we have the signal components (there are 15 additive components to the signal) we can loop over that range and look at each one individually.  We only look at the first 5 elements below:
The additive signal elements are stored in the object.Xs dictionary.
Can you write Python code for it?

rcParams['figure.figsize'] = 11, 2
for i in range(5):
    ssa.view_reconstruction(ssa.Xs[i], names=i, symmetric_plots=i!=0)
rcParams['figure.figsize'] = 11, 4


I am working on the file "Singular Spectrum Analysis for NH Monthly Land Temperature.ipynb".
The first task is:
That looks very reasonable! We can get closer to the original series by adding additional signals from the remaining singular values.  Let's use the first 15 singular values:
Can you write Python code for it?

streams15 = [i for i in range(15)]
reconstructed15 = ssa.view_reconstruction(*[ssa.Xs[i] for i in streams15],
                                          names=streams15, return_df=True, plot=False)
ts_copy15 = ssa.ts.copy()
ts_copy15['Reconstruction'] = reconstructed15.Reconstruction.values
ts_copy15.plot(title='Original vs. Reconstructed Time Series [First 15 RCs]');


I am working on the file "Singular Spectrum Analysis for NH Monthly Land Temperature.ipynb".
The first task is:
We can view the contribution of each of the signals (corresponding to each singular value) as follows:
Can you write Python code for it?


I am working on the file "SingularValueDecomposition.ipynb".
The first task is:
As it can be seen above, more rank is used, less error occur. From another perspective, we get closer to the original data by increasing rank number.
On the other hand, after a certain rank, using more rank will not contribute as much as
Let's compare a reconstructed column to the original one by just naked eyes. Even it is reconstructed using only 4 dimention, we almost, with some error, get the original data.
Can you write Python code for it?

print("Original:\n", A[:,10])
print("Reconstructed:\n", reconstruct(U, S, V, 4)[:,10])

imread("data/pacman.png", flatten=True).shape

A = np.mat(imread("data/pacman.png", flatten=True))

U, S, V = np.linalg.svd(A)

A.shape, U.shape, S.shape, V.shape

for rank in range(1, len(S)):
    rA = reconstruct(U, S, V, rank)
    error = calcError(A, rA)
    coverage = S[:rank].sum() / S.sum()
    print("with rank {}, coverage: {:.4f}, error: {:.4f}".format(rank, coverage, error))

for i in range(1, 50, 5):
    rA = reconstruct(U, S, V, i)
    print(rA.shape)
    plt.imshow(rA, cmap='gray')
    plt.show()

plt.imshow(data, interpolation='nearest')

128 * 128

- (10*128*2)

from PIL import Image
A = np.mat(imread("data/noise.png", flatten=True))
img = Image.open('data/noise.png')
imggray = img.convert('LA')

imgmat = np.array(list(imggray.getdata(band=0)), float)

imgmat = np.array(list(imggray.getdata(band=0)), float)
imgmat.shape = (imggray.size[1], imggray.size[0])
imgmat = np.matrix(imgmat)
plt.figure(figsize=(9,6))
plt.imshow(imgmat, cmap='gray');
plt.show()

U, S, V = np.linalg.svd(imgmat)

for i in range(1, 10, 1):
    rA = reconstruct(U, S, V, i)
    print(rA.shape)
    plt.imshow(rA, cmap='gray');
    plt.show()


I am working on the file "SingularValueDecomposition.ipynb".
The first task is:
Expectedly, the error is infinitesimal.
However, most of the time this is not our intention. Instead of using all the dimentions(rank), we only use some of them, which have more variance, in other words, which provides more information. Let's see what we will get when using only the first three most significant dimentions.
Can you write Python code for it?

reconstruct(U, S, V, 3)

calcError(A, reconstruct(U, S, V, 3))


I am working on the file "SingularValueDecomposition.ipynb".
The first task is:
Left singular vectors
Can you write Python code for it?


I am working on the file "SingularValueDecomposition.ipynb".
The first task is:
Singular values
Can you write Python code for it?


I am working on the file "SingularValueDecomposition.ipynb".
The first task is:
We use all the dimentions to get back to the original matrix. As a result, we obtain the matrix which is almost identical. Let's calculate the difference between the two matrices.
Can you write Python code for it?

def calcError(A, B):
    return np.sum(np.power(A - B, 2))

calcError(A, reconstruct(U, S, V, r))


I am working on the file "Siphon/.ipynb_checkpoints/casestudy-checkpoint.ipynb".
The first task is:
Dimensions, Coordinates, Data Variables
As far as the dimensions and coordinates go, the most relevant and important coordinates variables are x and y. We can see the data variables, such as, temperature (t), mixing ratio (mr), and potential temperature (th), are mostly on a 1901 x 1801 grid. There is also the mysterious nav dimension and associated data variables which we will be examining later.
Let's set a goal of visualizing potential temperature with the Cartopy plotting package.
The first step is to get more information concerning the variables we are interested in. For example, let's look at potential temperature or th.
Can you write Python code for it?


I am working on the file "Siphon/.ipynb_checkpoints/casestudy-checkpoint.ipynb".
The first task is:
Exercises for the Reader

The extents are actually not perfect and snip the image. Why? Fix.
Add a colorbar and jazz up the plot.
Trick question: Can you label the axes with latitude and longitudes?
Try a different projection which should be fairly easy with Cartopy.
Can you write Python code for it?


I am working on the file "Siphon/.ipynb_checkpoints/casestudy-checkpoint.ipynb".
The first task is:
Google to the Rescue
A simple Google search of GRIB-1 GDS data representation type takes us to
A GUIDE TO THE CODE FORM FM 92-IX Ext. GRIB Edition 1 from 1994 document. Therein one can find an explanation of the variables needed to understand the map projection. Let's review these variables.
Can you write Python code for it?


I am working on the file "Siphon/.ipynb_checkpoints/casestudy-checkpoint.ipynb".
The first task is:
La1 and Lo1
Next let's get La1 and Lo1 which are defined as the "first grid points" These are probably the latitude and longitude for one of the corners of the grid.
Can you write Python code for it?

la1, lo1 = ds['La1'].values[0], ds['Lo1'].values[0]
print(la1, lo1)


I am working on the file "Siphon/.ipynb_checkpoints/casestudy-checkpoint.ipynb".
The first task is:
Lambert Conformal Grid Extents

To plot the data we need the left,right,bottom,top extents of the grid expressed in Lambert Conformal
coordinates.
Key point: The projection coordinate systems have flat topology and Euclidean distance.

Calculating the Extents
Remember, we have:

The number of grid points in x and y (Nx, Ny)
The delta in meters between grid point (Dx, Dy)
The first latitude and longitude of the data (first latitude, first longitude).

We have one of the corners in latitude and longitude, but we need to convert it LC coordinates and derive the other corner.
Platte Carrée Projection
The Platte Carrée Projection is a very simple X,Y/Cartesian projection. It is used a lot in Cartopy because it allows you to express coordinates in familiar Latitude and Longitudes. Remember: The projection coordinate systems have flat topology and Euclidean distance.
Platte Carrée

Source: Wikipedia Source
Create the PlatteCarre Cartopy Projection
Can you write Python code for it?


I am working on the file "Siphon/.ipynb_checkpoints/casestudy-checkpoint.ipynb".
The first task is:
Latin1 and Latin2
Next up are the rather mysteriously named Latin1 and Latin2 variables. When I first saw these identifiers, I thought they referred to a Unicode block, but in fact they relate to the secants of the projection cone. I do not know why they are called "Latin" and this name is confusing. At any rate, we can feel comfortable that we are dealing with Lambert Conformal rather than Polar Stereographic.

Credit: http://www.geo.hunter.cuny.edu/~jochen
Can you write Python code for it?

latin1, latin2 = ds['Latin1'].values[0], ds['Latin2'].values[0]
print(latin1, latin2)


I am working on the file "Siphon/.ipynb_checkpoints/casestudy-checkpoint.ipynb".
The first task is:
Let's Review What We Have
We now have all the information we need to understand the Lambert projection:

The secants of the Lambert Conformal projection (Latin1, Latin2)
The central meridian of the projection (LoV)

Moreover, we have additional information that shows how the data grid relates to the projection:

The number of grid points in x and y (Nx, Ny)
The delta in meters between grid point (Dx, Dy)
The first latitude and longitude of the data (first latitude, first longitude).

We are Ready for Visualization (almost)!
Let's import cartopy and matplotlib.
Can you write Python code for it?

%matplotlib inline

import cartopy.crs as ccrs
import matplotlib.pyplot as plt
import matplotlib as mpl


I am working on the file "Siphon/.ipynb_checkpoints/casestudy-checkpoint.ipynb".
The first task is:
Plot It Up!
We now have the extents, we are ready to plot.
Can you write Python code for it?

#Define the figure
fig = plt.figure(figsize=(12, 12))

# Define the extents and add the data
ax = plt.axes(projection=proj)
extents = (left, right, bottom, top)
ax.contourf(th, origin='lower', extent=extents, transform=proj)

# Add bells and whistles
ax.coastlines(resolution='50m', color='black', linewidth=2)
ax.add_feature(ccrs.cartopy.feature.NaturalEarthFeature(category='cultural', name='admin_1_states_provinces_lines', scale='50m',facecolor='none'))
ax.add_feature(ccrs.cartopy.feature.BORDERS, linewidth='1', edgecolor='black')
ax.gridlines()

plt.show()


I am working on the file "Siphon/.ipynb_checkpoints/casestudy-checkpoint.ipynb".
The first task is:
The Central Meridian for the Lambert Conformal Projection, LoV
If we are defining a Lambert Conformal projection, we will require the central meridian that the GRIB documentation refers to as LoV.
Can you write Python code for it?


I am working on the file "Siphon/.ipynb_checkpoints/casestudy-checkpoint.ipynb".
The first task is:
Uh oh! Polar Stereographic or Lambert Conformal??
Note that this newest piece of information relating to a Lambert Conformal projection disagrees with the  earlier projection information about a Polar Stereographic projection. There is a bug in the metadata description of the projection.
Moving on Anyway, next Nx and Ny
According to the grib documentation Nx and Ny represent the number grid points along the x and y axes. Let's grab those.
Can you write Python code for it?

nx, ny = ds['Nx'].values[0], ds['Ny'].values[0]
print(nx, ny)


I am working on the file "Siphon/.ipynb_checkpoints/casestudy-checkpoint.ipynb".
The first task is:
Units for Dx and Dy
The units for the deltas are in meters.
Can you write Python code for it?

dx,dy = ds['Dx'].values[0],ds['Dy'].values[0]
print(dx,dy)


I am working on the file "Siphon/.ipynb_checkpoints/casestudy-checkpoint.ipynb".
The first task is:
potential temperature (th)
Let's grab the data array for potential temperature (th).
Can you write Python code for it?


I am working on the file "Siphon/casestudy.ipynb".
The first task is:
Define the Lambert Conformal Projection with Cartopy
Can you write Python code for it?


I am working on the file "Siphon/casestudy.ipynb".
The first task is:
Dimensions, Coordinates, Data Variables
As far as the dimensions and coordinates go, the most relevant and important coordinates variables are x and y. We can see the data variables, such as, temperature (t), mixing ratio (mr), and potential temperature (th), are mostly on a 1901 x 1801 grid. There is also the mysterious nav dimension and associated data variables which we will be examining later.
Let's set a goal of visualizing potential temperature with the Cartopy plotting package.
The first step is to get more information concerning the variables we are interested in. For example, let's look at potential temperature or th.
Can you write Python code for it?


I am working on the file "Siphon/casestudy.ipynb".
The first task is:
Dx and Dy
Finally, let's look at the grid increments. In particular, we need to find the units.
Can you write Python code for it?


I am working on the file "Siphon/casestudy.ipynb".
The first task is:
Exercises for the Reader

The extents are actually not perfect and snip the image. Why? Fix.
Add a colorbar and jazz up the plot.
Trick question: Can you label the axes with latitude and longitudes?
Try a different projection which should be fairly easy with Cartopy.
Can you write Python code for it?


I am working on the file "Siphon/casestudy.ipynb".
The first task is:
Lambert Conformal Grid Extents

To plot the data we need the left,right,bottom,top extents of the grid expressed in Lambert Conformal
coordinates.
Key point: The projection coordinate systems have flat topology and Euclidean distance.

Calculating the Extents
Remember, we have:

The number of grid points in x and y (Nx, Ny)
The delta in meters between grid point (Dx, Dy)
The first latitude and longitude of the data (first latitude, first longitude).

We have one of the corners in latitude and longitude, but we need to convert it LC coordinates and derive the other corner.
Platte Carrée Projection
The Platte Carrée Projection is a very simple X,Y/Cartesian projection. It is used a lot in Cartopy because it allows you to express coordinates in familiar Latitude and Longitudes. Remember: The projection coordinate systems have flat topology and Euclidean distance.
Platte Carrée

Source: Wikipedia Source
Create the PlatteCarre Cartopy Projection
Can you write Python code for it?


I am working on the file "Siphon/casestudy.ipynb".
The first task is:
Latin1 and Latin2
Next up are the rather mysteriously named Latin1 and Latin2 variables. When I first saw these identifiers, I thought they referred to a Unicode block, but in fact they relate to the secants of the projection cone. I do not know why they are called "Latin" and this name is confusing. At any rate, we can feel comfortable that we are dealing with Lambert Conformal rather than Polar Stereographic.

Credit: http://www.geo.hunter.cuny.edu/~jochen
Can you write Python code for it?


I am working on the file "Siphon/casestudy.ipynb".
The first task is:
Let's Review What We Have
We now have all the information we need to understand the Lambert projection:

The secants of the Lambert Conformal projection (Latin1, Latin2)
The central meridian of the projection (LoV)

Moreover, we have additional information that shows how the data grid relates to the projection:

The number of grid points in x and y (Nx, Ny)
The delta in meters between grid point (Dx, Dy)
The first latitude and longitude of the data (first latitude, first longitude).

We are Ready for Visualization (almost)!
Let's import cartopy and matplotlib.
Can you write Python code for it?


I am working on the file "Siphon/casestudy.ipynb".
The first task is:
Plot It Up!
We now have the extents, we are ready to plot.
Can you write Python code for it?


I am working on the file "Siphon/casestudy.ipynb".
The first task is:
To Visualize the Data, We have to Decrypt the Projection
In order, to visualize the data that are contained within a two-dimensional array onto a map that represents a three-dimensional globe, we need to understand the projection of the data.
We can make an educated guess these are contained in the data variables with the nav cooridinate variable.
Specifically,

grid_type
grid_type_code
x_dim
y_dim
Nx
Ny
La1
Lo1
LoV
Latin1
Latin2
Dx
Dy

But what are these??
For Grins, Let's Scrutinize the grid_type_code
Can you write Python code for it?


I am working on the file "Siphon/casestudy.ipynb".
The first task is:
Uh oh! Polar Stereographic or Lambert Conformal??
Note that this newest piece of information relating to a Lambert Conformal projection disagrees with the  earlier projection information about a Polar Stereographic projection. There is a bug in the metadata description of the projection.
Moving on Anyway, next Nx and Ny
According to the grib documentation Nx and Ny represent the number grid points along the x and y axes. Let's grab those.
Can you write Python code for it?


I am working on the file "Siphon/casestudy.ipynb".
The first task is:
Units for Dx and Dy
The units for the deltas are in meters.
Can you write Python code for it?


I am working on the file "Siphon/casestudy.ipynb".
The first task is:
What is grid_type_code of 5?
Let's look at Table 6 . A grid_type_code of 5 corresponds to a projection of Polar Stereographic.
Next up grid_type
Can you write Python code for it?

grid_type = ds['grid_type'].values
print('The grid type is ', grid_type[0])


I am working on the file "Siphon/casestudy.ipynb".
The first task is:
potential temperature (th)
Let's grab the data array for potential temperature (th).
Can you write Python code for it?


I am working on the file "Sistemas De Recomendacion.ipynb".
The first task is:
BUENAS PRACTICAS: Validacion Cruzada
```python
lista_error_trainig=[]
lista_error_tests=[]
max_iteraciones=[1,3,12,15,18,24]
for max_ite in max_iteraciones:
    algoritmo = ALS(regParam=0.001,rank=4,maxIter=max_ite, userCol=USERID, itemCol=PRODUCTID, ratingCol=TARGET)
    modelo=algoritmo.fit(train_data)
    (error_trainig,error_tests)=error_train_test(modelo,train_data,test_data,algoritmo.getPredictionCol(),algoritmo.getRatingCol())
    lista_error_trainig.append(error_trainig)
    lista_error_tests.append(error_tests)
    print ("MaxIteraciones: {}, ErrorValidacion: {}, ErrorTest: {}".format(max_ite,error_trainig,error_tests))
MaxIteraciones: 1, ErrorValidacion: 2.944017127892633, ErrorTest: 3.3589322628528784
MaxIteraciones: 3, ErrorValidacion: 0.841464796797687, ErrorTest: 0.9710850706836164
MaxIteraciones: 12, ErrorValidacion: 0.798266313670666, ErrorTest: 0.9474099321913148
MaxIteraciones: 15, ErrorValidacion: 0.7970887322939875, ErrorTest: 0.9487053356238591
MaxIteraciones: 18, ErrorValidacion: 0.7962955748882709, ErrorTest: 0.9532481382416105
MaxIteraciones: 24, ErrorValidacion: 0.7954616942763565, ErrorTest: 0.958973712003286
```

Para cada tupla de parametros se itera tantas veces como hemos expecificado y se guarda la media de errores generados

Eligemos el modelo con menor error medio

Conjunto de entrenamiento y testeo
Vamos a crear dos conjuntos para poder afinar los hiperparametros del algoritmo ALS:
- Entrenamiento: Con este conjunto de datos se validan los hiperparametros del modelo
- Tests: Una vez que nuestro modelo está entrenado con unos hiperparaemtros adecuados, se testea con otro conjunto el desempeño del modelo
Can you write Python code for it?

from sklearn.cross_validation import train_test_split
X_historico, X_test = train_test_split(
    historico_valoraciones, test_size=0.3, random_state=0)

X_historico = pd.DataFrame(X_historico,columns=[USERID,PRODUCTID,TARGET])
X_test = pd.DataFrame(X_test,columns=[USERID,PRODUCTID,TARGET])

# Añado mis recomendaciones
X_train = pd.concat([mis_valoraciones,X_historico])

from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)

train_data = sqlContext.createDataFrame(data=X_train).cache()
test_data = sqlContext.createDataFrame(data=X_test).cache()

algoritmo = ALS(rank=5,userCol=USERID, itemCol=PRODUCTID, ratingCol=TARGET)
grid_parametros = ParamGridBuilder().addGrid(algoritmo.regParam, [0.1,0.25,0.5]).build()
evaluador = CF.EvaluadorRMSE(predictionCol=algoritmo.getPredictionCol(), targetCol=algoritmo.getRatingCol())

cv = CrossValidator(estimator=algoritmo, estimatorParamMaps=grid_parametros, evaluator=evaluador, numFolds=2)
cv_model = cv.fit(train_data)

(error_training_als,error_test_als)=error_train_test(cv_model,train_data,test_data,algoritmo.getPredictionCol(),algoritmo.getRatingCol())
print ("ErrorValidacion: {}, ErrorTest: {}".format(error_training_als,error_test_als))


I am working on the file "Sistemas De Recomendacion.ipynb".
The first task is:
Comparativa de las mejoras realizadas

tf: Poderamos y normalizados los tags de cada articulo
tf-idx: Poderamos y normalizados los tags de cada articulo teniendo en cuenta la frecuencia de cada tag en los demas articulos
tf_idx_filtrado: Eliminamos las palabras mas comunes y los numeros a tf-idx
Can you write Python code for it?


I am working on the file "Sistemas De Recomendacion.ipynb".
The first task is:
Forma rapida
Todos los pasos anteriores se puede simplificar por medio de un metodo de sklearn que los agrutina
Can you write Python code for it?

tfidf_vectorizer = TfidfVectorizer( stop_words=cachedStopWords, token_pattern='(?u)\\b[a-zA-Z]\\w\\w+\\b')
tfidf_vectorizer.fit(datos.text)

CB.mostrar_pesos_tags(tfidf_vectorizer.transform(entrada_elton_john.text),tfidf_vectorizer)


I am working on the file "Sistemas De Recomendacion.ipynb".
The first task is:
Fuck, los tags mas importante no describen los articulos
- SOLUCION: Normalizar y poderar teniendo en cuenta la frecuencia de cada tag en los demas documenos
Can you write Python code for it?


I am working on the file "Sistemas De Recomendacion.ipynb".
The first task is:
Generamos nuestas valoraciones
A partir de las peliculas mas vistras
Can you write Python code for it?

id_peliculas_mas_vistas=historico_valoraciones.groupby(by=PRODUCTID).count().sort(USERID,ascending=False).head(10).index.values
id_peliculas_mas_vistas

peliculas_mas_vistas = relacion_peliculas[relacion_peliculas[PRODUCTID].isin(id_peliculas_mas_vistas)]
peliculas_mas_vistas

ranking_mio = []
for fila in peliculas_mas_vistas.itertuples():
    puntuacion = input("Puntuacion de 1 al 5 de {}: ".format(fila[2]))
    ranking_mio.append([0,fila[1],float(puntuacion)])
mis_valoraciones = pd.DataFrame(data=ranking_mio,columns=[USERID,PRODUCTID,TARGET])
mis_valoraciones[TARGET] = mis_valoraciones[TARGET].astype(np.float64,copy=False)


I am working on the file "Sistemas De Recomendacion.ipynb".
The first task is:
Hora de la verdad
Vamos a comparar el sistema con un modelo base definido como la media de las valoraciones en el conjunto entrenado
Can you write Python code for it?

valoracion_media = X_train[TARGET].mean()
prediccion = (X_test[TARGET]-valoracion_media)**2
rmse_modelo_base = sqrt(prediccion.sum()/prediccion.count())

print ("la mejora del modelo ALS respecto al modelo base es: {0:.2f}%".format((rmse_modelo_base-error_test_als)/rmse_modelo_base*100))


I am working on the file "Sistemas De Recomendacion.ipynb".
The first task is:
NOTA: Concepto de metrica
Decir similitud, necesariamente tenemos que hablar de metricas, normalmente utilizamos la euclidia en la vida real pero en sistemas recomendados, lo mas habitual es la llamada cosine
Can you write Python code for it?

from sklearn.metrics.pairwise import pairwise_distances

a=np.array([2,2])
b=np.array([4,4])
print ("Distancia Euclida: {}, Distancia cosine: {}".format(pairwise_distances(a,b, metric='l2'),pairwise_distances(a,b, metric='cosine')))
ax1 = plt.axes(xlim=[0,5],ylim=[0,5])
ax1.arrow(0, 0, b[0], b[1], head_width=0.05, fc='k', ec='k')
ax1.arrow(0, 0, a[0], a[1], width=0.05,head_width=0.2, fc='k', ec='k')
plt.show()

vecinos.fit(datos_por_tags)


I am working on the file "Sistemas De Recomendacion.ipynb".
The first task is:
Normalizacion
Pesos los tags
Can you write Python code for it?


I am working on the file "Sistemas De Recomendacion.ipynb".
The first task is:
Prediccion
Hemos creado un envoltorio al modelo ALS para ayudarnos a generar predicciones a partir de las matrices con los factores ocultos
Rating de nuestras valoraciones
Can you write Python code for it?


I am working on the file "Sistemas De Recomendacion.ipynb".
The first task is:
Prediccion
Una vez que ya hemos extraidos los tags de los productos, buscamos por similitud los mas parecidos
Can you write Python code for it?

vecinos            = NearestNeighbors(n_neighbors=5,metric='cosine',algorithm='brute')
datos_por_tags     = tfidf_vectorizer.transform(datos.text)


I am working on the file "Sistemas De Recomendacion.ipynb".
The first task is:
Se puede comprobar que de los 5, 4 de ellos ha ganado por menos un oscar !!!!
Can you write Python code for it?

content_based.predict(['Award Actor Oscar'])

content_based.predict(datos.query('name == "Al Pacino"').text)


I am working on the file "Sistemas De Recomendacion.ipynb".
The first task is:
Sistemas de recomendacion
Tabla de contenidos

Sistemas de recomendacion
Content Based
CollaborativeFiltering

Sistemas de recomendacion
Los sistemas de recomendación están extendidos en todo sitios y nos dicen donde comer, que ver, escuchar y incluso que amigos tener

Los tipos mas extendidos son:
- Content-Based: Las recomendaciones son dadas por medio de los atributos del producto buscando productos similares
- Collaborative Filtering: Recomendaciones basadas en el comportamiento de otros individuos

Basado en el Contenido (ContendBased)
Por medio de la descripción del producto. Relaciones de tags

Vectorizacion del contenido, es decir, generar los tags
Ponderar los tags
Generar el motor de buscada de los productos similares

Ejemplo1: Recomendacion de articulos basados en su contenido
Vamos a realizar una demostracion de sistema de recomendacion basado en contenido para recomendar articulos
Can you write Python code for it?

import pandas as pd
import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
from sklearn.neighbors import NearestNeighbors

import ContendBased as CB

NOMBRE_COLUMNAS = ['UsuarioID','PeliculaID','Valoracion']

datos = pd.read_csv(RUTA_BD+"/people_wiki.csv")
datos.head(3)

entrada_elton_john=datos.query('name == "Elton John"')
entrada_elton_john


I am working on the file "Sistemas De Recomendacion.ipynb".
The first task is:
Vamos por el buen camino, vamos a ayudar al algoritmo de dos formas:
1. Eliminando las palabras mas comunes --> stopwords
2. Eliminado los numeros: Fuera de su contexto pierden todo su significado
Can you write Python code for it?

import nltk
nltk.download('stopwords')

from nltk.corpus import stopwords
cachedStopWords = stopwords.words("english")
cachedStopWords[:10]

vectorizacion_stop_words     = CountVectorizer(stop_words = cachedStopWords, token_pattern='(?u)\\b[a-zA-Z]\\w\\w+\\b')
bag_of_words_stop_words      = vectorizacion_stop_words.fit_transform(datos.text)
tfidf_transformer_stop_words = TfidfTransformer(use_idf=True).fit(bag_of_words_stop_words)

matrix_elton_john_tf_idx_stop_words=tfidf_transformer_stop_words.transform(bag_of_words_stop_words[entrada_elton_john.index.values])
pesos_tf_idx_filtrado=CB.mostrar_pesos_tags(matrix_elton_john_tf_idx_stop_words,vectorizacion_stop_words,descripcion='tf_idx_filtrado')
pesos_tf_idx_filtrado


I am working on the file "Six for a Spell Solver.ipynb".
The first task is:
The Graph
The letters and connections are well suited to a graph representation.  We could use simpler data structures, but using NetworkX seems like a nice match.  And it will give me a chance to play with that library a little more.
Can you write Python code for it?


I am working on the file "Six for a Spell Solver.ipynb".
The first task is:
The Search
The idea is to search depth first but abandon the search tree if it is clear that no words are possible down that path.
Can you write Python code for it?

def search_dfs(G, node, is_word, max_len=6, prefix=None):
    """Search the graph `G` starting at `node` for any words that pass `is_word`"""
    prefix = (prefix or '') + node.letter
    # print('prefix: {}'.format(prefix))
    candidates = []
    for n in G[node]:
        candidate = prefix + n.letter
        if len(candidate) == max_len:
            if is_word(candidate):
                candidates.append(candidate)
        else:
            candidates.extend(search_bfs(G, n, is_word, max_len=max_len, prefix=prefix))
    return candidates

def find_words(G, is_word, max_len=6):
    word_list = []
    for node in G:
        print(node.letter)
        w = search_dfs(G, node, is_word=is_word, max_len=max_len)
        print(w)
        word_list.extend(w)
    print('Word list:')
    print(sorted(set(word_list)))
    print('Words found:', len(set(word_list)))

find_words(G, is_word=check_word, max_len=6)


I am working on the file "Sjoin/Sjoin_Pyspark_4_byWDandHour.ipynb".
The first task is:
Function
Can you write Python code for it?

def countLine(partID, records):
    import pyproj
    import csv
    import shapely.geometry as geom
    import fiona
    import fiona.crs
    import shapely
    import rtree
    import geopandas as gpd
    import numpy as np
    import operator
    import pandas as pd
    import datetime

    shapefile = '../why_yellow_taxi/Buffer/entr_buffer_100_feet_epsg4269_nad83/entr_buffer_100_feet_epsg4269_nad83.shp'
    entr_buf = gpd.read_file(shapefile)
    entr_buf = entr_buf.to_crs(fiona.crs.from_epsg(2263))

    routes = ['Route_' + str(n) for n in range(1, 12)]
    entr2line = []
    for i in xrange(len(entr_buf)):
        lines = []
        for line in list(entr_buf.loc[:,routes].ix[i].dropna().values):
            try:
                line = str(int(line))
            except ValueError:
                pass
            lines.append(line)
        entr2line.append(lines)
    entr_buf['entr2line'] = entr2line

    index = rtree.Rtree()
    for idx, geometry in enumerate(entr_buf.geometry):
        index.insert(idx, geometry.bounds)

    entr_pair = {}
    pick_entr = {}
    drop_entr = {}
    entr_lines = {}

    proj = pyproj.Proj(init='epsg:2263', preserve_units=True)

    if partID==0:
        records.next()
    reader = csv.reader(records)
    for row in reader:
        if ((float(row[5])!=0) and float(row[9]!=0)):
            if row[1]:
                wd_h = datetime.datetime.strptime(row[1], '%Y-%m-%d %H:%M:%S')
                wd = wd_h.weekday()
                hour = wd_h.hour
            else:
                wd = None
                hour = None

            p = geom.Point(proj(float(row[5]), float(row[6])))
            d = geom.Point(proj(float(row[9]), float(row[10])))
            p_potential = index.intersection((p.x,p.y,p.x,p.y))
            d_potential = index.intersection((d.x,d.y,d.x,d.y))
            p_match = None # The first one match, should be the closest one? No!
            d_match = None

            for p_idx in p_potential:
                if...


I am working on the file "Skin_Proximity_effect/ImpedanceMethod/impedance_method.ipynb".
The first task is:
Power loss calculation
Can you write Python code for it?

#Calculating the module of the current
modI = [abs(Ical[i]) for i in range(0, n)]

Pac = sum([abs(Ical[i])**2*R for i in range(0, n)])

Pdc = I1**2*l/(s*h*w)

k = Pac/Pdc

Pac, Pdc, k


I am working on the file "Skin_Proximity_effect/ImpedanceMethod/impedance_method.ipynb".
The first task is:
The goal is to solve a matrix equation given with this formula
$\mathbf{Z} \mathbf{I} = \mathbf {u} $
where:
Z is the impedance matrix;

I is the current distribution vector;

u is the voltage vector
Can you write Python code for it?

#supply voltage [V]
u1 = 1

#voltage vector
u = [u1 for i in range(0, n)]

#own inductance of a piece of the conductor
L = 1e-9*2*100*l*(math.log(2*l/(0.5*dy))-3/4)

#resistance of a piece of the conductor [Ohm]
R = l/(s*dA)

L, R

import numpy as np

#impedance matrix calculation
Z = np.zeros((n,n), dtype=complex)

for i in range(0, n):
    for j in range(0, n):
        if i==j:
            Z[i,j]=R+1j*omega*L
        else:
            Z[i,j]=1j*omega*1e-9*2*l*100*(math.log(2*l*100/(abs(10*i-10*j)/10))-3/4)

#Mutual inductance matrix

M = np.zeros((n,n), dtype=complex)

for i in range(0, n):
    for j in range(0, n):
        if i==j:
            M[i,j]=0
        else:
            M[i,j]=1e-9*2*l*100*(math.log(2*l*100/(abs(10*i-10*j)/10))-3/4)

from scipy import linalg

I = linalg.solve(Z,u)

#Calibrating for the current
cal = np.sum(I)

#calibrated current
Ical = [I1*I[i]/cal for i in range(0, n)]

#Current density
J = [Ical[i]/(w*dy*10**6) for i in range(0, n)]

#mode of the current density
modJ = [abs(J[i]) for i in range(0, n)]

#phase of the current density
phJ = [math.degrees(math.atan(J[i].imag/J[i].real)) for i in range(0, n)]

#Plotting
import seaborn
import matplotlib.pyplot as plt
%matplotlib inline

#conductor height vector
y = np.linspace(0, 100, n)

#plot preparation
fig, ax = plt.subplots(2, figsize=(10,10))

ax[0].plot(y,modJ, label="|J| [A/mm2]", color = "green")
ax[0].set_ylabel("Current density [A/mm2]")
ax[0].set_xlabel("Conductor height [mm]")
ax[0].set_title("Current density distribution along conductor width")
ax[0].legend()

ax[1].plot(y,phJ, label="Arg{J} [deg]", color = "blue")
ax[1].set_ylabel("Current density phase [deg]")
ax[1].set_xlabel("Conductor height [mm]")
ax[1].legend()
plt.tight_layout()


I am working on the file "Sklearn_MLPython/CH01.ipynb".
The first task is:
Classification Report
Accuracy = (TP+TN)/m
Precision = TP/(TP+FP)
Recall = TP/(TP+FN)
F1-score = 2 * Precision * Recall / (Precision + Recall)
Can you write Python code for it?

# predicting
print(classification_report(y_pred=y_pred, y_true=y_test))

confusion_matrix(y_pred=y_pred, y_true=y_test)


I am working on the file "Sklearn_MLPython/cross_validation-0.18.ipynb".
The first task is:
Cross-validation iterators for grouped data
The i.i.d. assumption is broken if the underlying generative process yield groups of dependent samples.
Such a grouping of data is domain specific. An example would be when there is medical data collected from multiple patients, with multiple samples taken from each patient. And such data is likely to be dependent on the individual group. In our example, the patient id for each sample will be its group identifier.
In this case we would like to know if a model trained on a particular set of groups generalizes well to the unseen groups. To measure this, we need to ensure that all the samples in the validation fold come from groups that are not represented at all in the paired training fold.
The following cross-validation splitters can be used to do that. The grouping identifier for the samples is specified via the groups parameter.
 Group k-fold
class:GroupKFold is a variation of k-fold which ensures that the same group is not represented in both testing and training sets. For example if the data is obtained from different subjects with several samples per-subject and if the model is flexible enough to learn from highly person specific features it could fail to generalize to new subjects. class:GroupKFold makes it possible to detect this kind of overfitting situations.
Can you write Python code for it?


I am working on the file "Sklearn_MLPython/cross_validation-0.18.ipynb".
The first task is:
Group Shuffle Split
The GroupShuffleSplit iterator behaves as a combination of ShuffleSplit and LeavePGroupsOut, and generates a sequence of randomized partitions in which a subset of groups are held out for each split.
Can you write Python code for it?


I am working on the file "Sklearn_MLPython/cross_validation-0.18.ipynb".
The first task is:
Leave One Out (LOO)   - LeaveOneOut (or LOO) is a simple cross-validation. Each learning set is created by taking all the samples except one, the test set being the sample left out. Thus, for n samples, we have n different training sets and n different tests set. This cross-validation procedure does not waste much data as only one sample is removed from the training set:
Can you write Python code for it?

X = [1,2,3,4,5]
loo = LeaveOneOut()
print(loo)
for i in loo.split(X):
    print(i)


I am working on the file "Sklearn_MLPython/cross_validation-0.18.ipynb".
The first task is:
Leave P Groups Out
LeavePGroupsOut is similar as LeaveOneGroupOut, but removes samples related to P groups for each training/test set.
Can you write Python code for it?


I am working on the file "Sklearn_MLPython/cross_validation-0.18.ipynb".
The first task is:
Leave P Out (LPO)  - LeavePOut is very similar to LeaveOneOut as it creates all the possible training/test sets by removing p samples from the complete set. For n samples, this produces {n \choose p} train-test pairs. Unlike LeaveOneOut and KFold, the test sets will overlap for p > 1
Can you write Python code for it?

X = [1,2,3,4,5]
loo = LeavePOut(p=3)
print(loo)
for i in loo.split(X):
    print(i)


I am working on the file "Sklearn_MLPython/cross_validation-0.18.ipynb".
The first task is:
Random permutations cross-validation a.k.a. Shuffle & Split  - The ShuffleSplit iterator will generate a user defined number of independent train / test dataset splits. Samples are first shuffled and then split into a pair of train and test sets.
It is possible to control the randomness for reproducibility of the results by explicitly seeding the random_state pseudo random number generator.
Can you write Python code for it?

X = [1,2,3,4,5]
loo = ShuffleSplit(n_splits=3, test_size=0.25,random_state=0)
print(loo)
for i in loo.split(X):
    print(i)


I am working on the file "Sklearn_MLPython/cross_validation-0.18.ipynb".
The first task is:
Some classification problems can exhibit a large imbalance in the distribution of the target classes: for instance there could be several times more negative samples than positive samples. In such cases it is recommended to use stratified sampling as implemented in StratifiedKFold and StratifiedShuffleSplit to ensure that relative class frequencies is approximately preserved in each train and validation fold.
 Stratified k-fold
StratifiedKFold is a variation of k-fold which returns stratified folds: each set contains approximately the same percentage of samples of each target class as the complete set.
Can you write Python code for it?

X = np.ones(10)
y = [0, 0, 0, 0, 1, 1, 1, 1, 1, 1]

skf = StratifiedKFold(n_splits=3)
for i in skf.split(X, y):
    print(i)


I am working on the file "Sklearn_MLPython/cross_validation-0.18.ipynb".
The first task is:
Time Series Split
TimeSeriesSplit is a variation of k-fold which returns first k folds as train set and the (k+1) th fold as test set. Note that unlike standard cross-validation methods, successive training sets are supersets of those that come before them. Also, it adds all surplus data to the first training partition, which is always used to train the model.
This class can be used to cross-validate time series data samples that are observed at fixed time intervals.
Can you write Python code for it?

X = np.array([[1, 2], [3, 4], [1, 2], [3, 4], [1, 2], [3, 4]])
y = np.array([1, 2, 3, 4, 5, 6])
tscv = TimeSeriesSplit(n_splits=3)
print(tscv)
for train, test in tscv.split(X):
    print("%s %s" % (train, test))


I am working on the file "Sklearn_MLPython/cross_validation-0.18.ipynb".
The first task is:
cross_val_score uses the KFold or StratifiedKFold strategies by default
Can you write Python code for it?

# define cross_val func

def xVal_score(clf, X, y, K):

    # creating K using KFold
    cv = KFold(n_splits=2)

    # Can use suffle as well
    # cv = ShuffleSplit(n_splits=3, test_size=0.3, random_state=0)

    # doing cross validation
    scores = cross_val_score(clf, X, y, cv=cv)
    print(scores)
    print("Accuracy Mean : %0.3f" %np.mean(scores))
    print("Std : ", np.std(scores))
    print("Standard Err : +/- {0:0.6f} ".format(sem(scores)))

svc1 = SVC()
xVal_score(svc1, X_train, y_train, 10)

# define cross_val predict
# The function cross_val_predict has a similar interface to cross_val_score, but returns,
# for each element in the input, the prediction that was obtained for that element when it
# was in the test set. Only cross-validation strategies that assign all elements to a test
# set exactly once can be used (otherwise, an exception is raised).

def xVal_predict(clf, X, y, K):

    # creating K using KFold
    cv = KFold(n_splits=K)

    # Can use suffle as well
    # cv = ShuffleSplit(n_splits=3, test_size=0.3, random_state=0)

    # doing cross validation prediction
    predicted = cross_val_predict(clf, X, y, cv=cv)
    print(predicted)
    print("Accuracy Score : %0.3f" % accuracy_score(y, predicted))

xVal_predict(svc1, X_train, y_train, 10)


I am working on the file "Sklearn_MLPython/cross_validation.ipynb".
The first task is:
Leave One Out (LOO)   - LeaveOneOut (or LOO) is a simple cross-validation. Each learning set is created by taking all the samples except one, the test set being the sample left out. Thus, for n samples, we have n different training sets and n different tests set. This cross-validation procedure does not waste much data as only one sample is removed from the training set:
Can you write Python code for it?

X = [1,2,3,4,5]
loo = LeaveOneOut(len(X))
print(loo)
for i in loo:
    print(i)


I am working on the file "Sklearn_MLPython/cross_validation.ipynb".
The first task is:
Some classification problems can exhibit a large imbalance in the distribution of the target classes: for instance there could be several times more negative samples than positive samples. In such cases it is recommended to use stratified sampling as implemented in StratifiedKFold and StratifiedShuffleSplit to ensure that relative class frequencies is approximately preserved in each train and validation fold.
 Stratified k-fold
StratifiedKFold is a variation of k-fold which returns stratified folds: each set contains approximately the same percentage of samples of each target class as the complete set.
Can you write Python code for it?

X = np.ones(10)
y = [0, 0, 0, 0, 1, 1, 1, 1, 1, 1]

skf = StratifiedKFold(n_folds=4, y=y)
for i in skf:
    print(i)

skf.


I am working on the file "Skripsi- Parsing Engine.ipynb".
The first task is:
Testing
feeds = [['detik','http://detik.feedsportal.com/c/33613/f/656089/index.rss'],
         ['viva','http://rss.viva.co.id/get/bisnis'],
         ['merdeka','http://www.merdeka.com/feed/'],
         ['liputan6','http://www.liputan6.com/feed/rss2'],
         ['tribun','http://www.tribunnews.com/rss/bisnis'],
         ['okezone','http://sindikasi.okezone.com/index.php/rss/11/RSS2.0'],
         ['jpnn','http://www.jpnn.com/index.php?mib=rss&id=216'],
         ['suara','http://www.suara.com/rss/bisnis'],
         ['bisniscom','http://www.bisnis.com/rss/index?c=382']]
Can you write Python code for it?

# server = 'localhost'
# port = 27017
# db_name = 'thosangs-news'
# username = 'userSkripsi'

import pymongo
print 'Python version', sys.version
print 'Pymongo version', pymongo.version

# connect to server
print '\nConnecting ...'
conn = MongoClient(server, port)

# Get the database
print '\nGetting database ...'
db = conn[db_name]

# Have to authenticate to get access
print '\nAuthenticating ...'
db.authenticate(username, password)


I am working on the file "SkyCoordAperturePhotometry.ipynb".
The first task is:
Annotate plot axis with coordinate positions and designations
Invoked by show_image()
Can you write Python code for it?

def annotate_image(axis, designation, xy, aperture_size):
    axis.plot(xy[0], xy[1], 'o', markersize=aperture_size,
              markeredgecolor='r', markerfacecolor='none',
              markeredgewidth=2)

    offsetbox = TextArea(designation, minimumdescent=False)

    ab = AnnotationBbox(offsetbox, xy,
                        xybox=(-20, 40+random()*10-10),
                        xycoords='data',
                        boxcoords="offset points",
                        arrowprops=dict(arrowstyle="->"))

    axis.add_artist(ab)


I am working on the file "SkyCoordAperturePhotometry.ipynb".
The first task is:
Aperture location sanity check by visual inspection
Arbitrarily choose the first G FITS file
Can you write Python code for it?

fits_file = fits_files[5]
print(fits_file)
hdus = fits.open(os.path.join(fits_root, fits_file))
image_data = hdus[data_index].data

median = np.median(image_data)
show_image(image_data, position_map, measurement_aperture, annotate=True, vmin=10, vmax=median*4)


I am working on the file "SkyCoordAperturePhotometry.ipynb".
The first task is:
Aperture photometry, yielding instrumental magnitudes
Can you write Python code for it?

# Create empty table with time and object headers
pd.options.display.float_format = '{:,.6f}'.format
instr_mag_df = pd.DataFrame()
names = [name for name in position_map]
instr_mag_df['name'] = [time_name] + names
instr_mag_df.set_index('name', inplace=True)

# Carry out photometry on B, G, R FITS files, yielding instrumental magnitudes
positions = position_map.values()

multi_file_photometry(fits_root, fits_files, data_index, positions, instr_mag_df,
                      measurement_aperture, inner_sky_annulus, outer_sky_annulus, gain)

# Save photometry table as CSV
instr_mag_csv_path = os.path.join(output_file_root, instr_mag_csv_file)
instr_mag_df.T.to_csv(instr_mag_csv_path)

# Display photometry table
instr_mag_df.T


I am working on the file "SkyCoordAperturePhotometry.ipynb".
The first task is:
Aperture radii and gain
Can you write Python code for it?

# FWHM (e.g. from PSF in IRIS)
fwhm = 6

# Aperture radii
measurement_aperture = 9
inner_sky_annulus = 12
outer_sky_annulus = 20

# ph/ADU
# Note: PythonPhot's aperture photometry function takes a phadu parameter.
# Assumption: this is photons/ADU or e-/ADU, i.e. gain.
gain=1.67


I am working on the file "SkyCoordAperturePhotometry.ipynb".
The first task is:
Compute standardised magnitude given target's instrumental magnitude, a numpy array of comparison star instrumental magnitudes and catalog comparison star magnitudes
Can you write Python code for it?

def standardised_magnitude(target_instr_mag, comp_instr_mags, catalog_comp_mags):
    deltas = target_instr_mag - comp_instr_mags
    mags = deltas + catalog_comp_mags
    return mags.mean()


I am working on the file "SkyCoordAperturePhotometry.ipynb".
The first task is:
Compute standardised magnitudes given a data frame of all instrumental magnitudes, a list of all star names, a list of row names of interest (e.g. stk-median-g*) in the instrumental magnitude data frame, and a dictionary of comparison star magnitudes
Can you write Python code for it?

def standardised_magnitudes(instr_mag_df_trans, star_names, row_names, catalog_mags):
    # exclude target star and check star to get list of possible comparison star names
    comp_names = star_names[2:]
    # obtain available comparison star names and magnitudes, ignoring any star not in catalog
    avail_comp_names = [name for name in comp_names if name in catalog_mags.keys()]
    avail_comp_mags = [catalog_mags[name] for name in comp_names if name in catalog_mags.keys()]
    target_name = star_names[0]
    #print(avail_comp_names, avail_comp_mags, target_name)
    std_mags = np.array([])
    for row_name in row_names:
        # get instrumental magnitudes for the current row of data and compute
        # standardised magnitude of the target star
        comp_instr_mags = [instr_mag_df_trans.loc[row_name][comp_name] for comp_name in avail_comp_names]
        target_mag = standardised_magnitude(instr_mag_df_trans.loc[row_name][target_name],
                                            np.array(comp_instr_mags),
                                            np.array(avail_comp_mags))
        # collect standardised magnitudes for each row
        std_mags = np.append(std_mags, target_mag)
    # TODO: also compute/return check star mags and look at std error; is that what spreadsheet uses?
    return std_mags


I am working on the file "SkyCoordAperturePhotometry.ipynb".
The first task is:
Convert RA and Dec to local coordinates
Can you write Python code for it?

position_map = get_local_coords(target_comp_ra_dec, wcs)
position_map


I am working on the file "SkyCoordAperturePhotometry.ipynb".
The first task is:
Differential Photometry and Standarised Magnitude
Can you write Python code for it?

b_row_names = [row_name for row_name in instr_mag_df.T.index if "-b" in row_name]
g_row_names = [row_name for row_name in instr_mag_df.T.index if "-g" in row_name]
r_row_names = [row_name for row_name in instr_mag_df.T.index if "-r" in row_name]

catalog_v_mags = get_ref_mags_for_band(target_comp_ra_dec, "V")
tg = standardised_magnitudes(instr_mag_df.T, names, g_row_names, catalog_v_mags)
tg.mean(), np.median(tg), tg.std()

catalog_b_mags = get_ref_mags_for_band(target_comp_ra_dec, "B")
tb = standardised_magnitudes(instr_mag_df.T, names, b_row_names, catalog_b_mags)
tb.mean(), np.median(tb), tb.std()

obscode = "BDJB"
cal_software = "IRIS"
target = names[0]
check = names[1]
airmass = "NA" # TODO: compute (look at AAVSO spreadsheet)
chart_id = "X15962DX"
comment = "Canon 1100D; 100mm; ISO 100; f2.0; 5 sec x 20 images median stacked in groups of 5"

jd = instr_mag_df.T.iloc[0]["JD"]
check_instr_b = instr_mag_df.T.loc[b_row_names][check].mean()
check_instr_g = instr_mag_df.T.loc[g_row_names][check].mean()

results = [(jd, tb.mean(), tb.std(), "TB", check_instr_b),
           (jd, tg.mean(), tg.std(), "TG", check_instr_g)]

webobs_path = os.path.join(output_file_root, webobs_file)
write_webobs_file(webobs_path, obscode, cal_software, target, check, airmass, results, chart_id, comment)

# Questions:
# - is mean or median best per T[BGR] row?
# - std() or some other std dev function (e.g. population vs sample)
# - how to compute R; use catalog B-V, V-R? may just want to report TG, TB
# - is there a role for linear regression here or only for transformation coefficients?
# - can/should we do airmass correction independent of transformation?


I am working on the file "SkyCoordAperturePhotometry.ipynb".
The first task is:
Extract all sources from image
Can you write Python code for it?

def extract_all_sources(fits_path, fwhm, source_snr=20):
    hdulist = fits.open(fits_path)
    data = hdulist[0].data.astype(float)
    header = hdulist[0].header
    wcs = WCS(header)
    bkg_sigma = mad_std(data)
    daofind = DAOStarFinder(fwhm=fwhm, threshold=source_snr*bkg_sigma)
    sources = daofind(data)
    return sources, wcs


I am working on the file "SkyCoordAperturePhotometry.ipynb".
The first task is:
Extract all sources from plate-solved image
Can you write Python code for it?

sources, wcs = extract_all_sources(wcs_file, fwhm=fwhm)
sources


I am working on the file "SkyCoordAperturePhotometry.ipynb".
The first task is:
Functions
RA and Dec for a list of stars
Can you write Python code for it?

def get_ra_and_dec(stars, maglimit):
    result = []
    for star in stars:
        vsp_template = 'https://www.aavso.org/apps/vsp/api/chart/?format=json&fov=10&star={}&maglimit={}'
        query = vsp_template.format(star, maglimit)
        record = requests.get(query).json()
        # assume that first element corresponds to the star
        if len(record["photometry"]) != 0 and record["photometry"][0]["auid"] == record["auid"]:
            bands = record["photometry"][0]["bands"]
        else:
            bands = []
        result.append({"star":record["star"], "ra":record["ra"], "dec":record["dec"], "bands":bands})
    return result


I am working on the file "SkyCoordAperturePhotometry.ipynb".
The first task is:
Inputs
Change these to suit your environment
File settings
Can you write Python code for it?

# Output file directory
output_file_root = "/Users/david/aavso/dslr-photometry/working"

# WebObs file
webobs_file = "webobs.csv"

# Instrumental magnitude output file path
instr_mag_csv_file = "instr_mags.csv"

# FITS file directory
fits_root = "/Users/david/aavso/dslr-photometry/working"

# Plate-solved FITS file name
wcs_file = "stk-median-g1-wcs.fit"

# B, G, and R FITS file prefixes to identify files,
# e.g. stk-median-g matches stk-median-g1.fit, stk-median-g2.fit, ...
fits_prefixes = ["stk-median-b", "stk-median-g", "stk-median-r"]

# FITS file data HDU index
data_index = 0

# Time column name
time_name = "JD"


I am working on the file "SkyCoordAperturePhotometry.ipynb".
The first task is:
Magnitude limit for comparison star lookups
Can you write Python code for it?


I am working on the file "SkyCoordAperturePhotometry.ipynb".
The first task is:
Names or AUIDs for target and comparison stars
Can you write Python code for it?


I am working on the file "SkyCoordAperturePhotometry.ipynb".
The first task is:
Photometry of a list of FITS files, creating a table of times and instrumental magnitudes
Can you write Python code for it?

def multi_file_photometry(fits_root, fits_files, data_index, coords, dataframe,
                          aperture_radius, inner_sky_radius, outer_sky_radius,
                          gain=1, zeropoint=0, suffix='.fit'):

    for fits_file in fits_files:
        fits_file_path = os.path.join(fits_root, fits_file)
        hdus = fits.open(fits_file_path)

        instr_mags = []
        for x, y in coords:
            time, mag = aperture_photometry(hdus[data_index], x, y,
                                            aperture_radius, inner_sky_radius, outer_sky_radius,
                                            gain, zeropoint)
            instr_mags.append(mag)

        dataframe[fits_file[0:fits_file.rindex(suffix)]] = [time] + instr_mags


I am working on the file "SkyCoordAperturePhotometry.ipynb".
The first task is:
Single image+coordinate photometry, returning a time and instrumental magnitude
Invoked by multi_file_photometry()
Can you write Python code for it?

def aperture_photometry(hdu, x, y,
                        aperture_radius, inner_sky_radius, outer_sky_radius,
                        gain, zeropoint):

    image_data = hdu.data
    time = hdu.header[time_name]

    mag, magerr, flux, fluxerr, sky, skyerr, badflag, outstr = \
                aper.aper(image_data, x, y, phpadu=gain,
                          apr=aperture_radius, zeropoint=zeropoint,
                          skyrad=[inner_sky_radius, outer_sky_radius],
                          exact=True)

    return time, mag[0]


I am working on the file "SkyCoordAperturePhotometry.ipynb".
The first task is:
Write AAVSO Extended Upload Format file suitable for upload to WebObs

the results parameter is a list of tuples containing jd, mag, mag_err, band, check, check_instr_mag for each photometry result
Can you write Python code for it?

def write_webobs_file(path, obscode, cal_software, target, check, airmass, results, chart_id, comment):
    header_template = """#TYPE=EXTENDED
#OBSCODE={0}
#SOFTWARE={1}, Python scripts, Jupyter notebook
#DELIM=,
#DATE=JD
#OBSTYPE=DSLR
#NAME,DATE,MAG,MERR,FILT,TRANS,MTYPE,CNAME,CMAG,KNAME,KMAG,AMASS,GROUP,CHART,NOTES
"""
    if type(airmass) is float:
        airmass = "{0:1.6f}".format(airmass)
    result_template = "{0},{1:1.6f},{2:1.6f},{3:1.6f},{4},NO,STD,ENSEMBLE,NA,{5},{6:1.6f},{7},NA,{8},{9}\n"
    with open(path, "w") as webobs:
        webobs.write(header_template.format(obscode, cal_software))
        for result in results:
            jd, mag, mag_err, band, check_instr_mag = result
            webobs.write(result_template.format(target, jd, mag, mag_err, band, check, check_instr_mag,
                                                airmass, chart_id, comment))


I am working on the file "Slide Notebooks/Product of consecutive numbers.ipynb".
The first task is:
Let us look at the right hand side of the equation first, i.e. $k^2 - 1$.
This can be rewritten as $\textbf{(k-1)*(k+1)}$
Now, this is where a hint lies.
What the right hand side means that it is a product of two integers ($k-1$ and $k+1$) which differ by 2.
We can see that this is the case:
$
\begin{equation}
(k+1) - (k-1) \
= k + 1 - k - (-1) \
= k - k + 1 - (-1) \
= 0 + 1 + 1 \
= 2 \
\end{equation}
$
So, if we can somehow show that the left hand side of the original equation, i.e. $n(n+1)(n+2)*(n+3)$:

can be represented as a product of two numbers which differ by 2, then we are done,
as these numbers can then be mapped to $k-1$ and $k+1$ for some $k \in \mathbb{Z}$.

We can group the numbers $\textbf{n, n+1, n+2, n+3}$ into pairs, with the hope of getting $k-1$ and $k+1$.
We can utilize following facts to choose the two pairs:

The difference of the products should be constant, and hence independent of $n$
Knowing that product of two factors of type $(n+i)(n+j) = n^2 + (i+j)n + i*j$,
We can observe that $i+j$ will be same for numbers which are equidistant from the middle of all numbers.

Now we can select our pair of numbers.

The first pair is $n$ and $(n+3)$,

and their product is $\textbf{n * (n+3)}$
which can be expanded as $\color{red}{\textbf{n^2 + 3n}}$

And, the second pair $(n+1)$ and $(n+2)$,

and their product is $\textbf{(n+1)*(n+2)}$
which can be expanded as $\color{red}{\textbf{n^2 + 3n}} + \textbf{2}$

Based on the above pairing we can immediately see that the difference of these pair products is as follows:
$
\begin{equation}
[(n+1)*(n+2)] - [n * (n+3)]\
= [\color{red}{n^2 + 3n} + 2] - [\color{red}{n^2 + 3n}]\
= n^2 + 3n + 2 - n^2 - 3n\
= (n^2 -n^2) + (3n - 3n) + 2\
= 0 + 0 + 2\
= 2
\end{equation}
$
Hence, based on the above simplification, we can map:

$(\color{red}{n^2 + 3n} + 2) \rightarrow (k+1)$, and
$(\color{red}{n^2 + 3n}) \rightarrow (k-1)$.

Now, if we choose $\color{blue}{\textbf{k = (n^2 + 3n + 1)}}$, the following equations hold:

$n^2 + 3n + 2 =...
Can you write Python code for it?


I am working on the file "Slide Notebooks/Product of consecutive numbers.ipynb".
The first task is:
More videos to come

Shubhanshu Mishra (shubhanshu.com)

![Twitter Follow](https://img.shields.io/twitter/follow/TheShubhanshu?style=social) ![YouTube Channel Subscribers](https://img.shields.io/youtube/channel/subscribers/UCZpSoW1pm0jk-jUaGwVWzLA?style=social)

Can you write Python code for it?

fig, ax = plt.subplots(1,3, figsize=(12, 6))
fig.patch.set_facecolor('white')

ax[0].plot("n1", "prod", "bo-", data=df)
ax[0].set_xlabel("n", fontsize=20)
ax[0].set_ylabel(f"$y = \prod_{{i=0}}^{{i={i_max}}} (n+i)$", fontsize=20)

ax[1].plot(df["k"], df["prod"], "ko-")
ax[1].set_xlabel("$k = \sqrt{y + 1}$", fontsize=20)
ax[1].set_title("$y = k^2 - 1$", fontsize=20)

ax[2].plot(df["n1"], df["k"], "ko-", label="$k = \sqrt{y + 1}$")
ax[2].plot(df["n1"], df["k = n^2 + 3n + 1"], "r--", label="$k = n^2 + 3n + 1$")
ax[2].legend(fontsize=14)

ax[2].set_ylabel("$k = \sqrt{y + 1}$", fontsize=20)
ax[2].set_xlabel("$n$", fontsize=20)

fig.suptitle(f"Product of 4 consecutive integers is 1 less than a perfect square.", fontsize=20)

fig.tight_layout()


I am working on the file "Slideshow.ipynb".
The first task is:
Now we'll initialize the TFT LCD display and clear it.
Can you write Python code for it?

disp = TFT.ILI9341(DC, rst=RST, spi=SPI.SpiDev(SPI_PORT,SPI_DEVICE,SPEED))
disp.begin()


I am working on the file "Slideshow.ipynb".
The first task is:
Now we'll start by invoking the GPIO class, which will identify our board and initialize the pins. We will use two pins for input for scrolling through the slideshow. We default to the spidev device at <code>/dev/spidev0.0</code> for the minnow
Additionally, the Data/Command and Reset pins are defined for the TFT LCD display.
Can you write Python code for it?

myGPIO = GPIO.get_platform_gpio()

myGPIO.setup(12,GPIO.IN)
myGPIO.setup(16,GPIO.IN)

lcd = ADA_LCD()
lcd.clear()

SPI_PORT = 0
SPI_DEVICE = 0
SPEED = 16000000
DC = 10
RST = 14


I am working on the file "Slideshow.ipynb".
The first task is:
The following functions collect all the images in the specified directory and place them into a list. It will filter out all the non-image files in the directory. It will fail if no images are found.
Can you write Python code for it?

imageList = []
rawList = os.listdir("/notebooks")
for i in range(0,len(rawList)):
    if (rawList[i].lower().endswith(('.png', '.jpg', '.jpeg', '.gif'))==True):
        imageList.append("/notebooks" + "/" + rawList[i])

if len(imageList)==0:
    print "No images found!"
    exit(1)

count = 0

print imageList


I am working on the file "Slideshow.ipynb".
The first task is:
This long infinite loop will work like so:
Clear the char LCD, write name of new image
Wait for a button press
Try to open an image
Display the image on the TFT LCD
--If we fail to open the file, print an error message to the LCD display--
----If we failed, open up the next file in the list. If we're at the end, restart at the beginning ----
Can you write Python code for it?

while True:

    lcd.clear()
    time.sleep(0.25)
    message = " Image " + str(count+1) + " of " + str(len(imageList)) + "\n" + imageList[count][len(sys.argv[1]):]
    lcd.message(message)
    lcd.scroll()
    try:
        image = Image.open(imageList[count])
    except(IOError):
        lcd.clear()
        time.sleep(0.25)
        message = " ERR: " + str(count+1) + " of " + str(len(imageList)) + "\n" + imageList[count][len(sys.argv[1]):]
        lcd.scroll()
        lcd.message(message)
        if(count == len(imageList)-1):
            image = Image.open(imageList[0])
        else:
            image = Image.open(imageList[count+1])

    image = image.rotate(90).resize((240, 320))
    disp.display(image)

    try:
        while True:
            if (myGPIO.input(12) != 1 and count != 0):
                count = count - 1
                break
            if (myGPIO.input(16) != 1 and count != len(imageList)-1):
                count = count + 1
                break
    except (KeyboardInterrupt):
        lcd.clear()
        lcd.message("Terminated")
        print
        exit(0)


I am working on the file "SlidingWindow1-Basics.ipynb".
The first task is:
Questions

Comment on the relationship between the eigenvalues and the extent (width) of the window.

When are the eigenvalues near each other? When are they not?

Comment on the change in geometry when the second sinusoid is incommensurate to the first. Specifically, comment on the intrinsic dimension of the object in the projection. Can you name the shape in the 3-D projection in the incommensurate case?

Try adding noise in like you did in the single frequency case.
Can you distinguish in the projection between the incommensurate case and the noisy, but harmonic one with second frequency 3? Explain.
What can you say about the eigenvalues in the two cases? Explain your answer.

It seems reasonable to ask what the ideal dimension to embed into is. While that question may be answerable, it would be better to bypass the question altogether. Similarly, it seems that beyond detecting the largest period, these tools are limited in detecting the secondary ones. Topological tools that we will see beginning in the next lab will allow us to make some progress toward that goal.
Power Spectrum
We saw above that for a rather subtle change in frequency changing the second sinusoid from harmonic to noncommensurate, there is a marked change in the geometry.  By contrast, the power spectral density functions are very close between the two, as shown below.  Hence, it appears that geometric tools are more appropriate for telling the difference between these two types of signals
Can you write Python code for it?

T = 20 #The period of the first sine in number of samples
NPeriods = 10 #How many periods to go through, relative to the faster sinusoid
N = T*NPeriods*3 #The total number of samples
t = np.arange(N) #Time indices

#Make the harmonic signal cos(t) + cos(3t)
xH = np.cos(2*np.pi*(1.0/T)*t) + np.cos(2*np.pi*(1.0/(3*T)*t))

#Make the incommensurate signal cos(t) + cos(pi*t)
xNC = np.cos(2*np.pi*(1.0/T)*t) + np.cos(2*np.pi*(1.0/(np.pi*T)*t))

plt.figure()
P1 = np.abs(np.fft.fft(xH))**2
P2 = np.abs(np.fft.fft(xNC))**2
plt.plot(np.arange(len(P1)), P1)
plt.plot(np.arange(len(P2)), P2)
plt.xlabel("Frequency Index")
plt.legend({"Harmonic", "Noncommensurate"})
plt.xlim([0, 50])
plt.show();


I am working on the file "SlidingWindow1-Basics.ipynb".
The first task is:
Questions

For fixed $\tau$:
What does varying the dimension do to the extent (the length of the window)?
what dimensions give eigenvalues nearest each other? (Note: dimensions! Plural!) Explain why this is the case. Explain how you might use this information to deduce the period of a signal.

What does varying $dT$ do to the PCA embedding? Explain this in terms of the definition of sliding windows above.

The command
python
np.random.randn(pts)
generates an array of length pts filled with random values drawn from a standard normal distribution ($\mu=0$, $\sigma=1$). Modify the code above to add random noise to signal.
Can you still detect the period visually by inspecting the plot of the signal?
Does your method of detecting the period from the first question still work?
How does adding noise change the geometry of the PCA embedding?
Modify the amplitude of the noise (for example, by multiplying the noise-generating command by a constant) and examine the 2D projection. What feature of the 2D projection appears to imply that the signal is periodic? At what noise amplitude does this feature appear to vanish?

Non-Periodic Signal Sliding Window
For a contrasting example, we will now examine the sliding window embedding of a non-periodic signal which is a linear function plus Gaussian noise.  The code below sets up the signal and then does the sliding window embedding, as before.
 Exercise:  Execute the code. Using the sliders, play around with the parameters of the sliding window embedding and examine the results.  Then answer the questions below.
Can you write Python code for it?

noise = 0.05*np.random.randn(400)

def on_value_change(change):
    execute_computation2()

dimslider = widgets.IntSlider(min=1,max=40,value=20,description='Dimension:',continuous_update=False)
dimslider.observe(on_value_change, names='value')

Tauslider = widgets.FloatSlider(min=0.1,max=5,step=0.1,value=1,description='Tau: ',continuous_update=False)
Tauslider.observe(on_value_change, names='value')

dTslider = widgets.FloatSlider(min=0.1,max=5,step=0.1,value=0.5,description='dT: ',continuous_update=False)
dTslider.observe(on_value_change, names='value')

display(widgets.HBox(( dimslider,Tauslider,dTslider)))

plt.figure(figsize=(9.5, 3))

def execute_computation2():
    plt.clf()
    # Step 1: Set up the signal
    x = np.arange(400)
    x = x/float(len(x))
    x = x + noise # Add some noise

    # Get slider values
    dim = dimslider.value
    Tau = Tauslider.value
    dT = dTslider.value

    #Step 2: Do a sliding window embedding
    X = getSlidingWindow(x, dim, Tau, dT)
    extent = Tau*dim

    #Step 3: Perform PCA down to 2D for visualization
    pca = PCA(n_components = 2)
    Y = pca.fit_transform(X)
    eigs = pca.explained_variance_
    print("lambda1 = %g, lambda2 = %g"%(eigs[0], eigs[1]))

    #Step 4: Plot original signal and PCA of the embedding
    gs = gridspec.GridSpec(1, 2)
    ax = plt.subplot(gs[0,0])
    ax.plot(x)
    ax.set_ylim((-2, 2))
    ax.set_title("Original Signal")
    ax.set_xlabel("Sample Number")
    yr = np.max(x)-np.min(x)
    yr = [np.min(x)-0.1*yr, np.max(x)+0.1*yr]
    ax.plot([extent, extent], yr, 'r')
    ax.plot([0, 0], yr, 'r')
    ax.plot([0, extent], [yr[0]]*2, 'r')
    ax.plot([0, extent], [yr[1]]*2, 'r')
    ax2 = plt.subplot(gs[0, 1])
    ax2.set_title("PCA of Sliding Window Embedding")
    ax2.scatter(Y[:, 0], Y[:, 1])
    ax2.set_aspect('equal', 'datalim')

execute_computation2()


I am working on the file "SlidingWindow1-Basics.ipynb".
The first task is:
Questions

Notice how changing the window extent doesn't have the same impact as it did in the periodic example above.  Why might this be?

Why is the second eigenvalue always tiny?

Multiple Sines Sliding Window
We will now go back to periodic signals, but this time we will increase the complexity by adding two sines together.  If the ratio between the two sinusoids is a rational number, then they are called harmonics of each other.  For example, $\sin(t)$ and $\sin(3t)$ are harmonics of each other. By contrast, if the ratio of the two frequencies is irrational, then the sinusoids are called incommensurate.  For example, $\sin(t)$ and $\sin(\pi t)$ are incommensurate.
The plots below are initialized with
$$f(t) = \sin(\omega t) + \sin(3\omega t),$$
a sum of two harmonics.
This time, the eigenvalues of PCA will be plotted (up to the first 10), in addition to the red line showing the extent of the window.  Also, 3D PCA will be displayed instead of 2D PCA, and you can click and drag your mouse to view it from different angles.  Colors will be drawn to indicate the position of the window in time, with cool colors (greens and yellows) indicating earlier windows and hot colors (oranges and reds) indicating later windows (using the "jet" colormap).
 Exercise:  Execute the code. Then play with the sliders, as well as the embedding dimension (note that for the 3-D projection, you can change the view by dragging around. Do!). Then, try changing the second sinusoid to be another multiple of the first. Try both harmonic and incommensurate values. Once you have gotten a feel for the geometries and the eigenvalues, answer the questions below.
Can you write Python code for it?

def on_value_change(change):
    execute_computation3()

embeddingdimbox = widgets.Dropdown(options=[2, 3],value=3,description='Embedding Dimension:',disabled=False)
embeddingdimbox.observe(on_value_change,names='value')

secondfreq = widgets.Dropdown(options=[2, 3, np.pi],value=3,description='Second Frequency:',disabled=False)
secondfreq.observe(on_value_change,names='value')

noiseampslider = widgets.FloatSlider(min=0,max=6,step=0.5,value=0,description='Noise Amplitude',continuous_update=False)
noiseampslider.observe(on_value_change, names='value')

dimslider = widgets.IntSlider(min=1,max=100,value=30,description='Dimension:',continuous_update=False)
dimslider.observe(on_value_change, names='value')

Tauslider = widgets.FloatSlider(min=0.1,max=5,step=0.1,value=1,description='Tau: ',continuous_update=False)
Tauslider.observe(on_value_change, names='value')

dTslider = widgets.FloatSlider(min=0.1,max=5,step=0.1,value=0.5,description='dT: ',continuous_update=False)
dTslider.observe(on_value_change, names='value')

display(widgets.HBox(( secondfreq,embeddingdimbox,noiseampslider)))
display(widgets.HBox((dimslider,Tauslider,dTslider)))

noise = np.random.randn(10000)

fig = plt.figure(figsize=(9.5, 4))
def execute_computation3():
    plt.clf()

    # Step 1: Setup the signal
    T1 = 20 # The period of the first sine in number of samples
    T2 = T1*secondfreq.value # The period of the second sine in number of samples
    NPeriods = 10 # How many periods to go through, relative to the second sinusoid
    N = T2*NPeriods # The total number of samples
    t = np.arange(N) # Time indices
    x = np.cos(2*np.pi*(1.0/T1)*t) # The first sinusoid
    x += np.cos(2*np.pi*(1.0/T2)*t) # Add the second sinusoid
    x += noiseampslider.value*noise[:len(x)]

    # Get widget values
    dim = dimslider.value
    Tau = Tauslider.value
    dT = dTslider.value
    embeddingdim = embeddingdimbox.value

    # Step 2: Do a sliding window embedding
    X = getSlidingWindow(x, dim, Tau, dT)
    extent = Tau*dim

    #...


I am working on the file "SlidingWindow1-Basics.ipynb".
The first task is:
Sliding Window Code
The code below performs a sliding window embedding on a 1D signal.  The parameters are as follows:
| | |
|:-:|---|
|$x$   | The 1-D signal (numpy array)   |
|dim|The dimension of the embedding|
|$\tau$   | The skip between samples in a given window  |
|$dT$   | The distance to slide from one window to the next  |
That is, along the signal given by the array $x$, the first three windows will will be $$\begin{bmatrix} x(\tau)\ x(2\tau) \ \ldots \ x((\mbox{dim}-1)\cdot\tau)\end{bmatrix},  \begin{bmatrix} x(dT + \tau)\ x(dT +2\tau) \ \ldots \ x(dT +(\mbox{dim}-1)\cdot\tau)\end{bmatrix},  \begin{bmatrix} x(2dT + \tau)\ x(2dT +2\tau) \ \ldots \ x(2dT +(\mbox{dim}-1)\cdot\tau)\end{bmatrix}$$
Spline interpolation is used to fill in information between signal samples, which is necessary for certain combinations of parameters, such as a non-integer $\tau$ or $dT$.
The function getSlidingWindow below creates an array $X$ containing the windows as its columns.
Can you write Python code for it?

def getSlidingWindow(x, dim, Tau, dT):
    """
    Return a sliding window of a time series,
    using arbitrary sampling.  Use linear interpolation
    to fill in values in windows not on the original grid
    Parameters
    ----------
    x: ndarray(N)
        The original time series
    dim: int
        Dimension of sliding window (number of lags+1)
    Tau: float
        Length between lags, in units of time series
    dT: float
        Length between windows, in units of time series
    Returns
    -------
    X: ndarray(N, dim)
        All sliding windows stacked up
    """
    N = len(x)
    NWindows = int(np.floor((N-dim*Tau)/dT))
    if NWindows <= 0:
        print("Error: Tau too large for signal extent")
        return np.zeros((3, dim))
    X = np.zeros((NWindows, dim))
    spl = InterpolatedUnivariateSpline(np.arange(N), x)
    for i in range(NWindows):
        idxx = dT*i + Tau*np.arange(dim)
        start = int(np.floor(idxx[0]))
        end = int(np.ceil(idxx[-1]))+2
        # Only take windows that are within range
        if end >= len(x):
            X = X[0:i, :]
            break
        X[i, :] = spl(idxx)
    return X


I am working on the file "SlidingWindow1-Basics.ipynb".
The first task is:
Sliding Window Result
We will now perform a sliding window embedding with various choices of parameters.  Principal component analysis will be performed to project the result down to 2D for visualization.
The first two eigenvalues computed by PCA will be printed.  The closer these eigenvalues are to each other, the rounder and more close to a circle the 2D projection of the embedding is. A red vertical line will be drawn to show the product of $\tau$ and the dimension, or "extent" (window length).
An important note: we choose to project the results to 2D (or later, to 3D). Nothing in particular tells us that this is the best choice of dimension. We merely make this choice to enable visualization. In general, when doing PCA, we want to choose enough eigenvalues to account for a significant portion of explained variance.
 Exercise:  Execute the code. Using the sliders, play around with the parameters of the sliding window embedding and examine the results.  Then answer the questions below.
Can you write Python code for it?

def on_value_change(change):
    execute_computation1()

dimslider = widgets.IntSlider(min=1,max=40,value=20,description='Dimension:',continuous_update=False)
dimslider.observe(on_value_change, names='value')

Tauslider = widgets.FloatSlider(min=0.1,max=5,step=0.1,value=1,description=r'\(\tau :\)' ,continuous_update=False)
Tauslider.observe(on_value_change, names='value')

dTslider = widgets.FloatSlider(min=0.1,max=5,step=0.1,value=0.5,description='dT: ',continuous_update=False)
dTslider.observe(on_value_change, names='value')

display(widgets.HBox(( dimslider,Tauslider,dTslider)))

plt.figure(figsize=(9.5, 3))
def execute_computation1():
    plt.clf()
    # Step 1: Setup the signal again in case x was lost
    T = 40 # The period in number of samples
    NPeriods = 4 # How many periods to go through
    N = T*NPeriods # The total number of samples
    t = np.linspace(0, 2*np.pi*NPeriods, N+1)[0:N] # Sampling indices in time
    x = np.cos(t)  # The final signal

    # Get slider values
    dim = dimslider.value
    Tau = Tauslider.value
    dT = dTslider.value

    #Step 2: Do a sliding window embedding
    X = getSlidingWindow(x, dim, Tau, dT)
    extent = Tau*dim

    #Step 3: Perform PCA down to 2D for visualization
    pca = PCA(n_components = 2)
    Y = pca.fit_transform(X)
    eigs = pca.explained_variance_
    print("lambda1 = %g, lambda2 = %g"%(eigs[0], eigs[1]))

    #Step 4: Plot original signal and PCA of the embedding
    ax = plt.subplot(121)
    ax.plot(x)
    ax.set_ylim((-2*max(x), 2*max(x)))
    ax.set_title("Original Signal")
    ax.set_xlabel("Sample Number")
    yr = np.max(x)-np.min(x)
    yr = [np.min(x)-0.1*yr, np.max(x)+0.1*yr]
    ax.plot([extent, extent], yr, 'r')
    ax.plot([0, 0], yr, 'r')
    ax.plot([0, extent], [yr[0]]*2, 'r')
    ax.plot([0, extent], [yr[1]]*2, 'r')
    ax2 = plt.subplot(122)
    ax2.set_title("PCA of Sliding Window Embedding")
    ax2.scatter(Y[:, 0], Y[:, 1])
    ax2.set_aspect('equal', 'datalim')
    plt.tight_layout()
   ...


I am working on the file "SlidingWindow2-PersistentHomology.ipynb".
The first task is:
Now, we will look at PCA of the sliding window embeddings of the two signals
Can you write Python code for it?

####g1
#Step 2: Do a sliding window embedding
dim = 20
Tau = 5
dT = 2
X1 = getSlidingWindow(g1, dim, Tau, dT)

#Step 3: Perform PCA down to 2D for visualization
pca = PCA()
Y = pca.fit_transform(X1)
eigs = pca.explained_variance_

c = plt.get_cmap('jet')
C = c(np.array(np.round(np.linspace(0, 255, Y.shape[0])), dtype=np.int32))
C = C[:, 0:3]

#Step 4: Plot original signal and PCA of the embedding
fig = plt.figure(figsize=(9.5,6))
ax = fig.add_subplot(221)
ax.plot(g1)
ax.set_title("Original Signal")
ax.set_xlabel("Sample Index")

ax2 = fig.add_subplot(222, projection = '3d')
ax2.set_title("g1 = %.2gcos(t) + %.2gcos(2t)"%(coeff1, coeff2))
ax2.scatter(Y[:, 0], Y[:, 1], Y[:, 2], c=C)
ax2.set_aspect('equal', 'datalim')

#####g2
X2 = getSlidingWindow(g2, dim, Tau, dT)
#Perform PCA down to 2D for visualization
pca = PCA()
Y = pca.fit_transform(X2)
eigs = pca.explained_variance_

ax = fig.add_subplot(223)
ax.plot(g2)
ax.set_title("Original Signal")
ax.set_xlabel("Sample Index")

ax2 = fig.add_subplot(224, projection = '3d')
ax2.set_title("g2 = %.2gcos(t) + %.2gcos(2t)"%(coeff2, coeff1))
ax2.scatter(Y[:, 0], Y[:, 1], Y[:, 2], c=C)
ax2.set_aspect('equal', 'datalim')
plt.tight_layout();


I am working on the file "SlidingWindow2-PersistentHomology.ipynb".
The first task is:
Persistent Homology
Now, we will compute the persistent homology of the above signal following a sliding window embedding.  Run this code. Then examine the outputs with both harmonic sinusoids and noncommensurate sinusoids, and note the difference in the persistence diagrams. Note that the two points with highest persistence are highlighted in red on the diagram.
Can you write Python code for it?

def on_value_change(change):
    execute_computation3()

secondfreq = widgets.Dropdown(options=[ 2, 3, np.pi],value=3,description='Second Frequency:',disabled=False)
secondfreq.observe(on_value_change,names='value')

noiseampslider = widgets.FloatSlider(min=0,max=2,step=0.1,value=0,description='Noise Amplitude',continuous_update=False)
noiseampslider.observe(on_value_change, names='value')

dimslider = widgets.IntSlider(min=1,max=100,value=20,description='Dimension:',continuous_update=False)
dimslider.observe(on_value_change, names='value')

Tauslider = widgets.FloatSlider(min=0.1,max=5,step=0.1,value=1,description=r'\(\tau :\)' ,continuous_update=False)
Tauslider.observe(on_value_change, names='value')

display(widgets.HBox(( dimslider,Tauslider)))
display(widgets.HBox(( secondfreq,noiseampslider)))

noise = np.random.randn(10000)

fig = plt.figure(figsize=(9.5, 5))

def execute_computation3():
    # Step 1: Setup the signal
    T1 = 10 # The period of the first sine in number of samples
    T2 = T1*secondfreq.value # The period of the second sine in number of samples
    NPeriods = 5 # How many periods to go through, relative to the second sinusoid
    N = T2*NPeriods # The total number of samples
    t = np.arange(N) # Time indices
    x = np.cos(2*np.pi*(1.0/T1)*t) # The first sinusoid
    x += np.cos(2*np.pi*(1.0/T2)*t) # The second sinusoid
    x += noiseampslider.value * noise[:len(x)]

    #Step 2: Do a sliding window embedding
    dim = dimslider.value
    Tau = Tauslider.value
    dT = 0.35
    X = getSlidingWindow(x, dim, Tau, dT)
    extent = Tau*dim

    #Step 3: Do Rips Filtration
    PDs = ripser(X, maxdim=1)['dgms']

    #Step 4: Perform PCA down to 2D for visualization
    pca = PCA()
    Y = pca.fit_transform(X)
    eigs = pca.explained_variance_

    #Step 5: Plot original signal and the persistence diagram
    gs = gridspec.GridSpec(3, 2,width_ratios=[1, 2],height_ratios=[2,2,1])

    ax = plt.subplot(gs[0,1])
    ax.plot(x)
    ax.set_ylim((1.25*min(x), 1.25*max(x)))
   ...


I am working on the file "SlidingWindow2-PersistentHomology.ipynb".
The first task is:
Questions

Examining the effect of window extent on maximal persistence (with no noise):
Describe the effect of dimension and $\tau$ on maximal persistence. What does increasing one of these factors while keeping the other constant to do the window extent? What does it do to the maximal persistence? Explain your observations.
Is the maximal persistence a function of the window extent? Justify your answer and explain it geometrically (you may want to refer to the PCA projection plots).
Describe the relation between the eigenvalues and the maximal persistence. (Hint: How do the eigenvalues affect roundness? How does roundness affect persistence?)
Write code to plot scatter plots of maximal persistence vs dimension for fixed $\tau$ and vs $\tau$ for fixed dimension, and maximal persistence vs. dimension for fixed extent (say, extent = 40). Comment on your results.

Now add some noise to your plots. Notice that the maximal persistence point on the persistence diagram is colored in red.
What do you observe regarding the persistence diagram? Explain your observations in terms of your understading of persistence.
At what noise amplitude does the point with maximal persistence appear to get 'swallowed up' by the noise in the diagram? How does this correspond with the 2-D projection?
Note that the original signal has amplitude 1. As you increase noise, is it clear by looking at the signal that there is a periodic function underlying it? How does persistence allow detection of periodicity? Explain.
For fixed noise amplitude (say 1), increase dimension. What effect does this have on detection of periodicity using your method?
Does varying $\tau$ for the same fixed amplitude have the same effect? Explain.

Two Sines
Now let's examine the persistent homology of a signal consisting of the sum of two sinusoids.  First, setup and examine the signal.  We will use a slightly coarser sampling rate than we did in the first module to keep the persistent homology code running quickly.
Can you write Python code for it?

# Step 1: Setup the signal
T1 = 10 # The period of the first sine in number of samples
T2 = T1*3 # The period of the second sine in number of samples
NPeriods = 10 # How many periods to go through, relative to the second sinusoid
N = T2*NPeriods # The total number of samples
t = np.arange(N) # Time indices
x = np.cos(2*np.pi*(1.0/T1)*t) # The first sinusoid
x += np.cos(2*np.pi*(1.0/T2)*t) # The second sinusoid
plt.figure();
plt.plot(x);


I am working on the file "SlidingWindow2-PersistentHomology.ipynb".
The first task is:
Single Sine: Maximum Persistence vs Window Size
First, let's examine the 1D persistent homology of the sliding window embedding of a single perfect sinusoid.  Choose dim and $\tau$ to change the extent (window size) to different values, and examine how the maximum persistence changes.  How does this support what you saw in the first module?
Can you write Python code for it?

def on_value_change(change):
    execute_computation1()

dimslider = widgets.IntSlider(min=1,max=100,value=20,description='Dimension:',continuous_update=False)
dimslider.observe(on_value_change, names='value')

Tauslider = widgets.FloatSlider(min=0.1,max=5,step=0.1,value=1,description=r'\(\tau :\)' ,continuous_update=False)
Tauslider.observe(on_value_change, names='value')

noiseampslider = widgets.FloatSlider(min=0,max=2,step=0.1,value=0,description='Noise Amplitude',continuous_update=False)
noiseampslider.observe(on_value_change, names='value')

display(widgets.HBox(( dimslider,Tauslider, noiseampslider)))

noise = np.random.randn(10000)

fig = plt.figure(figsize=(9.5, 4))

def execute_computation1():
    plt.clf()
    # Step 1: Setup the signal
    T = 40 # The period in number of samples
    NPeriods = 4 # How many periods to go through
    N = T*NPeriods # The total number of samples
    t = np.linspace(0, 2*np.pi*NPeriods, N+1)[0:N] # Sampling indices in time
    x = np.cos(t) # The final signal
    x += noiseampslider.value * noise[:len(x)]

    # Step 2: Do a sliding window embedding
    dim = dimslider.value
    Tau = Tauslider.value
    dT = 0.5
    X = getSlidingWindow(x, dim, Tau, dT)
    extent = Tau*dim

    # Step 3: Do Rips Filtration
    PDs = ripser(X, maxdim=1)['dgms']
    I = PDs[1]

    # Step 4: Perform PCA down to 2D for visualization
    pca = PCA(n_components = 2)
    Y = pca.fit_transform(X)
    eigs = pca.explained_variance_

    # Step 5: Plot original signal, 2-D projection, and the persistence diagram
    gs = gridspec.GridSpec(2, 2)
    ax = plt.subplot(gs[0,0])
    ax.plot(x)
    ax.set_ylim((2*min(x), 2*max(x)))
    ax.set_title("Original Signal")
    ax.set_xlabel("Sample Number")
    yr = np.max(x)-np.min(x)
    yr = [np.min(x)-0.1*yr, np.max(x)+0.1*yr]
    ax.plot([extent, extent], yr, 'r')
    ax.plot([0, 0], yr, 'r')
    ax.plot([0, extent], [yr[0]]*2, 'r')
    ax.plot([0, extent], [yr[1]]*2, 'r')

    ax2 = plt.subplot(gs[1,0])
    plot_diagrams(PDs)
   ...


I am working on the file "SlidingWindow3-AudioApplications.ipynb".
The first task is:

Now let's do a sliding window with a window length equal to the sample rate over 2, corresponding to the fact that a beat period is a half of a second in this song.  We will have to have a large Tau and dT, since there is such a high sampling rate, because otherwise the TDA code will grind to a halt with way too many points.  We will also need to set up special sliding window code that skips the spline interpolation step, because this will also be prohibitively slow at this sampling rate.  In other words, we will assume that Tau and dT are integers.  Here's the code that does all of this on the first three seconds of audio:
Can you write Python code for it?

#Sliding window code here assumes integer x, dim, and Tau so no interpolation
#is needed (for computational efficiency)
def getSlidingWindowInteger(x, dim, Tau, dT):
    N = len(x)
    NWindows = int(np.floor((N-dim*Tau)/dT)) #The number of windows
    if NWindows <= 0:
        print("Error: Tau too large for signal extent")
        return np.zeros((3, dim))
    X = np.zeros((NWindows, dim)) #Create a 2D array which will store all windows
    idx = np.arange(N)
    for i in range(NWindows):
        #Figure out the indices of the samples in this window
        idxx = np.array(dT*i + Tau*np.arange(dim), dtype=np.int32)
        X[i, :] = x[idxx]
    return X

#Note that dim*Tau here spans a half a second of audio,
#since Fs is the sample rate
dim = round(Fs/200)
Tau = 100
dT = Fs/100
Y = getSlidingWindowInteger(X[0:Fs*3], dim, Tau, dT)
print("Y.shape = ", Y.shape)
#Mean-center and normalize
Y = Y - np.mean(Y, 1)[:, None]
Y = Y/np.sqrt(np.sum(Y**2, 1))[:, None]

PDs = ripser(Y, maxdim=1)['dgms']
pca = PCA()
Z = pca.fit_transform(Y)

plt.figure(figsize=(8, 4))
plt.subplot(121)
plt.title("2D PCA")
plt.scatter(Z[:, 0], Z[:, 1])
plt.subplot(122)
plot_diagrams(PDs)
plt.title("Persistence Diagram")
plt.show()


I am working on the file "SlidingWindow3-AudioApplications.ipynb".
The first task is:

Biphonation Overview

Biphonation refers to the presence of two or more simultaneous frequencies in a signal which are "incommensurate"; that is, their frequencies are linearly independent over the rational numbers.  In other words, the frequencies are "inharmonic."  We saw a synthetic example in class 1 of cos(x) + cos(pi x).  Today, we will examine how this manifests itself in biology with horse whinnies that occur during states of high emotional valence.  During the steady state of a horse whinnie, biphonation is found

Figure 1: Audio of a horse whinnie.  Courtesy of http://www.nature.com/articles/srep09989#s1

Biphonation Example with Horse Whinnies

Let's now load the audio from the horse whinnie example, interactively plot the audio waveform, and listen to it.
Can you write Python code for it?

#Read in the audio file.  Fs is the sample rate, and
#X is the audio signal
Fs, X = scipy.io.wavfile.read("horsewhinnie.wav")
plt.figure()
plt.plot(np.arange(len(X))/float(Fs), X)
plt.xlabel("Time (Seconds)")
plt.title("Horse Whinnie Waveform")
plt.show()

from IPython.display import Audio
# load a remote WAV file
Audio('horsewhinnie.wav')


I am working on the file "SlidingWindow3-AudioApplications.ipynb".
The first task is:

Music Analysis

Music is full of repetition.  For instance, there is usually a hierarchy of rhythm which determines how the music "pulses," or repeates itself in beat patterns.  Often, a dominant rhythm level is deemed the "tempo" of the music.  Typical tempos range from about 50 beats per minute to 200 beats per minute.  Let's take a moderate tempo level of 120 beats per minute, for instance, which occurs in the song "Don't Stop Believin'" by Journey.  This corresponds to a period of 0.5 seconds.  As we saw in the horse example, sound is sampled at 44100 samples per second.  This corresponds to an ideal sliding window interval length of 22050.  Let's try to compute the sliding window embedding of the raw audio to see if the tempo manifests itself with TDA.  First, we will load in "Don't Stop Believing" below:
Can you write Python code for it?

Fs, X = scipy.io.wavfile.read("journey.wav") #Don't Stop Believing
X = X/(2.0**15) #Loaded in as 16 bit shorts, convert to float
plt.figure()
plt.plot(np.arange(len(X))/float(Fs), X)
plt.xlabel("Time (Seconds)")
plt.title("Don't Stop Believin")
plt.show()

Audio('journey.wav')


I am working on the file "SlidingWindow3-AudioApplications.ipynb".
The first task is:

Speech Example

In our final experiment in this module, we will look at a sliding window embedding of the audio novelty function on a speech excerpt which does not have a clear rhythmic structure (courtesy of http://marsyasweb.appspot.com/download/data_sets/).  Click on the cell below to load the speech audio, and click on the cell below that to run the sliding window embedding + persistent homology
Can you write Python code for it?

#Read in the audio file.  Fs is the sample rate, and
#X is the audio signal
Fs, X = scipy.io.wavfile.read("speech.wav")
X = X/(2.0**15)
(S, novFn) = getAudioNoveltyFn(X, Fs, winSize, hopSize)
plt.figure()
plt.plot(np.arange(len(novFn))*hopSize/float(Fs), novFn)
plt.xlabel("Time (Seconds)")
plt.title("Audio Novelty Function for Speech")
Audio('speech.wav')
plt.show()

#Get the novelty function for the first three seconds, and use the
#exact same parameters as before
novFn = novFn[0:int((Fs/hopSize)*3)]
dim = 20
Tau = (Fs/2)/(float(hopSize)*dim)
dT = 1
Y = getSlidingWindowInteger(novFn, dim, Tau, dT)
print("Y.shape = ", Y.shape)
#Mean-center and normalize
Y = Y - np.mean(Y, 1)[:, None]
Y = Y/np.sqrt(np.sum(Y**2, 1))[:, None]

PDs = ripser(Y, maxdim=1)['dgms']
pca = PCA()
Z = pca.fit_transform(Y)

plt.figure(figsize=(12, 6))
plt.subplot(121)
plt.title("2D PCA")
plt.scatter(Z[:, 0], Z[:, 1])
plt.subplot(122)
plot_diagrams(PDs[1], labels=['H1'])
plt.title("Persistence Diagram")
plt.show()


I am working on the file "SlidingWindow3-AudioApplications.ipynb".
The first task is:
You might notice that there are vertical streaks in a semi-periodic pattern.  These correspond to "broadband percussive events," or, on other words, likely onsets for beats when drums occur.  An audio novelty function is derived from a spectrogram by looking at the difference between successive frames to try to pick up on this.  The code below extracts the audio novelty function and displays it for the same audio snippet.
Can you write Python code for it?

plt.figure(figsize=(8, 4))
#Plot the spectrogram again
plt.subplot(211)
plt.imshow(np.log(S.T), cmap = 'afmhot', aspect = 'auto')
plt.ylabel('Frequency Bin')
plt.title('Log-frequency power spectrogram')

#Plot the audio novelty function
plt.subplot(212)
plt.plot(np.arange(len(novFn))*hopSize/float(Fs), novFn)
plt.xlabel("Time (Seconds)")
plt.ylabel('Audio Novelty')
plt.xlim([0, len(novFn)*float(hopSize)/Fs])
plt.show()