I am working on the file "20170510_RunningABokehServerOnHerokuPart1.ipynb".
The first task is:
Next, the base-level functions which will interact with the data are defined. Typically, these type of functions do not need to be adjusted on a regular basis, but if a bug is found or the algorithm needs to be changed, it's important to keep all the functions that interact with data in the same place. This code in particular will create a random data set, and fit it using a linear regression model. (This is the first part of the "Controller" in a Model-View-Controller).
Can you write Python code for it?

###-----------------------------------------------------------------------###
###-----------------------BASE-LEVEL FUNCTIONS----------------------------###
### This section contains the low-level calculation for the inspection    ###
### intervals. ("Controller" Part 1)                                      ###
###-----------------------------------------------------------------------###
def create_data(n_samp, bias, noise):
    # Creates a set of random data based on user parameters
    data = make_regression(n_samp, 1, 1, 1, bias, noise=noise)
    return data

def fit_data(data):
    # Uses linear regression to find the coefficients and intercept
    x = data[0]
    y = data[1]
    l_model = LR()
    l_model.fit(x, y)
    return l_model.coef_, l_model.intercept_


I am working on the file "20170510_RunningABokehServerOnHerokuPart1.ipynb".
The first task is:
The "Callbacks" section is the second piece of the "Controller" part of the Model-View-Controller. It defines what happens when the user controls are interacted with. In this simple case, there are two buttons in the UI which may be clicked. If the "Simulate" button is clicked, data will be generated based on the user controls, and it will be fitted to a line. Then, the data sources and plot ranges will be updated accordingly. If the "Clear" button is clicked, the data in the plot will be removed by setting the source data arrays to []. Finally, the last two lines actually define which function to execute when the button is clicked (e.g. plot_sim.on_click(update_plot)).
Can you write Python code for it?

###-----------------------------------------------------------------------###
###----------------------------CALLBACKS----------------------------------###
### This section defines the behavior of the GUI as the user interacts    ###
### with the controls. ("Controller" Part 2)                              ###
###-----------------------------------------------------------------------###
# Behavior when the "Simulate" button is clicked
def update_plot():
    # Pull the parameters from the controls
    num_samples = ctl_nsamp.value
    bias = ctl_bias.value
    noise = ctl_noise.value
    # Generate and fit the data
    data = create_data(num_samples, bias, noise)
    coef, inter = fit_data(data)
    # Find the min and maxes of the data
    x_min = min(data[0])[0]
    x_max = max(data[0])[0]
    y_min = min(data[1])
    y_max = max(data[1])
    # Find the regression line
    x = [x_min, x_max]
    y = [x_min*coef+inter, x_max*coef+inter]
    # Update the data sources
    source_data.data = dict(x=data[0], y=data[1])
    source_line.data = dict(x=x, y=y)
    # Update the data ranges
    xrange_data.start = x_min
    xrange_data.end = x_max
    yrange_data.start = y_min
    yrange_data.end = y_max

# Behavior when the "Clear" button is clicked
def clear_plot():
    source_data.data = dict(x=[], y=[])

# Button callbacks, using the above functions
plot_sim.on_click(update_plot)
plot_clear.on_click(clear_plot)


I am working on the file "20170510_RunningABokehServerOnHerokuPart1.ipynb".
The first task is:
The final section of the Bokeh server is a simple page layout. Calling the "add_root" method will define the controls that should show up on the main page. Note that this code must typically come at the end, because it requires some of the objects to be defined above.
Can you write Python code for it?


I am working on the file "20170510_RunningABokehServerOnHerokuPart1.ipynb".
The first task is:
The following section handles the "Model" (e.g. data) sources in the Bokeh server. First, I like to generate some example data to initialize the plots. This initialization serves two purposes: it allows preliminary validation of the results which can be immediately tested, and it shows the user exactly what the visualization will look like when the controls are interacted with.
If you're not familiar with data sources in Bokeh, they are required to update the plot in real-time. If a data source is not defined, the plot can only contain the data that it was initialized with. The four data sources below (source_data, source_line, xrange_data, and yrange_data) will be updated in the next section, "Callbacks".
Can you write Python code for it?

###-----------------------------------------------------------------------###
###------------------DATA SOURCES AND INITIALIZATION----------------------###
### This section defines the data sources which will be used in the Bokeh ###
### plots. To update a Bokeh plot in the server, each of the sources will ###
### be modified in the CALLBACKS section. ("Model")                       ###
###-----------------------------------------------------------------------###
# Generating some initial data for the plots, based on the parameter defaults
d_data = create_data(d_nsamp, d_bias, d_noise)
d_coef, d_int = fit_data(d_data)
# Find the minimum and maximum values of the data, for ranges, etc.
# d_data[0] is the "X" data, and d_data[1] is the "Y" data
# Remember that the "X" data is of size [n,1] while the "Y" data is [n]
# That's the reason for the extra [0] in the "X" data below
d_x_min = min(d_data[0])[0]
d_x_max = max(d_data[0])[0]
d_y_min = min(d_data[1])
d_y_max = max(d_data[1])
# Find the default regression line
d_x = [d_x_min, d_x_max]
d_y = [d_x_min*d_coef+d_int, d_x_max*d_coef+d_int]
# Defining the Bokeh data sources
source_data = ColumnDataSource(data=dict(x=d_data[0], y=d_data[1]))
source_line = ColumnDataSource(data=dict(x=d_x, y=d_y))
# Associating the sources with plots
plot_data.scatter('x', 'y', source=source_data)
plot_data.line('x', 'y', source=source_line, line_color='orange')
# Defining the plot ranges
xrange_data = Range1d(bounds=[None, None], start=d_x_min, end=d_x_max)
yrange_data = Range1d(bounds=[None, None], start=d_y_min, end=d_y_max)
# Associating the ranges with plots
plot_data.x_range = xrange_data
plot_data.y_range = yrange_data


I am working on the file "20170510_RunningABokehServerOnHerokuPart1.ipynb".
The first task is:
The next section defines the Bokeh controls that will be present in the GUI. This includes: a plot of the data, the "simulate" and "clear" buttons, and the sliders for the parameters (number of samples, bias, and noise). Note that the defaults and ranges for the controls were defined in the previous section, so this section should not need modification unless new controls are being added. (This is the second part of the "View" in a Model-View-Controller).
Can you write Python code for it?

###-----------------------------------------------------------------------###
###----------------------GRAPHICAL USER INTERFACE-------------------------###
### This code defines the Bokeh controls that are used for the user       ###
### interface. All the defaults for the controls are above. This code     ###
### should not need to be modified. ("View" Part 2)                       ###
###-----------------------------------------------------------------------###
# Plot- Regression Data
plot_data = figure(plot_height=300, plot_width=600,
                   title="Regression Data", toolbar_location="above",
                   x_axis_label='X', y_axis_label='Y',
                   tools="pan,save,box_zoom,wheel_zoom")
# Plot Control Buttons
plot_sim = Button(label="Simulate")
plot_clear = Button(label="Clear")
plot_ctls = column(plot_sim, plot_clear)
# Main Control Buttons
ctl_title = Div(text="
Parameters
")
ctl_nsamp = Slider(title="Number of Samples", value=d_nsamp,
                  start=r_nsamp[0], end=r_nsamp[1], step=r_nsamp[2])
ctl_bias = Slider(title="Bias", value=d_bias,
                  start=r_bias[0], end=r_bias[1], step=r_bias[2])
ctl_noise = Slider(title="Noise", value=d_noise,
                  start=r_noise[0], end=r_noise[1], step=r_noise[2])
ctl_inputs = widgetbox(ctl_title, ctl_nsamp, ctl_bias, ctl_noise)


I am working on the file "20170515-getting-First-and-last-date-of-tweets-in-a-collection/20170515-getting-First-and-last-date-of-tweets-in-a-collection.ipynb".
The first task is:
2) Extracting "UserID, screen name, date created" from the input data
Can you write Python code for it?

# header
!echo "[]" | jq -r '["tweet_created_at","userID", "screen_name", "user_created_at"] | @csv' > "csvdata.csv"
!cat $INPUTFILE | jq -r '[(.created_at | strptime("%A %B %d %T %z %Y") | todate), .user.id_str, .user.screen_name, (.user.created_at | strptime("%A %B %d %T %z %Y") | todate)] | @csv' >> "csvdata.csv"
!head -5 "csvdata.csv"


I am working on the file "20170515-getting-First-and-last-date-of-tweets-in-a-collection/20170515-getting-First-and-last-date-of-tweets-in-a-collection.ipynb".
The first task is:
3) Getting First_tweet_date and Last_tweet_date for each user
Can you write Python code for it?

import pandas as pd

data = pd.read_csv("csvdata.csv", encoding = 'ISO-8859-1')
data2 = data.groupby(['userID', 'screen_name', 'user_created_at']).tweet_created_at.agg(['min', 'max'])
data3 = data2.reset_index()
data3.rename(columns={'min': 'first_tweet_date', 'max': 'last_tweet_date'}, inplace=True)
data3.head(5)

# the number of unique users
len(data3)


I am working on the file "20170515-getting-First-and-last-date-of-tweets-in-a-collection/20170515-getting-First-and-last-date-of-tweets-in-a-collection.ipynb".
The first task is:
4) Export the results to a csv file
Can you write Python code for it?

# Export the results to a csv file whose filename is OUTPUTFILE set by user in the beginning of thie notebook.
data3.to_csv(OUTPUTFILE, index=False)


I am working on the file "20170520_RunningABokehServerOnHerokuPart2.ipynb".
The first task is:
Finally, a file called "Procfile" tells Heroku how to run our files as a web application. It's essential that the "P" in "Procfile" be capitalized, and that the file does not have an extension. These are two common errors when defining a Procfile.
Our Procfile will contain the following code:
Can you write Python code for it?

web: bokeh serve --port=$PORT --num-procs=0 --host=linregex.herokuapp.com --address=0.0.0.0 --use-xheaders linear_example.py


I am working on the file "20170522-cohpy-ascii-versus-repr.ipynb".
The first task is:
For basic data types,
ascii makes a string suitable for pasting into source code,
same as repr.
Can you write Python code for it?


I am working on the file "20170526_FastFourierTransformInPython.ipynb".
The first task is:
In the figure above, the primary peak can be seen at 60 Hz (rotating frequency of the equipment), while the secondary peak is at 270 Hz, as expected (ball pass frequency).
What if the bearing had a fault? If so, the magnitude of the peak at 270 Hz would be expected to increase, as in the following figure:
Can you write Python code for it?

# Time-Domain Plot
peak2_mag2 = 8 # magnitude of the peak
vib_data2 = peak1_mag*np.sin(2*pi*peak1_hz*time) + \
            peak2_mag2*np.sin(2*pi*peak2_hz*time) + \
            np.random.normal(0, noise_mag, N)
plt.figure()
plt.plot(time[0:100], vib_data2[0:100])
plt.xlabel('Time')
plt.ylabel('Vibration (g)')
plt.title('Time Domain (Faulted Bearing)')

# Frequency-Domain Plot
yr2 = fft(vib_data2) # "raw" FFT with both + and - frequencies
y2 = 2/N * np.abs(yr2[0:np.int(N/2)]) # positive freqs only
plt.figure()
plt.plot(x, y2)
plt.xlabel('Frequency (Hz)')
plt.ylabel('Vibration (g)')
plt.title('Frequency Domain (Faulted Bearing)');


I am working on the file "20170526_FastFourierTransformInPython.ipynb".
The first task is:
Notice that the 60 Hz component in the time-domain signal is slightly less recognizable, and the magnitude of the 270 Hz peak in the frequency-domain has increased significantly.
For monitoring the bearing condition, the maximum "narrowband" ball pass frequency (e.g. the band of frequencies between 260 Hz and 280 Hz) will be trended over time. If the magnitude of the peak deviates significantly from the magnitude under healthy machinery operation (e.g. increasing beyond the 1% confidence interval of the distribution of the peak magnitudes under normal operation), a bearing failure is likely imminent (diagnostics). If these vibration measurements are fairly noise-free and trendable over time, the failure of the bearing can be predicted in advance (prognostics) by determining the amount of time until the bearing reaches a predefined fault threshold.
Finally, what would happen if our sampling rate had been lower than the Nyquist Sampling Theorem? Let's reduce the sampling rate to 500 Hz and find out:
Can you write Python code for it?

# Reduced sampling frequency
sampling_rate3 = 500 # Hz
N3 = (end_time - start_time)*sampling_rate3 # array size
time3 = np.linspace(start_time, end_time, N3)
vib_data3 = peak1_mag*np.sin(2*pi*peak1_hz*time3) + \
           peak2_mag*np.sin(2*pi*peak2_hz*time3) + \
           np.random.normal(0, noise_mag, N3)
T3 = 1/sampling_rate3 # inverse of the sampling rate
x3 = np.linspace(0.0, 1.0/(2.0*T3), int(N3/2))

# FFT algorithm
yr3 = fft(vib_data3) # "raw" FFT with both + and - frequencies
y3 = 2/N3 * np.abs(yr3[0:np.int(N3/2)]) # positive freqs only

# Plotting the results
plt.plot(x3, y3)
plt.xlabel('Frequency (Hz)')
plt.ylabel('Vibration (g)')
plt.title('Frequency Domain (Aliasing)');


I am working on the file "20170526_FastFourierTransformInPython.ipynb".
The first task is:
Notice that the sinusoidal shape is fairly recognizable; this effect occurs because the 60 Hz peak has a high magnitude compared to the 270 Hz peak, and the amount of added Gaussian noise is minimal. The peak-to-peak measurement in the figure is greater than 0.015 seconds but less than 0.02 seconds; in fact, it's equal to 1/60 Hz, or about 0.017 seconds.
Both numpy and scipy contain functions for evaluating a Discrete Fourier Transform. We will be using the scipy.fftpack.ftt function in this example. This function will use the time-domain vibration data we created above to generate frequency-domain data.
According to the Nyquist-Shannon Sampling Theorem, beyond the scope of this blog post, we can only examine frequency-spectrum components up to the sampling rate divided by two. Increasing the sampling rate will widen the available frequency spectrum (we can look at higher frequencies); increasing the number of data samples will increase the resolution of the FFT (generally, the frequency spectrum is more accurate). Our sampling rate was 1000 Hz, so we can safely examine frequencies up to 500 Hz, which is greater than the largest frequency of interest (270 Hz).
The following code will generate the frequency spectrum of the vibration data using the scipy.fftpack.ftt function. The fft function will generate both the "positive-frequency" and "negative-frequency" parts of the spectrum (see here for more information), so we
Can you write Python code for it?

from scipy.fftpack import fft

# Nyquist Sampling Criteria
T = 1/sampling_rate # inverse of the sampling rate
x = np.linspace(0.0, 1.0/(2.0*T), int(N/2))

# FFT algorithm
yr = fft(vib_data) # "raw" FFT with both + and - frequencies
y = 2/N * np.abs(yr[0:np.int(N/2)]) # positive freqs only

# Plotting the results
plt.plot(x, y)
plt.xlabel('Frequency (Hz)')
plt.ylabel('Vibration (g)')
plt.title('Frequency Domain (Healthy Machinery)');


I am working on the file "20170615-dojo-days-of-months.ipynb".
The first task is:
The maximum number of days in the above output
matches that output by days_spanned.sh.
Now to add the stuff that keeps track of which
starting months can yield those maximum number of days.
Can you write Python code for it?

from collections import defaultdict

def max_n_days_for_months():
    """Yield tuples of
        number of consecutive months,
        maximum number of days for those consecutive months
        and list of starting months which produce that above maximum
    for all numbers of consecutive months up to a year."""
    for n_months in range(1, MONTHS_PER_YEAR+1):
        d = defaultdict(list)
        for starting_month in range(MONTHS_PER_YEAR):
            n_days = max_days_n_months(n_months, starting_month)
            d[n_days].append(starting_month)
        max_n_days = max(d)
        yield n_months, max_n_days, sorted(d[max_n_days])

# %timeit list(max_n_days_for_months())
list(max_n_days_for_months())

def pretty():
    """Yields lines to be absolutely identical
    to that from days_spanned.sh."""
    for selector, name in ((min, '-gt'), (max, '-ge')):
        yield f'Compare: {name} '
        for n_months, max_n_days, months in max_n_days_for_months():
            month = selector(months) + 1
            if n_months == 1:
                yield f'month: {month} month spans: {max_n_days} days  '
            elif n_months == 2:
                yield f'month: {month} plus following month spans: {max_n_days} days  '
            else:
                yield f'month: {month} plus following {n_months - 1} months spans: {max_n_days} days  '
        yield ''

for line in pretty():
    print(line)


I am working on the file "20170615-splitting-data.ipynb".
The first task is:
Tuple unpacking is nice and in this case
enables
DRY
(aka Single Source of Truth)
code. Compare:
month, n_days = month_n_days.split(':')

with
FUTLHS=`echo "$j" | awk -F: {'print $1'}`
FUTRHS=`echo "$j" | awk -F: {'print $2'}`

One can do all the chopping up front.
The chopping code is less readable,
but the subsequent for loop is just pretty.
Can you write Python code for it?

MONTH_NDAYS = [list(map(int, s.split(':'))) for s in '''
    0:31
    1:29
    2:31
    3:30
    4:31
    5:30
    6:31
    7:31
    8:30
    9:31
    10:30
    11:31
'''.split()]

MONTH_NDAYS

for month, n_days in MONTH_NDAYS:
    print(f'{month} has {n_days}')


I am working on the file "20170616_PlotlyDashRegressionExample.ipynb".
The first task is:
As one of the first things that will be seen when the code is opened, a parameter defaults section allows the controls to be easily changed, without digging too deeply into the code. In the code below, the defaults and ranges for the number of samples, the bias, and the amount of noise can be adjusted easily. These will be inputs to the "make_regression" function.
Can you write Python code for it?


I am working on the file "20170616_PlotlyDashRegressionExample.ipynb".
The first task is:
Based on the different way that Plotly apps are laid out, the order of the sections will be changed somewhat, compared to the Bokeh example. The next section contains the base level function(s), which in this case will be used to generate data.
Can you write Python code for it?


I am working on the file "20170616_PlotlyDashRegressionExample.ipynb".
The first task is:
Finally, the "Callbacks" section defines what happens when the user controls are interacted with (sample size, noise, and bias sliders). If the user controls are changed in any way, new data will be generated based on their values and the figure will be updated automatically. Plotly Dash does this using a function decorator called @app.callback. It's important to remember that the inputs specified in the decorator will be in the same order as the parameters to the function.
Can you write Python code for it?

###-----------------------------------------------------------------------###
###----------------------------CALLBACKS----------------------------------###
### This section defines the behavior of the GUI as the user interacts    ###
### with the controls.                                                  ###
###-----------------------------------------------------------------------###
# Define which inputs (the sliders) go to which outputs (the figure)
@app.callback(
    dash.dependencies.Output('regression-example', 'figure'),
    [dash.dependencies.Input('samp_slider', 'value'),
     dash.dependencies.Input('bias_slider', 'value'),
     dash.dependencies.Input('noise_slider', 'value')])
def update_graph(sampling_rate, bias, noise):
    xdata, ydata = create_data(sampling_rate, bias, noise)
    return {
        'data': [go.Scatter(x=xdata, y=ydata, mode='markers',
                        marker={
                            'size': 15,
                            'opacity': 0.5,
                            'line': {'width': 0.5, 'color': 'white'}
                        })],
        'layout': {'title': 'Regression Data Example in Plotly'}}

if __name__ == '__main__':
    app.run_server()


I am working on the file "20170706-dojo-clear-to-end-of-table.ipynb".
The first task is:
The above is what the output should look like.
Can you write Python code for it?

%%script 20170706_c_foo

void blank(char buf[LINES][COLUMNS], int row, int column)
{
    for ( ; row < LINES; row++) {
        for ( ; column < COLUMNS; column++)
            buf[row][column] = ' ';
        column = 0;
    }
}

%%script 20170706_c_foo

void blank_to_end_of_row(char buf[LINES][COLUMNS], int row, int column)
{
    for ( ;column < COLUMNS; column++)
        buf[row][column] = ' ';
}

void blank_row(char buf[LINES][COLUMNS], int row)
{
    blank_to_end_of_row(buf, row, 0);
}

void blank(char buf[LINES][COLUMNS], int row, int column)
{
    blank_to_end_of_row(buf, row++, column);

    for ( ; row < LINES; row++)
        blank_row(buf, row);
}

%%script 20170706_c_foo

void blank_to_end_of_row(char buf[LINES][COLUMNS], int row, int column)
{
    for ( ;column < COLUMNS; column++)
        buf[row][column] = ' ';
}

void blank(char buf[LINES][COLUMNS], int row, int column)
{
    blank_to_end_of_row(buf, row++, column);

    for ( ; row < LINES; row++)
        blank_to_end_of_row(buf, row, 0);
}


I am working on the file "20170709_TransformerPrognosticsPart2Simulator.ipynb".
The first task is:
As expected, when the DP of the paper insulation is low (due to damage from heat stresses), the CO accumulation is high (due to byproducts from the degradation process).
Transformer Simulation Overview
The following figure demonstrates the basic "flow" of the simulation:
[IMAGE IN BLOG POST]
First, the ambient temperature will be estimated based on the current month and the monthly lows, averages, and highs specified by the user. Per day, it is assumed that the transformer experiences 6 hours at the low temperature, 12 hours at the average temperature, and 6 hours at the high temperature.
Next, the core hot spot temperature, paper insulation damage. and oil contamination is calculated. The core hot spot temperature requires the ambient temperature and overload ratio as inputs. The paper insulation degradation module will use the core hot spot temperature to assess the damage to the paper insulation. The oil contamination module will take the current DP of the paper insulation and find the total accumulated CO and CO2.
Finally, the simulation must assess whether the paper insulation of the transformer has failed or not. Rather than set a “hard” threshold at 150 DP (as recommended by Emsley), the simulation was designed to give the paper insulation a probability to fail at values between 150 DP and 250 DP, resulting in a “soft” distribution of failures.
The following code implements the process described above.
Can you write Python code for it?

# Transformer INSULation SIMulator
# temps is an array of size 12x3, containing monthly low, average, and high temps
# start_month is an integer, with January = 1, February = 2, etc.
# num_transformers are the number of transformers to simulate (e.g. 20)
# overload_ratio is R in the equation above (suggest between 0.75 and 1.1)
def tinsul_sim(temps, start_month, num_transformers, overload_ratio = 1):
    trans_co = []
    trans_co2 = []
    # Iterate over the number of transformers:
    for ii in range(num_transformers):
        month_current = start_month
        failed = False
        dp_current = 1000
        co_accumulation = []
        co2_accumulation = []
        # dp_failed determines the DP at which the paper insulation will
        # fail, drawn from a logistic distribution centered at 200
        dp_failed = np.random.logistic(loc=200.0, scale=40.0)
        # Run the simulation until the transformer fails
        while not failed:
            # Remember- 6 hours at low, 12 hours at avg, and 6 hours at high temps
            # The first index is temp, the second index is hours at that temp
            ambient_low = [temps[int(month_current)-1][0], 6]
            ambient_avg = [temps[int(month_current)-1][1], 12]
            ambient_high = [temps[int(month_current)-1][2], 6]
            # Update DP based on the heat stresses from the core hot spot
            for ambient, time in [ambient_low, ambient_avg, ambient_high]:
                chs = core_hot_spot(ambient, overload_ratio)
                dp_current = calculate_dp(chs, time, dp_current)
            # Calculate the CO and CO2 accumulation based on DP
            co, co2 = oil_contamination(dp_current)
            co_accumulation.append(co)
            co2_accumulation.append(co2)
            # Check if the transformer has failed, using a "soft" distribution
            if dp_current<=dp_failed:
                failed = True
            # Add the value (Months/Weeks) to get proportion of month per week
            month_current +=...


I am working on the file "20170709_TransformerPrognosticsPart2Simulator.ipynb".
The first task is:
The figure above plots the ratio of initial DP (assumed to be starting at 1000) to final DP after 24 hours at a particular core hot spot temperature. Notice that below about 80 degrees C there is very little change, but after 110 degrees C the ratio drops off sharply. This behavior is the reason for derating; as the overload ratio of the transformer decreases, its core hot spot temperature will also decrease, which reduces the amount of degradation the paper insulation will experience.
Oil Contamination
As mentioned in Part 1 of this topic, damage to the paper insulation will release dissolved gas byproducts (such as hydrogen, carbon monoxide, and carbon dioxide) in the insulating oil. These dissolved gases are condition indicators, correlated with the amount of degradation that the paper insulation has experienced.
For the oil contamination module, several papers were used to simulate the accumulation of carbon monoxide and carbon dioxide in a transformer based on paper insulation DP. Unfortunately, academic papers describing this relationship are limited, and power companies are very protective of their DGA data. Hence, measurements from a paper by [Pradhan and Ramu] were used to create a linear relationship between DP and carbon monoxide/dioxide.
While this relationship may not be entirely accurate, it is good enough for the purposes of this simulation. To account for the random variation in the paper's data, normally-distributed noise was added in this module based on the standard deviation of the empirical data.
(Note that this situation is also commonly modeled using chemical equations as a "physics-of-failure" model. Paper insulation is an organic compound, and its breakdown should result in hydrogen and carbon compounds, etc.)
The equations developed from empirical data in Pradhan and Ramu's paper are:
[IMAGE IN BLOG POST]
The following code will implement these equations within the oil contamination module. Note that the CO and CO2 values that this function returns are the total accumulation of these...
Can you write Python code for it?

def oil_contamination(dp):
    # dp is the degree of polymerization
    # CO and CO2 are the TOTAL accumulation
    # of each of those dissolved gases
    CO=(-0.0028*dp + 6.28) + 0.13*np.random.randn()
    CO2=(-0.0026*dp + 8.08) + 0.66*np.random.randn()
    return CO, CO2

# Plot the CO as a function of DP over time
co_accumulation = [oil_contamination(dp)[0] for dp in range(150, 1000)]
plt.plot(range(150, 1000), co_accumulation)
plt.xlabel('Degree of Polymerization (DP)')
plt.ylabel('CO Accumulation (ppm)')
plt.title('CO Accumulation at Paper Insulation DPs');


I am working on the file "20170713_TransformerPrognosticsPart3Prognostics.ipynb".
The first task is:
Simply knowing which environment the transformer has been exposed to will significantly improve the accuracy of the model by about half a year.
Type 3 Prognostics: Equipment-Based
Finally, Karen has an epiphany- if the ambient temperature each individual transformer experiences over its lifetime can be estimated, why not take this approach a step further and make RUL predictions for each transformer? After all, the transformers that she’s examining have Dissolved Gas Analysis (DGA) devices installed. Since her research indicated that dissolved gases are well-correlated with paper insulation degradation, she can create a Type 3 prognostics model by trending the carbon monoxide and carbon dioxide measurements towards a functional failure.
To establish the measurement threshold at which the paper insulation is likely to fail, Karen finds the sum of the CO and CO2 accumulation at failure (i.e. the last available data point for each transformer in the training set) for all transformers in the training dataset:
Can you write Python code for it?

# Determine the CO + CO2 accumulation at failure by finding
# the last value in each array
co_co2_accum = []
for co, co2 in [dc_train, ph_train, de_train]:
    for co_per_trans, co2_per_trans in zip(co, co2):
        co_co2_accum.append(co_per_trans[-1] + co2_per_trans[-1])
type3_mean = np.array(co_co2_accum).mean()

# Plot the results
sns.distplot(co_co2_accum, kde=False)
plt.title('Type 3 CO+CO2 Distribution at Failure\nMean={} ppm'.format(
         str(round(type3_mean,2))))
plt.xlabel('CO+CO2 (ppm)')
plt.ylabel('Occurrences');


I am working on the file "20170713_TransformerPrognosticsPart3Prognostics.ipynb".
The first task is:
The first parameter above is the slope of the log model, and the second parameter is the intercept. Why did Karen find the average of the log model parameters from the training dataset instead of simply regressing the data from the testing transformers directly? There might be some unacceptable variation in the testing transformers, especially if very little data has been accumulated (such as when the transformer is in the early stages of its lifetime). For example, what if there is a negative slope during the initial data of a testing transformer? By projecting future DGA values using the "typical" fit established by the training data, the accuracy (and "safety") of the model is increased.
Of course, using the full CO+CO2 accumulation of the testing transformers seems like cheating- if Karen already knew the full accumulation, she would also know the transformer's exact failure time, which would trivially solve the problem. To make the situation more fair, Karen will make a prediction at a random time in each testing transformer's lifetime (starting at its middle-of-life), and predicting the failure time accordingly:
Can you write Python code for it?

# ae_arr3 will contain the absolute error from each transformer
ae_arr3 = []
# Iterate over all transformers from the testing data set
for co, co2 in [dc_test, ph_test, de_test]:
    for co_per_trans, co2_per_trans in zip(co, co2):
        fail_time = len(co_per_trans)
        # 1. Clip the array at a random time in the transformer lifetime
        np.random.seed(11)
        end_time = np.random.randint(int(fail_time/2), fail_time-1)
        #end_time = np.random.randint(1, fail_time-1)
        co_per_trans = co_per_trans[0:end_time]
        co2_per_trans = co2_per_trans[0:end_time]
        # 2. Find the functional fit of the testing data
        # t is in weeks; max t is the length of the array
        t = np.linspace(0, 1, end_time)
        co_co2 = np.add(co_per_trans, co2_per_trans)
        # fit using a log function
        fit_t = np.polyfit(t, np.log(co_co2), 1)
        # find the predicted co+co2 at the last data point
        # why do this? to smooth out the noise in the "actual"
        # last data point
        cur_co_co2 = fit_t[0]*end_time + fit_t[1]
        # 3. Find the predicted remaining useful life
        # now, determine where the future co+co2 values will
        # reach the failure threshold determined above
        t = 0 # time until predicted failure
        while cur_co_co2<= type3_mean:
            t += 1
            cur_co_co2 += type3_param1
        # 4. Find the prediction error
        actual_rul = fail_time - end_time
        ae = np.abs(t-actual_rul)
        ae_arr3.append(ae)
type3_mae = np.mean(ae_arr3)
print('Type 3 MAE is {} weeks'.format(str(round(type3_mae,2))))


I am working on the file "20170720-building-social-network-graphs-CSV.ipynb".
The first task is:
2. Leave only the tweets whose strength level >= user specified level (directed)
Can you write Python code for it?

strengthLevel = 3  # Network connection strength level: the number of times in total each of the tweeters responded to or mentioned the other.
                   # If you have 1 as the level, then all tweeters who mentioned or replied to another at least once will be displayed. But if you have 5, only those who have mentioned or responded to a particular tweeter at least 5 times will be displayed, which means that only the strongest bonds are shown.

edges2 = edges.groupby(['Source','Target'])['Strength'].count()
edges2 = edges2.reset_index()
edges2 = edges2[edges2['Strength'] >= strengthLevel]


I am working on the file "20170720-building-social-network-graphs-CSV.ipynb".
The first task is:
4. Export nodes and edges to csv files
Can you write Python code for it?

# change column names for Kumu import (Run this when using Kumu)
edges2.columns = ['From','To','Strength']

# Print nodes to check
nodes.head(3)

# Print edges to check
edges2.head(3)

# Export nodes and edges to csv files
nodes.to_csv('nodes.csv', encoding='utf-8', index=False)
edges2.to_csv('edges.csv', encoding='utf-8', index=False)


I am working on the file "20170720-building-social-network-graphs-JSON.ipynb".
The first task is:
1. Export edges from Retweets, Mentions, or Replies

Run one of three blocks of codes below for your purpose.
Can you write Python code for it?

# 1. Export edges from Retweets

fh = open(tweetfile, 'r')

userdata = pd.DataFrame(columns=('Id','Label','user_created_at','profile_image','followers_count','friends_count' ))
edges = pd.DataFrame(columns=('Source','Target','Strength'))

for line in fh:
    try:
        tweet = json.loads(line)
    except:
        continue
    if 'retweeted_status' not in tweet:
        continue

    userdata = userdata.append(pd.DataFrame([[tweet['user']['id_str'],
                                tweet['user']['screen_name'],
                                tweet['user']['created_at'],
                                tweet['user']['profile_image_url_https'],
                                tweet['user']['followers_count'],
                                tweet['user']['friends_count']]], columns=('Id','Label','user_created_at','profile_image','followers_count','friends_count')), ignore_index=True)
    userdata = userdata.append(pd.DataFrame([[tweet['retweeted_status']['user']['id_str'],
                                tweet['retweeted_status']['user']['screen_name'],
                                tweet['retweeted_status']['user']['created_at'],
                                tweet['retweeted_status']['user']['profile_image_url_https'],
                                tweet['retweeted_status']['user']['followers_count'],
                                tweet['retweeted_status']['user']['friends_count']]], columns=('Id','Label','user_created_at','profile_image','followers_count','friends_count')), ignore_index=True)
    edges = edges.append(pd.DataFrame([[tweet['user']['id_str'],
                                tweet['retweeted_status']['user']['id_str'],
                                str(datetime.strptime(tweet['created_at'], '%a %b %d %H:%M:%S +0000 %Y'))]]
                                , columns=('Source','Target','Strength')), ignore_index=True)

# 2. Export edges from Mentions

fh = open(tweetfile, 'r')

userdata =...


I am working on the file "20170720-building-social-network-graphs-JSON.ipynb".
The first task is:
2. Leave only the tweets whose strength level >= user specified level (directed)
Can you write Python code for it?


I am working on the file "20170720-dojo-primes-revisited.ipynb".
The first task is:
Do it all in one expression. It works, but is ugly and hard to read.
Can you write Python code for it?

def gen_primes(start=2):
    return filter(
        lambda p: all(p % divisor != 0 for divisor in range(2, p)),
        count(start)
    )

check()


I am working on the file "20170720-dojo-primes-revisited.ipynb".
The first task is:
This is pretty readable.
Can you write Python code for it?

def is_prime(p):
    return all(p % divisor != 0 for divisor in range(2, p))

def gen_primes(start=2):
    return filter(is_prime, count(start))

check()


I am working on the file "20170913_python_decorator_contextmanager/decorator_and_contextmanager.ipynb".
The first task is:
Case 2: timeit
前面有提到透過 decorator 來做 function 的 profiling, 但假如我今天不是要對 function, 而是單純要對某個 block 的程式碼做 profiling 呢? 是否要把這個 block 包成 function 再做 profiling ?
這個時候上面提到的 @contextmanager 就派上用場了, 透過 contextmanager 在 entry point 跟 exit point 都紀錄時間來計算耗時
Can you write Python code for it?

from contextlib import contextmanager
import time

@contextmanager
def timeit(name=''):
    start_time = time.time()
    yield
    elapse_time = time.time() - start_time
    print('{} - completed in {:.6f}'.format(name, elapse_time))

# note: comprehension speed seens slower when range(10)
test = [i for i in range(100000)]

with timeit('afun 1'):
    a = [i for i in test if i % 2 == 0]

with timeit('afun 2'):
    b = []
    for i in test:
        if i % 2 == 0:
            b.append(i)

@timeit('afun 3')
def func(test_list):
    result = []
    for i in test_list:
        if i % 2 == 0:
            result.append(i)
    return result

c = func(test)


I am working on the file "20170913_python_decorator_contextmanager/decorator_and_contextmanager.ipynb".
The first task is:
Case 4: Cache to json file
對於快速的 debug 非常有幫助, 會把 exception 內容都存成 json
Hint: 這邊 demo 的時候可以操作連續兩次看哪裡不同, 試著觸發 wrapped 內的 exception
Can you write Python code for it?

import json
from functools import wraps

def cache_json(filename):
    def deco(func):
        @wraps(func)
        def wrapped(*args, **kwargs):
            try:
                print('Before try wrapped')
                return json.load(open(filename))
            except FileNotFoundError:
                print('Before except wrapped')
                data = func(*args, **kwargs)
                json.dump(data, open(filename, 'w'))
                return data
        return wrapped
    return deco

@cache_json('heavy.json')
def heavy_jobs(*args, **kwargs):
    print('do heavy jobs')
    if 'count' in kwargs:
        return kwargs['count']
    return

print(heavy_jobs(user='afun', count=5))


I am working on the file "20170913_python_decorator_contextmanager/decorator_and_contextmanager.ipynb".
The first task is:
Context manager
Basic concept
所謂 context 就是上下文的意思, 跟 decorator 同樣的地方是他可以在執行跟結束 function 的時候有額外操作
Can you write Python code for it?

class my_context(object):
    def __enter__(self):
        print('in enter')
        return 'enter'
    def __exit__(self, *excinfo):
        print('in exit')
        return 'exit'

with my_context() as f:
    print('Hello')
    print('World')


I am working on the file "20170913_python_decorator_contextmanager/decorator_and_contextmanager.ipynb".
The first task is:
Contextmanager limitation
Generator 中只允許一個物件
Can you write Python code for it?

from contextlib import contextmanager

@contextmanager
def my_context():
    yield 'Hello'
    yield 'World'

with my_context():
    print('line 1')
    print('line 2')

from contextlib import contextmanager

@contextmanager
def context_loop():
    for i in range(100):
        yield i

with context_loop():
    print('line 1')
    print('line 2')

from contextlib import contextmanager

@contextmanager
def context_condition(cond):
    if cond:
        yield 'in condition'
    else:
        yield 'else case'

with context_condition(True) as f1:
    print(f1)
    print('line 1')
    print('line 2')

with context_condition(False) as f2:
    print(f2)
    print('line 3')
    print('line 4')

from contextlib import contextmanager

@contextmanager
def context_try(cond):
    try:
        yield 'normal'
    except Exception:
        print('exception')
    finally:
        print('finally')

with context_try(True):
    print('line 1')
    print('line 2')


I am working on the file "20170913_python_decorator_contextmanager/decorator_and_contextmanager.ipynb".
The first task is:
Decorator assignment
Python 裏面所有的東西都是 Object, 既然是這樣那 decorator 應該也可以存成其他變數名稱並且正常使用
Can you write Python code for it?

from functools import wraps

def tag(name):
    def deco(func):
        @wraps(func)
        def wrapped(*args, **kwargs):
            print('<{}>'.format(name))
            func(*args, **kwargs)
            print('</{}>'.format(name))
        return wrapped
    return deco

tag_p = tag('p')

@tag_p
def content(*args, **kwargs):
    for i in args:
        print(i)

content('Hello World.', 'This is second argument.')


I am working on the file "20170913_python_decorator_contextmanager/decorator_and_contextmanager.ipynb".
The first task is:
Decorator assignment (2)
既然 decorator 是包裝其他 function, 那 decorator 應該也可以包裝 decorator ?
Can you write Python code for it?

from functools import wraps

def tag(name):
    def deco(func):
        @wraps(func)
        def wrapped(*args, **kwargs):
            print('<{}>'.format(name))
            func(*args, **kwargs)
            print('</{}>'.format(name))
        return wrapped
    return deco

tag_div = tag('div')
tag_p = tag('p')

@tag_div
@tag_p
def content(*args, **kwargs):
    for i in args:
        print(i)

content('Hello World.', 'This is second argument.')


I am working on the file "20170913_python_decorator_contextmanager/decorator_and_contextmanager.ipynb".
The first task is:
Decorator scenario
Case 1: Profiling Log
可以透過 decorator 寫出漂亮的 Pythonic code (e.g. link)
Task: 重複操作「紀錄當下時間為起始時間」->「執行 function」->「當下時間減掉起始時間為 function 耗時時間」
搭配 logging 還可以設定 log level 為 DEBUG,　這樣可以非常輕鬆的 enable/ disable 這些訊息
Can you write Python code for it?

import time
import logging
from functools import wraps

LOGGER = logging.getLogger(__name__)

def func_profiling(func):
    @wraps(func)
    def wrapped(*args, **kwargs):
        start_time = time.time()
        result = func(*args, **kwargs)
        time_spent = time.time() - start_time
        fullname = '{}.{}'.format(func.__module__, func.__name__)
        LOGGER.debug('{}[args={}, kwargs={}] completed in {}'.format(
            fullname, args, kwargs, time_spent
        ))
        return result
    return wrapped

@func_profiling
def test_func_profiling(msg=None):
    import random
    sleep_sec = random.randrange(1,3)
    LOGGER.debug('random sleep in {} sec'.format(sleep_sec))
    time.sleep(sleep_sec)
    LOGGER.info(msg)
    LOGGER.debug('Wake up')

if __name__ == '__main__':
    """testing"""
    import sys
    logging.basicConfig(
        level=logging.DEBUG,
        format='%(asctime)s %(filename)12s:L%(lineno)3s [%(levelname)8s] %(message)s',
        datefmt='%Y-%m-%d %H:%M:%S',
        stream=sys.stdout
        )
    test_func_profiling('Hello World')


I am working on the file "20170913_python_decorator_contextmanager/decorator_and_contextmanager.ipynb".
The first task is:
Study 1: Detail of contextmanager
contextmanger 是一個為了 with statement 做出來不用特別建立具有 __enter__ 跟 __exit__ class 的 decorator
我們可以觀察原始碼近一步了解內部運作機制

contextmanager
ContextDecorator
Can you write Python code for it?

from functools import wraps

# contextmanager 原始碼
# 這邊就是我們上面提過的 decorator 寫法
# 雖然他是回傳 _GeneratorContextManager 這個 helper class, 但其實這邊主要是透過 ContextDecorator 實作
def contextmanager(func):
    @wraps(func)
    def helper(*args, **kwds):
        return _GeneratorContextManager(func, args, kwds)
    return helper

# ContextDecorator 原始碼
# _recreate_cm 會回傳自己的 instance
class ContextDecorator(object):

    def _recreate_cm(self):
        return self

    def __call__(self, func):
        @wraps(func)
        def inner(*args, **kwds):
            with self._recreate_cm():
                return func(*args, **kwds)
        return inner


I am working on the file "20170913_python_decorator_contextmanager/decorator_and_contextmanager.ipynb".
The first task is:
Supplement: pass argument with decorator
如果執行 function 需要參數, 記得在 decorator 裡包裝 function 的時候也一並要傳入
Can you write Python code for it?

import time
from functools import wraps

def deco(func):
    @wraps(func)
    def wrapped(*args, **kwargs):
        print('Before func')
        print('---')
        print('{} argument:\nargs = {}\nkwargs = {}\n'.format(
            func.__name__, args, kwargs
        ))
        func(*args, **kwargs)
        print('---')
        print('After func')
    return wrapped

@deco
def f1(*args, **kwargs):
    print('This is f1 function')

f1(1, '2', [3], {'4': 4}, time=time.ctime())


I am working on the file "20170913_python_decorator_contextmanager/decorator_and_contextmanager.ipynb".
The first task is:
到這邊你可能會疑惑這兩邊哪裡一樣？然後 yield 的東西呢？
其實你可以把 contextlib.contextmanager 視為一個轉換工具
Can you write Python code for it?

@contextmanager
def my_context():
    print('do things in enter')
    yield 'It is a feature, not a bug!!!'
    print('do things in exit')

# 透過 contextmanager 轉換, 與下面這段程式碼等價

class my_context(object):
    def __enter__(self):
        print('do things in enter')
        return 'It is a feature, not a bug!!!'
    def __exit__(self, *excinfo):
        print('do things in exit')
        return 'exit'


I am working on the file "20170928-dojo-references-map-zip_longest-fstrings.ipynb".
The first task is:
Show what map() does.
Can you write Python code for it?

def foo(s):
    return s + ' on the Beach.'

list(map(foo, ('sand', 'clams', 'dunes')))

a = (1, 2, 3)
b = (2, 3, 4, 5)

from itertools import zip_longest

sum(map(sum, zip_longest(a, b, fillvalue=0)))

sum((sum(a), sum(b)))

sum(a + b)


I am working on the file "2017_05_19__Work_with_images.ipynb".
The first task is:
Calling Image Magick from Python
TBD
Image Magick : http://www.imagemagick.org/script/index.php
Can you write Python code for it?


I am working on the file "2017_05_19__Work_with_images.ipynb".
The first task is:
If you are downloading it from Internet, use the following command by modifying the local_loc to the folder you are interested in saving the image.
Can you write Python code for it?

import urllib.request
local_loc = 'C:\data\local_image.tif'
urllib.request.urlretrieve('http://folk.uio.no/sebastcc/images/Mtg01_bl1_MOAB2_s023_scaled20perc.tif', local_loc)
im = Image.open(local_loc)

p=1

type(p)

type(im)

print(im.height,im.width)

im.mode?

im.show()

# split the image into individual bands
source = im.split()

R, G, B = 0, 1, 2

type(source)

type(source[R])

type(source[G])

source[R]


I am working on the file "2017_05_19__Work_with_images.ipynb".
The first task is:
Much faster solution using eval method from Image module (thanks to Gergely). No need to split the channels.
Can you write Python code for it?

im = Image.open(t)
imfast = Image.eval(im,inv_pix)


I am working on the file "2017_3/Mask_pipeline_2017_V6.ipynb".
The first task is:
Create a polygon mask

check beam center
Can you write Python code for it?


I am working on the file "2017_3/Mask_pipeline_2017_V6.ipynb".
The first task is:
Get the image series and metadata from the uid
Can you write Python code for it?

uid = 'e09db5' # (scan num: 6358) (Measurement: Single image for masking )
uid = '727729' # (scan num: 6708) (Measurement: test series with feedback )
uid =  '2bc66e'# (scan num: 6709) (Measurement: .2s 1k 36/.2% )

md = get_meta_data( uid )
detector = get_detector( db[uid ] )
print ('Detector is:  %s'%detector  )
sud = get_sid_filenames(db[uid])

print ('scan_id, full-uid, data path are:  %s--%s--%s'%(sud[0], sud[1], sud[2][0] ))

print(md['beam_center_y'], md['beam_center_x'])

imgs = load_data( uid, detector, reverse= False  )
#imgs = load_data( uid, detector, reverse= True  )
md.update( imgs.md );Nimg = len(imgs);
#if 'number of images'  not in list(md.keys()):
md['number of images']  = Nimg
pixel_mask =  1- np.int_( np.array( imgs.md['pixel_mask'], dtype= bool)  )
print( 'The data are: %s' %imgs )

pixel_mask =  1- np.int_( np.array( md['pixel_mask'], dtype= bool)  )

img_choice_N = 1  #can change this number to select more frames for average
img_samp_index = random.sample( range(len(imgs)), img_choice_N)
avg_img =  get_avg_img( imgs, img_samp_index, plot_ = False, uid = uid)


I am working on the file "2017_3/Mask_pipeline_2017_V6.ipynb".
The first task is:
Path for Saving Results
Can you write Python code for it?

CYCLE= '2017_3'  #change clycle here

path = '/XF11ID/analysis/%s/masks/'%CYCLE
print ("The analysis results will be saved in : %s"%path)


I am working on the file "2017_3/Mask_pipeline_2017_V6.ipynb".
The first task is:
Save the combined mask to use in further data analysis
Can you write Python code for it?

np.save(  path +   uid +"_mask", mask)

path +   uid +"_mask"


I am working on the file "2017_3/Mask_pipeline_2017_V6.ipynb".
The first task is:
To create multi-rectangle masks, for each sub-mask
Can you write Python code for it?

path

#creat the right part mask
partial_mask = create_cross_mask(  avg_img,center=[1000,1218],
                            wy_left= 0, wy_right= 35,
                              wx_up= 0, wx_down= 0,center_radius= 0 )

show_img(  partial_mask   )

#np.save( '/XF11ID/analysis/2017_1/masks/Ver_Beamstop', partial_mask )
#Ver_Beamstop = np.load( '/XF11ID/analysis/2017_1/masks/Ver_Beamstop.npy' )
#Ver_Beamstop = move_beamstop( Vertical_Beamstop, xshift=0, yshift=0 )

#creat the left/right/up/down part mask
partial_mask *=  create_cross_mask(  avg_img, center=[ 1099,1218],
                            wy_left= 0, wy_right= 0,
                              wx_up= 5, wx_down=5,center_radius= 0 )
#partial_mask2[1285:1350,1430:1440,] = False

#creat the left/right/up/down part mask
partial_mask *=  create_cross_mask(  avg_img, center=[ 1030,395],
                            wy_left= 145, wy_right= 0,
                              wx_up= 0, wx_down= 0,center_radius= 0 )
#partial_mask2[1285:1350,1430:1440,] = False

#creat the left/right/up/down part mask
partial_mask *=  create_cross_mask(  avg_img, center=[ 1139,1225],
                            wy_left= 0, wy_right= 0,
                              wx_up= 0, wx_down=0,center_radius= 0 )
#partial_mask2[1285:1350,1430:1440,] = False

#from chxanalys.chx_generic_functions import create_multi_rotated_rectangle_mask

#np.save( '/XF11ID/analysis/2017_1/masks/Hor_Beamstop', partial_mask )
#Hor_Beamstop = np.load( '/XF11ID/analysis/2017_1/masks/Hor_Beamstop.npy' )
#Hor_Beamstop = move_beamstop( Hor_Beamstop, xshift=0, yshift=0 )

show_img(  partial_mask   )

#creat the left/right/up/down part mask
#partial_mask *=  create_cross_mask(  avg_img, center=[ 1615,2000],
#                            wy_left= 0, wy_right= 100,
#                              wx_up= 0, wx_down=0,center_radius= 0 )
#partial_mask2[1285:1350,1430:1440,] = False

#np.save( '/XF11ID/analysis/2017_1/masks/Bad_4M', partial_mask )
#Bad_4M = np.load( '/XF11ID/analysis/2017_1/masks/Bad_4M.npy' )
...


I am working on the file "2017_3/Mask_pipeline_2017_V6.ipynb".
The first task is:
show image and the pixel mask

show image
Can you write Python code for it?

show_img(  avg_img*pixel_mask , vmin=.001, vmax=1e8, logs=True,
         image_name ='uid=%s'%uid, aspect=1, cmap= cmap_albula  )


I am working on the file "2017_Fall/CSCI-599/Assignment01/Problem_1.ipynb".
The first task is:
Comparing the optimizers
Run the following code block to compare the plotted results among all the above optimizers
Can you write Python code for it?

model_rms      = FullyConnectedNetwork()
model_adam     = FullyConnectedNetwork()
loss_f_rms     = cross_entropy()
loss_f_adam    = cross_entropy()
optimizer_rms  = RMSProp(model_rms.net, 5e-4)
optimizer_adam = Adam(model_adam.net, 5e-4)

print "Training with RMSProp..."
results_rms = train_net(small_data_dict, model_rms, loss_f_rms, optimizer_rms, batch_size=100,
                        max_epochs=5, show_every=100, verbose=True)

print "\nTraining with Adam..."
results_adam = train_net(small_data_dict, model_adam, loss_f_adam, optimizer_adam, batch_size=100,
                         max_epochs=5, show_every=100, verbose=True)

opt_params_rms,  loss_hist_rms,  train_acc_hist_rms,  val_acc_hist_rms  = results_rms
opt_params_adam, loss_hist_adam, train_acc_hist_adam, val_acc_hist_adam = results_adam

plt.subplot(3, 1, 1)
plt.title('Training loss')
plt.xlabel('Iteration')

plt.subplot(3, 1, 2)
plt.title('Training accuracy')
plt.xlabel('Epoch')

plt.subplot(3, 1, 3)
plt.title('Validation accuracy')
plt.xlabel('Epoch')

plt.subplot(3, 1, 1)
plt.plot(loss_hist_sgd, 'o', label="Vanilla SGD")
plt.subplot(3, 1, 2)
plt.plot(train_acc_hist_sgd, '-o', label="Vanilla SGD")
plt.subplot(3, 1, 3)
plt.plot(val_acc_hist_sgd, '-o', label="Vanilla SGD")

plt.subplot(3, 1, 1)
plt.plot(loss_hist_sgdm, 'o', label="SGD with Momentum")
plt.subplot(3, 1, 2)
plt.plot(train_acc_hist_sgdm, '-o', label="SGD with Momentum")
plt.subplot(3, 1, 3)
plt.plot(val_acc_hist_sgdm, '-o', label="SGD with Momentum")

plt.subplot(3, 1, 1)
plt.plot(loss_hist_rms, 'o', label="RMSProp")
plt.subplot(3, 1, 2)
plt.plot(train_acc_hist_rms, '-o', label="RMSProp")
plt.subplot(3, 1, 3)
plt.plot(val_acc_hist_rms, '-o', label="RMSProp")

plt.subplot(3, 1, 1)
plt.plot(loss_hist_adam, 'o', label="Adam")
plt.subplot(3, 1, 2)
plt.plot(train_acc_hist_adam, '-o', label="Adam")
plt.subplot(3, 1, 3)
plt.plot(val_acc_hist_adam, '-o', label="Adam")

for i in [1, 2, 3]:
  plt.subplot(3, 1, i)
  plt.legend(loc='upper center',...


I am working on the file "2017_Fall/CSCI-599/Assignment01/Problem_1.ipynb".
The first task is:
Dropout Backward
Please complete the backward pass. Again remember that the dropout is only applied during training phase, handle this in the backward pass as well.
Can you write Python code for it?

x = np.random.randn(5, 5) + 5
dout = np.random.randn(*x.shape)

p = 0.75
dropout_b = dropout(p, seed=100)
out = dropout_b.forward(x, True)
dx = dropout_b.backward(dout)
dx_num = eval_numerical_gradient_array(lambda xx: dropout_b.forward(xx, True), x, dout)

# The error should not be larger than 1e-9
print 'dx relative error: ', rel_error(dx, dx_num)


I am working on the file "2017_Fall/CSCI-599/Assignment01/Problem_1.ipynb".
The first task is:
Dropout Forward
In the class "dropout", please complete the forward pass. Remember that the dropout is only applied during training phase, you should pay attention to this while implementing the function.
Can you write Python code for it?

x = np.random.randn(100, 100) + 5.0

print "----------------------------------------------------------------"
for p in [0.25, 0.50, 0.75]:
    dropout_f = dropout(p)
    out = dropout_f.forward(x, True)
    out_test = dropout_f.forward(x, False)

    print "Dropout p = ", p
    print "Mean of input: ", x.mean()
    print "Mean of output during training time: ", out.mean()
    print "Mean of output during testing time: ", out_test.mean()
    print "Fraction of output set to zero during training time: ", (out == 0).mean()
    print "Fraction of output set to zero during testing time: ", (out_test == 0).mean()
    print "----------------------------------------------------------------"


I am working on the file "2017_Fall/CSCI-599/Assignment01/Problem_1.ipynb".
The first task is:
FC Backward
Please complete the function "backward" as the backward pass of the fc layer. Follow the instructions in the comments to store gradients into the predefined dictionaries in the attributes of the class. Parameters of the layer are also stored in the predefined dictionary.
Can you write Python code for it?

# Test the fc backward function
x = np.random.randn(10, 2, 2, 3)
w = np.random.randn(12, 10)
b = np.random.randn(10)
dout = np.random.randn(10, 10)

single_fc = fc(np.prod(x.shape[1:]), 10, init_scale=5e-2, name="fc_test")
single_fc.params[single_fc.w_name] = w
single_fc.params[single_fc.b_name] = b

dx_num = eval_numerical_gradient_array(lambda x: single_fc.forward(x), x, dout)
dw_num = eval_numerical_gradient_array(lambda w: single_fc.forward(x), w, dout)
db_num = eval_numerical_gradient_array(lambda b: single_fc.forward(x), b, dout)

out = single_fc.forward(x)
dx = single_fc.backward(dout)
dw = single_fc.grads[single_fc.w_name]
db = single_fc.grads[single_fc.b_name]

# The error should be around 1e-10
print "dx Error: ", rel_error(dx_num, dx)
print "dw Error: ", rel_error(dw_num, dw)
print "db Error: ", rel_error(db_num, db)


I am working on the file "2017_Fall/CSCI-599/Assignment01/Problem_1.ipynb".
The first task is:
Implement Standard Layers
You will now implement all the following standard layers commonly seen in a fully connected neural network. Please refer to the file layer_utils.py under the directory lib. Take a look at each class skeleton, and we will walk you through the network layer by layer. We provide results of some examples we pre-computed for you for checking the forward pass, and also the gradient checking for the backward pass.
FC Forward
In the class skeleton "fc", please complete the forward pass in function "forward", the input to the fc layer may not be of dimension (batch size, features size), it could be an image or any higher dimensional data. Make sure that you handle this dimensionality issue.
Can you write Python code for it?

# Test the fc forward function
input_bz = 3
input_dim = (6, 5, 4)
output_dim = 4

input_size = input_bz * np.prod(input_dim)
weight_size = output_dim * np.prod(input_dim)

single_fc = fc(np.prod(input_dim), output_dim, init_scale=0.02, name="fc_test")

x = np.linspace(-0.1, 0.5, num=input_size).reshape(input_bz, *input_dim)
w = np.linspace(-0.2, 0.3, num=weight_size).reshape(np.prod(input_dim), output_dim)
b = np.linspace(-0.3, 0.1, num=output_dim)

single_fc.params[single_fc.w_name] = w
single_fc.params[single_fc.b_name] = b

out = single_fc.forward(x)

correct_out = np.array([[0.70157129, 0.83483484, 0.96809839, 1.10136194],
                        [1.86723094, 2.02561647, 2.18400199, 2.34238752],
                        [3.0328906,  3.2163981,  3.3999056,  3.5834131]])

# Compare your output with the above pre-computed ones.
# The difference should not be larger than 1e-8
print "Difference: ", rel_error(out, correct_out)


I am working on the file "2017_Fall/CSCI-599/Assignment01/Problem_1.ipynb".
The first task is:
Loading the data (CIFAR-10)
Run the following code block to load in the properly splitted CIFAR-10 data.
Can you write Python code for it?

data = CIFAR10_data()
for k, v in data.iteritems():
    print "Name: {} Shape: {}".format(k, v.shape)


I am working on the file "2017_Fall/CSCI-599/Assignment01/Problem_1.ipynb".
The first task is:
Now train the network to achieve at least 50% validation accuracy
Can you write Python code for it?

results = None
#############################################################################
# TODO: Use the train_net function you completed to train a network         #
#############################################################################
opt_params, loss_hist, train_acc_hist, val_acc_hist = train_net(data_dict, model, loss_f, optimizer, 128, 10)
print (val_acc_hist[-1])
#############################################################################
#                             END OF YOUR CODE                              #
#############################################################################
#opt_params, loss_hist, train_acc_hist, val_acc_hist = results

# Take a look at what names of params were stored
print opt_params.keys()

# Demo: How to load the parameters to a newly defined network
model = TinyNet()
model.net.load(opt_params)
val_acc = compute_acc(model, data["data_val"], data["labels_val"])
print "Validation Accuracy: {}%".format(val_acc*100)
test_acc = compute_acc(model, data["data_test"], data["labels_test"])
print "Testing Accuracy: {}%".format(test_acc*100)

# Plot the learning curves
plt.subplot(2, 1, 1)
plt.title('Training loss')
loss_hist_ = loss_hist[1::100] # sparse the curve a bit
plt.plot(loss_hist_, '-o')
plt.xlabel('Iteration')

plt.subplot(2, 1, 2)
plt.title('Accuracy')
plt.plot(train_acc_hist, '-o', label='Training')
plt.plot(val_acc_hist, '-o', label='Validation')
plt.plot([0.5] * len(val_acc_hist), 'k--')
plt.xlabel('Epoch')
plt.legend(loc='lower right')
plt.gcf().set_size_inches(15, 12)
plt.show()


I am working on the file "2017_Fall/CSCI-599/Assignment01/Problem_1.ipynb".
The first task is:
ReLU Forward
In the class skeleton "relu", please complete the forward pass.
Can you write Python code for it?

# Test the relu forward function
x = np.linspace(-1.0, 1.0, num=12).reshape(3, 4)
relu_f = relu(name="relu_f")

out = relu_f.forward(x)
correct_out = np.array([[0.,          0.,        0.,         0.        ],
                        [0.,          0.,        0.09090909, 0.27272727],
                        [0.45454545, 0.63636364, 0.81818182, 1.        ]])

# Compare your output with the above pre-computed ones.
# The difference should not be larger than 1e-8
print "Difference: ", rel_error(out, correct_out)


I am working on the file "2017_Fall/CSCI-599/Assignment01/Problem_1.ipynb".
The first task is:
Run the following code block to train a multi-layer fully connected network with both SGD and SGD plus Momentum. The network trained with SGDM optimizer should converge faster.
Can you write Python code for it?

# Arrange a small data
num_train = 4000
small_data_dict = {
    "data_train": (data["data_train"][:num_train], data["labels_train"][:num_train]),
    "data_val": (data["data_val"], data["labels_val"]),
    "data_test": (data["data_test"], data["labels_test"])
}

model_sgd      = FullyConnectedNetwork()
model_sgdm     = FullyConnectedNetwork()
loss_f_sgd     = cross_entropy()
loss_f_sgdm    = cross_entropy()
optimizer_sgd  = SGD(model_sgd.net, 1e-2)
optimizer_sgdm = SGDM(model_sgdm.net, 1e-2, 0.9)

print "Training with Vanilla SGD..."
results_sgd = train_net(small_data_dict, model_sgd, loss_f_sgd, optimizer_sgd, batch_size=100,
                        max_epochs=5, show_every=100, verbose=True)

print "\nTraining with SGD plus Momentum..."
results_sgdm = train_net(small_data_dict, model_sgdm, loss_f_sgdm, optimizer_sgdm, batch_size=100,
                         max_epochs=5, show_every=100, verbose=True)

opt_params_sgd,  loss_hist_sgd,  train_acc_hist_sgd,  val_acc_hist_sgd  = results_sgd
opt_params_sgdm, loss_hist_sgdm, train_acc_hist_sgdm, val_acc_hist_sgdm = results_sgdm

plt.subplot(3, 1, 1)
plt.title('Training loss')
plt.xlabel('Iteration')

plt.subplot(3, 1, 2)
plt.title('Training accuracy')
plt.xlabel('Epoch')

plt.subplot(3, 1, 3)
plt.title('Validation accuracy')
plt.xlabel('Epoch')

plt.subplot(3, 1, 1)
plt.plot(loss_hist_sgd, 'o', label="Vanilla SGD")
plt.subplot(3, 1, 2)
plt.plot(train_acc_hist_sgd, '-o', label="Vanilla SGD")
plt.subplot(3, 1, 3)
plt.plot(val_acc_hist_sgd, '-o', label="Vanilla SGD")

plt.subplot(3, 1, 1)
plt.plot(loss_hist_sgdm, 'o', label="SGD with Momentum")
plt.subplot(3, 1, 2)
plt.plot(train_acc_hist_sgdm, '-o', label="SGD with Momentum")
plt.subplot(3, 1, 3)
plt.plot(val_acc_hist_sgdm, '-o', label="SGD with Momentum")

for i in [1, 2, 3]:
  plt.subplot(3, 1, i)
  plt.legend(loc='upper center', ncol=4)
plt.gcf().set_size_inches(15, 15)
plt.show()


I am working on the file "2017_Fall/CSCI-599/Assignment01/Problem_1.ipynb".
The first task is:
Test a Fully Connected Network regularized with Dropout
Please find the DropoutNet function in fully_conn.py under lib directory.

For this part you don't need to design a new network, just simply run the following test code

If something goes wrong, you might want to double check your dropout implementation
Can you write Python code for it?

N, D, C = 3, 15, 10
X = np.random.randn(N, D)
y = np.random.randint(C, size=(N,))
seed = 123

for dropout_p in [0., 0.25, 0.5]:
    print "Dropout p =", dropout_p
    model = DropoutNet(dropout_p=dropout_p, seed=seed)
    loss_func = cross_entropy()
    output = model.forward(X, True)
    loss = loss_func.forward(output, y)
    dLoss = loss_func.backward()
    dX = model.backward(dLoss)
    grads = model.net.grads
    print "Loss (should be ~2.30) : ", loss

    print "Error of gradients should be no larger than 1e-5"
    for name in sorted(model.net.params):
        f = lambda _: loss_func.forward(model.forward(X, True), y)
        grad_num = eval_numerical_gradient(f, model.net.params[name], verbose=False, h=1e-5)
        print "{} relative error: {}".format(name, rel_error(grad_num, grads[name]))
    print


I am working on the file "2017_Fall/CSCI-599/Assignment01/Problem_1.ipynb".
The first task is:
Test a Small Fully Connected Network
Please find the SmallFullyConnectedNetwork function in fully_conn.py under lib directory.

Again you only need to complete few lines of code in the TODO block.

Please design an FC --> ReLU --> FC --> ReLU network where the shapes of parameters match the given shapes

Please insert the corresponding names you defined for each layer to param_name_w, and param_name_b respectively.

Here you only modify the param_name part, the _w, and _b are automatically assigned during network setup
Can you write Python code for it?

model = SmallFullyConnectedNetwork()
loss_func = cross_entropy()

N, D, = 4, 4  # N: batch size, D: input dimension
H, C  = 30, 7 # H: hidden dimension, C: output dimension
std = 0.02
x = np.random.randn(N, D)
y = np.random.randint(C, size=N)

print "Testing initialization ... "
w1_std = abs(model.net.get_params("fc1_w").std() - std)
b1 = model.net.get_params("fc1_b").std()
w2_std = abs(model.net.get_params("fc2_w").std() - std)
b2 = model.net.get_params("fc2_b").std()

assert w1_std < std / 10, "First layer weights do not seem right"
assert np.all(b1 == 0), "First layer biases do not seem right"
assert w2_std < std / 10, "Second layer weights do not seem right"
assert np.all(b2 == 0), "Second layer biases do not seem right"
print "Passed!"

print "Testing test-time forward pass ... "
w1 = np.linspace(-0.7, 0.3, num=D*H).reshape(D, H)
w2 = np.linspace(-0.3, 0.4, num=H*C).reshape(H, C)
b1 = np.linspace(-0.1, 0.9, num=H)
b2 = np.linspace(-0.9, 0.1, num=C)

model.net.assign("fc1_w", w1)
model.net.assign("fc1_b", b1)
model.net.assign("fc2_w", w2)
model.net.assign("fc2_b", b2)

feats = np.linspace(-5.5, 4.5, num=N*D).reshape(D, N).T
scores = model.forward(feats)
correct_scores = np.asarray([[4.20670862, 4.87188359, 5.53705856, 6.20223352, 6.86740849, 7.53258346, 8.19775843],
                             [4.74826036, 5.35984681, 5.97143326, 6.58301972, 7.19460617, 7.80619262, 8.41777907],
                             [5.2898121,  5.84781003, 6.40580797, 6.96380591, 7.52180384, 8.07980178, 8.63779971],
                             [5.83136384, 6.33577326, 6.84018268, 7.3445921,  7.84900151, 8.35341093, 8.85782035]])
scores_diff = np.sum(np.abs(scores - correct_scores))
assert scores_diff < 1e-6, "Your implementation might went wrong!"
print "Passed!"

print "Testing the loss ...",
y = np.asarray([0, 5, 1, 4])
loss = loss_func.forward(scores, y)
dLoss = loss_func.backward()
correct_loss = 2.90181552716
assert abs(loss - correct_loss) < 1e-10, "Your implementation might went wrong!"
print "Passed!"

print "Testing the...


I am working on the file "2017_Fall/CSCI-599/Assignment01/Problem_1.ipynb".
The first task is:
Testing cascaded layers: FC + ReLU
Please find the TestFCReLU function in fully_conn.py under lib directory.

You only need to complete few lines of code in the TODO block.

Please design an FC --> ReLU two-layer-mini-network where the parameters of them match the given x, w, and b

Please insert the corresponding names you defined for each layer to param_name_w, and param_name_b respectively.

Here you only modify the param_name part, the _w, and _b are automatically assigned during network setup
Can you write Python code for it?

x = np.random.randn(2, 3, 4)  # the input features
w = np.random.randn(12, 10)   # the weight of fc layer
b = np.random.randn(10)       # the bias of fc layer
dout = np.random.randn(2, 10) # the gradients to the output, notice the shape

tiny_net = TestFCReLU()

tiny_net.net.assign("my_fc_w", w)
tiny_net.net.assign("my_fc_b", b)

out = tiny_net.forward(x)
dx = tiny_net.backward(dout)

dw = tiny_net.net.get_grads("my_fc_w")
db = tiny_net.net.get_grads("my_fc_b")

dx_num = eval_numerical_gradient_array(lambda x: tiny_net.forward(x), x, dout)
dw_num = eval_numerical_gradient_array(lambda w: tiny_net.forward(x), w, dout)
db_num = eval_numerical_gradient_array(lambda b: tiny_net.forward(x), b, dout)

# The errors should not be larger than 1e-7
print "dx error: ", rel_error(dx_num, dx)
print "dw error: ", rel_error(dw_num, dw)
print "db error: ", rel_error(db_num, db)


I am working on the file "2017_Fall/CSCI-599/Assignment01/Problem_1.ipynb".
The first task is:
Training a Network
In this section, we defined a TinyNet class for you to fill in the TODO block in fully_conn.py.
* Here please design a two layer fully connected network for this part.
* Please read the train.py under lib directory carefully and complete the TODO blocks in the train_net function first.
* In addition, read how the SGD function is implemented in optim.py, you will be asked to complete three other optimization methods in the later sections.
Can you write Python code for it?

# Arrange the data
data_dict = {
    "data_train": (data["data_train"], data["labels_train"]),
    "data_val": (data["data_val"], data["labels_val"]),
    "data_test": (data["data_test"], data["labels_test"])
}

data["data_train"].shape

set(data["labels_train"])

model = TinyNet()
loss_f = cross_entropy()
optimizer = SGD(model.net, 1e-4)


I am working on the file "2017_Fall/CSCI-599/Assignment01/Problem_1.ipynb".
The first task is:
Training a Network with Dropout
Run the following code blocks to compare the results with and without dropout
Can you write Python code for it?

# Train two identical nets, one with dropout and one without
num_train = 500
data_dict_500 = {
    "data_train": (data["data_train"][:num_train], data["labels_train"][:num_train]),
    "data_val": (data["data_val"], data["labels_val"]),
    "data_test": (data["data_test"], data["labels_test"])
}

solvers = {}
dropout_ps = [0, 0.25, 0.5, 0.75]  # you can try some dropout prob yourself

results_dict = {}
for dropout_p in dropout_ps:
    results_dict[dropout_p] = {}

for dropout_p in dropout_ps:
    print "Dropout =", dropout_p
    model = DropoutNetTest(dropout_p=dropout_p)
    loss_f = cross_entropy()
    optimizer = SGDM(model.net, 1e-4)
    results = train_net(data_dict_500, model, loss_f, optimizer, batch_size=100,
                        max_epochs=20, show_every=100, verbose=True)
    opt_params, loss_hist, train_acc_hist, val_acc_hist = results
    results_dict[dropout_p] = {
        "opt_params": opt_params,
        "loss_hist": loss_hist,
        "train_acc_hist": train_acc_hist,
        "val_acc_hist": val_acc_hist
    }

# Plot train and validation accuracies of the two models
train_accs = []
val_accs = []
for dropout_p in dropout_ps:
    curr_dict = results_dict[dropout_p]
    train_accs.append(curr_dict["train_acc_hist"][-1])
    val_accs.append(curr_dict["val_acc_hist"][-1])

plt.subplot(3, 1, 1)
for dropout_p in dropout_ps:
    curr_dict = results_dict[dropout_p]
    plt.plot(curr_dict["train_acc_hist"], 'o', label='%.2f dropout' % dropout_p)
plt.title('Train accuracy')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.legend(ncol=2, loc='lower right')

plt.subplot(3, 1, 2)
for dropout_p in dropout_ps:
    curr_dict = results_dict[dropout_p]
    plt.plot(curr_dict["val_acc_hist"], 'o', label='%.2f dropout' % dropout_p)
plt.title('Val accuracy')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.legend(ncol=2, loc='lower right')

plt.gcf().set_size_inches(15, 15)
plt.show()


I am working on the file "2017_Fall/CSCI-599/Assignment02/Problem_1.ipynb".
The first task is:
LSTM
Vanilla RNNs can be tough to train on long sequences due to vanishing and exploding gradiants. LSTMs solve this problem by replacing the simple update rule of the vanilla RNN with a gating mechanism as follows.
Similar to the vanilla RNN, at each timestep we receive an input $x_t\in\mathbb{R}^D$ and the previous hidden state $h_{t-1}\in\mathbb{R}^H$; what is different in the LSTM is to maintains an $H$-dimensional cell state, so we also receive the previous cell state $c_{t-1}\in\mathbb{R}^H$. The learnable parameters of the LSTM are an input-to-hidden matrix $W_x\in\mathbb{R}^{4H\times D}$, a hidden-to-hidden matrix $W_h\in\mathbb{R}^{4H\times H}$ and a bias vector $b\in\mathbb{R}^{4H}$.
At each timestep we first compute an activation vector $a\in\mathbb{R}^{4H}$ as $a=W_xx_t + W_hh_{t-1}+b$. We then divide this into four vectors $a_i,a_f,a_o,a_g\in\mathbb{R}^H$ where $a_i$ consists of the first $H$ elements of $a$, $a_f$ is the next $H$ elements of $a$, etc. We then compute the input gate $g\in\mathbb{R}^H$, forget gate $f\in\mathbb{R}^H$, output gate $o\in\mathbb{R}^H$ and block input $g\in\mathbb{R}^H$ as
$$
\begin{align}
i = \sigma(a_i) \hspace{2pc}
f = \sigma(a_f) \hspace{2pc}
o = \sigma(a_o) \hspace{2pc}
g = \tanh(a_g)
\end{align}
$$
where $\sigma$ is the sigmoid function and $\tanh$ is the hyperbolic tangent, both applied elementwise.
Finally we compute the next cell state $c_t$ and next hidden state $h_t$ as
$$
c_{t} = f\odot c_{t-1} + i\odot g \hspace{4pc}
h_t = o\odot\tanh(c_t)
$$
where $\odot$ is the elementwise product of vectors.
In the rest of the notebook we will implement the LSTM update rule and apply it to the text generation task.
In the code, we assume that data is stored in batches so that $X_t \in \mathbb{R}^{N\times D}$, and will work with transposed versions of the parameters: $W_x \in \mathbb{R}^{D \times 4H}$, $W_h \in \mathbb{R}^{H\times 4H}$ so that activations $A \in \mathbb{R}^{N\times 4H}$ can be computed efficiently as $A = X_t W_x + H_{t-1} W_h$
LSTM: step...
Can you write Python code for it?


I am working on the file "2017_Fall/CSCI-599/Assignment02/Problem_1.ipynb".
The first task is:
LSTM model
Now that you have implemented an LSTM, update the initialization of the TestRNN class in the file lib/rnn.py to handle the case where self.cell_type is lstm. This should require adding only one line of codes.
Once you have done so, run the following to check your implementation. You should see a difference of less than 1e-10.
Can you write Python code for it?

N, D, H = 10, 20, 40
V = 4
T = 13

model = TestRNN(D, H, cell_type='lstm')
loss_func = temporal_softmax_loss()

# Set all model parameters to fixed values
for k, v in model.params.items():
    model.params[k] = np.linspace(-1.4, 1.3, num=v.size).reshape(*v.shape)
model.assign_params()

features = np.linspace(-1.5, 0.3, num=(N * D * T)).reshape(N, T, D)
h0 = np.linspace(-1.5, 0.5, num=(N*H)).reshape(N, H)
labels = (np.arange(N * T) % V).reshape(N, T)

pred = model.forward(features, h0)

# You'll need this
mask = np.ones((N, T))

loss = loss_func.forward(pred, labels, mask)
dLoss = loss_func.backward()

expected_loss = 49.2140256354

print('loss: ', loss)
print('expected loss: ', expected_loss)
print('difference: ', abs(loss - expected_loss))


I am working on the file "2017_Fall/CSCI-599/Assignment02/Problem_1.ipynb".
The first task is:
LSTM: forward
In the class lstm in the file lib/layer_utils.py, implement the forward function to run an LSTM forward on an entire timeseries of data.
When you are done, run the following to check your implementation. You should see an error around 1e-7.
Can you write Python code for it?


I am working on the file "2017_Fall/CSCI-599/Assignment02/Problem_1.ipynb".
The first task is:
LSTM: step backward
Implement the backward pass for a single LSTM timestep in the function step_backward in the file lib/layer_utils.py. Once you are done, run the following to perform numeric gradient checking on your implementation. You should see errors around 1e-6 or less.
Can you write Python code for it?

np.random.seed(599)

N, D, H = 4, 5, 6

lstm = LSTM(D, H, init_scale=0.02, name='test_lstm')

x = np.random.randn(N, D)
prev_h = np.random.randn(N, H)
prev_c = np.random.randn(N, H)
Wx = np.random.randn(D, 4 * H)
Wh = np.random.randn(H, 4 * H)
b = np.random.randn(4 * H)

lstm.params[lstm.wx_name] = Wx
lstm.params[lstm.wh_name] = Wh
lstm.params[lstm.b_name] = b

next_h, next_c, cache = lstm.step_forward(x, prev_h, prev_c)

dnext_h = np.random.randn(*next_h.shape)
dnext_c = np.random.randn(*next_c.shape)

fx_h = lambda x: lstm.step_forward(x, prev_h, prev_c)[0]
fh_h = lambda h: lstm.step_forward(x, prev_h, prev_c)[0]
fc_h = lambda c: lstm.step_forward(x, prev_h, prev_c)[0]
fWx_h = lambda Wx: lstm.step_forward(x, prev_h, prev_c)[0]
fWh_h = lambda Wh: lstm.step_forward(x, prev_h, prev_c)[0]
fb_h = lambda b: lstm.step_forward(x, prev_h, prev_c)[0]

fx_c = lambda x: lstm.step_forward(x, prev_h, prev_c)[1]
fh_c = lambda h: lstm.step_forward(x, prev_h, prev_c)[1]
fc_c = lambda c: lstm.step_forward(x, prev_h, prev_c)[1]
fWx_c = lambda Wx: lstm.step_forward(x, prev_h, prev_c)[1]
fWh_c = lambda Wh: lstm.step_forward(x, prev_h, prev_c)[1]
fb_c = lambda b: lstm.step_forward(x, prev_h, prev_c)[1]

num_grad = eval_numerical_gradient_array

dx_num = num_grad(fx_h, x, dnext_h) + num_grad(fx_c, x, dnext_c)
dh_num = num_grad(fh_h, prev_h, dnext_h) + num_grad(fh_c, prev_h, dnext_c)
dc_num = num_grad(fc_h, prev_c, dnext_h) + num_grad(fc_c, prev_c, dnext_c)
dWx_num = num_grad(fWx_h, Wx, dnext_h) + num_grad(fWx_c, Wx, dnext_c)
dWh_num = num_grad(fWh_h, Wh, dnext_h) + num_grad(fWh_c, Wh, dnext_c)
db_num = num_grad(fb_h, b, dnext_h) + num_grad(fb_c, b, dnext_c)

dx, dh, dc, dWx, dWh, db = lstm.step_backward(dnext_h, dnext_c, cache)

print('dx error: ', rel_error(dx_num, dx))
print('dh error: ', rel_error(dh_num, dh))
print('dc error: ', rel_error(dc_num, dc))
print('dWx error: ', rel_error(dWx_num, dWx))
print('dWh error: ', rel_error(dWh_num, dWh))
print('db error: ', rel_error(db_num, db))


I am working on the file "2017_Fall/CSCI-599/Assignment02/Problem_1.ipynb".
The first task is:
Run the following cell to perform numeric gradient checking on the TestRNN class; you should errors around 1e-7 or less.
Can you write Python code for it?

np.random.seed(599)

batch_size = 2
timesteps = 3
input_dim = 4
hidden_dim = 6
label_size = 4

labels = np.random.randint(label_size, size=(batch_size, timesteps))
features = np.random.randn(batch_size, timesteps, input_dim)
h0 = np.random.randn(batch_size, hidden_dim)

model = TestRNN(input_dim, hidden_dim, cell_type='rnn')
loss_func = temporal_softmax_loss()

pred = model.forward(features, h0)

# You'll need this
mask = np.ones((batch_size, timesteps))

loss = loss_func.forward(pred, labels, mask)
dLoss = loss_func.backward()

dout, dh0 = model.backward(dLoss)

grads = model.grads

for param_name in sorted(grads):

    f = lambda _: loss_func.forward(model.forward(features, h0), labels, mask)
    param_grad_num = eval_numerical_gradient(f, model.params[param_name], verbose=False, h=1e-6)
    e = rel_error(param_grad_num, grads[param_name])
    print('%s relative error: %e' % (param_name, e))


I am working on the file "2017_Fall/CSCI-599/Assignment02/Problem_1.ipynb".
The first task is:
Simply run the following code block to check the loss and accuracy curve.
Can you write Python code for it?

opt_params, loss_hist, train_acc_hist = results

# Plot the learning curves
plt.subplot(2, 1, 1)
plt.title('Training loss')
loss_hist_ = loss_hist[1::100] # sparse the curve a bit
plt.plot(loss_hist_, '-o')
plt.xlabel('Iteration')

plt.subplot(2, 1, 2)
plt.title('Accuracy')
plt.plot(train_acc_hist, '-o', label='Training')
plt.xlabel('Epoch')
plt.legend(loc='lower right')
plt.gcf().set_size_inches(15, 12)

plt.show()


I am working on the file "2017_Fall/CSCI-599/Assignment02/Problem_1.ipynb".
The first task is:
Vanilla RNN: forward
Now that you have checked the forward and backward passes for a single timestep of a vanilla RNN, you will see how they are combined to implement a RNN that process an entire sequence of data.
In the VanillaRNN class in the file lib/layer_utils.py, check the function forward. We provide this function for you. This is implemented using the step_forward function that you defined above. After doing so run the following to check the implementation. You should see errors less than 1e-7.
Can you write Python code for it?


I am working on the file "2017_Fall/CSCI-599/Assignment02/Problem_1.ipynb".
The first task is:
We defined a LanguageModelRNN class for you to fill in the TODO block in rnn.py.
* Here design a recurrent neutral network consisting of a word_embedding layer, recurrent unit, and temporal fully connected layers so that they match the provided dimentions.
* Please read the train.py under lib directory carefully and complete the TODO blocks in the train_net function first.
Can you write Python code for it?

# you can change the following parameters.
D = 10 # input dimention
H = 20 # hidden space dimention
T = 50 # timesteps
N = 10 # batch size
max_epoch = 100 # max epoch size

loss_func = temporal_softmax_loss()
# you can change the cell_type between 'rnn' and 'lstm'.
model = LanguageModelRNN(dataSize, D, H, cell_type='rnn')
optimizer = Adam(model, 5e-4)

data = { 'data_train': data_labels, 'labels_train': gt_labels }

results = train_net(data, model, loss_func, optimizer, timesteps=T, batch_size=N, max_epochs=max_epoch, verbose=True)


I am working on the file "2017_Fall/CSCI-599/Assignment02/Problem_1.ipynb".
The first task is:
Word embedding: backward
Check the backward pass for the word embedding function in the function backward in the word_embedding class. We provide this function for you. After doing so run the following to numerically gradient check the implementation. You should see errors less than 1e-11.
Can you write Python code for it?

np.random.seed(599)

N, T, V, D = 50, 3, 5, 6

we = word_embedding(V, D, name="we")

x = np.random.randint(V, size=(N, T))
W = np.random.randn(V, D)

we.params[we.w_name] = W

out = we.forward(x)
dout = np.random.randn(*out.shape)
we.backward(dout)

dW = we.grads[we.w_name]

f = lambda W: we.forward(x)
dW_num = eval_numerical_gradient_array(f, W, dout)

print('dW error: ', rel_error(dW, dW_num))


I am working on the file "2017_Fall/CSCI-599/Assignment02/Problem_1.ipynb".
The first task is:
Word embedding: forward
In deep learning systems, we commonly represent words using vectors. Each word of the vocabulary will be associated with a vector, and these vectors will be learned jointly with the rest of the system.
In the file lib/layer_utils.py, check the function forward in the word_embedding class to convert words (represented by integers) into vectors. We provide this function for you. Run the following to check the implementation. You should see error around 1e-8.
Can you write Python code for it?


I am working on the file "2017_Fall/CSCI-599/Assignment02/Problem_1.ipynb".
The first task is:
simply run the following code to construct training dataset<br />
Can you write Python code for it?

import re

text = re.split(' |\n',input_text.value.lower()) # all words are converted into lower case
outputSize = len(text)
word_list = list(set(text))
dataSize = len(word_list)
output = np.zeros(outputSize)
for i in range(0, outputSize):
    index = np.where(np.asarray(word_list) == text[i])
    output[i] = index[0]
data_labels = output.astype(np.int)
gt_labels = data_labels[1:]
data_labels = data_labels[:-1]

print('Input text size: %s' % outputSize)
print('Input word number: %s' % dataSize)


I am working on the file "2018-02-01-TUT-DFW-Debugging.ipynb".
The first task is:
Catching Errors
Can you write Python code for it?

def sum_all_the_things():
    results = []
    import pdb; pdb.set_trace()
    with open('list_of_numbers.txt') as f:
        for number in f:
            try:
                results.append(number)
            except ValueError:
                pass

    print(f'The sum is {sum(results)}')

sum_all_the_things()


I am working on the file "2018-02-01-TUT-DFW-Debugging.ipynb".
The first task is:
Logging
Can you write Python code for it?

import logging

def main():
    logging.basicConfig(filename='sample.log', level=logging.INFO)

    logging.error('Start looping...')
    for item in range(10):
        item += 1
        logging.debug(f'item = {item}')
    logging.debug('...end looping.')

    if item == 10:
        print('Finished')
        logging.info('Finished')

    logging.info('~ Fin ~')

if __name__ == '__main__':
    main()


I am working on the file "2018-02-01-TUT-DFW-Debugging.ipynb".
The first task is:
Print
Can you write Python code for it?

def meh_function():

    for i in range(10):
        print(i)
        i += 2
        print(i)

    if i == 10:
        print('Finished')

meh_function()


I am working on the file "2018-02-01-TUT-DFW-Debugging.ipynb".
The first task is:
Use logging levels.
| Level | # |
|:---:|:---:|
| Critical | 50|
| Error | 40 |
| Warning | 30 |
| Info | 20 |
| Debug | 10 |
| Notset | 0 |
Can you write Python code for it?

import logging

def main():
    logging.basicConfig(filename='sample.log',
                        level=logging.INFO)

    logging.info('Start looping...')
    for item in range(10):
        item += 1
        logging.debug(f'item = {item}')
    logging.debug('...end looping.')

    if item == 10:
        print('Finished')
        logging.debug('Finished')

    logging.info('~ Fin ~')

if __name__ == '__main__':
    main()


I am working on the file "2018-03-06-Polytechnique-Jupyter/notebooks/08 - PyThreejs.ipynb".
The first task is:
Buffer Geometries
The PlainBufferGeometry object uses several tricks to speed up both the transfer of data and the rendering of the data.
Can you write Python code for it?

from pythreejs import *
import numpy as np
from IPython.display import display

vertices = np.asarray([
    [0, 0, 0],
    [0, 0, 1],
    [0, 1, 0],
    [0, 1, 1],
    [1, 0, 0],
    [1, 0, 1],
    [1, 1, 0],
    [1, 1, 1]
], dtype='float32')

faces = np.asarray([
    [0, 1, 3],
    [0, 3, 2],
    [0, 2, 4],
    [2, 6, 4],
    [0, 4, 1],
    [1, 4, 5],
    [2, 3, 6],
    [3, 7, 6],
    [1, 5, 3],
    [3, 5, 7],
    [4, 6, 5],
    [5, 6, 7]
], dtype='uint16').ravel()  # We need to flatten index array

vertexcolors = np.asarray([(0,0,0), (0,0,1), (0,1,0), (1,0,0),
                           (0,1,1), (1,0,1), (1,1,0), (1,1,1)], dtype='float32')

cubeGeometry = BufferGeometry(attributes=dict(
    position=BufferAttribute(vertices, normalized=False),
    index=BufferAttribute(faces, normalized=False),
    color=BufferAttribute(vertexcolors),
))

myobjectCube = Mesh(
    geometry=cubeGeometry,
    material=MeshLambertMaterial(vertexColors='VertexColors'),
    position=[-0.5, -0.5, -0.5]   # Center the cube
)
cCube = PerspectiveCamera(
    position=[3, 3, 3], fov=20,
    children=[DirectionalLight(color='#ffffff', position=[-3, 5, 1], intensity=0.5)])
sceneCube = Scene(children=[myobjectCube, cCube, AmbientLight(color='#dddddd')])

rendererCube = Renderer(camera=cCube, background='black', background_opacity=1,
                        scene = sceneCube, controls=[OrbitControls(controlling=cCube)])

display(rendererCube)


I am working on the file "2018-03-06-Polytechnique-Jupyter/notebooks/08 - PyThreejs.ipynb".
The first task is:
Clickable Surface
Can you write Python code for it?

# Generate surface data:
view_width = 600
view_height = 400
nx, ny = (20, 20)
xmax=1
x = np.linspace(-xmax, xmax, nx)
y = np.linspace(-xmax, xmax, ny)
xx, yy = np.meshgrid(x, y)
z = xx ** 2 - yy ** 2
#z[6,1] = float('nan')

# Generate scene objects from data:
surf_g = SurfaceGeometry(z=list(z[::-1].flat),
                         width=2 * xmax,
                         height=2 * xmax,
                         width_segments=nx - 1,
                         height_segments=ny - 1)

surf = Mesh(geometry=surf_g,
            material=MeshLambertMaterial(map=height_texture(z[::-1], 'YlGnBu_r')))

surfgrid = SurfaceGrid(geometry=surf_g, material=LineBasicMaterial(color='black'),
                       position=[0, 0, 1e-2])  # Avoid overlap by lifting grid slightly

# Set up picking bojects:
hover_point = Mesh(geometry=SphereGeometry(radius=0.05),
                   material=MeshLambertMaterial(color='hotpink'))

click_picker = Picker(controlling=surf, event='dblclick')
hover_picker = Picker(controlling=surf, event='mousemove')

# Set up scene:
key_light = DirectionalLight(color='white', position=[3, 5, 1], intensity=0.4)
c = PerspectiveCamera(position=[0, 3, 3], up=[0, 0, 1], aspect=view_width / view_height,
                      children=[key_light])

scene = Scene(children=[surf, c, surfgrid, hover_point, AmbientLight(intensity=0.8)])

renderer = Renderer(camera=c, scene=scene,
                    width=view_width, height=view_height,
                    controls=[OrbitControls(controlling=c), click_picker, hover_picker])

# Set up picking responses:
# Add a new marker when double-clicking:
out = Output()
def f(change):
    value = change['new']
    with out:
        print('Clicked on %s' % (value,))
    point = Mesh(geometry=SphereGeometry(radius=0.05),
                 material=MeshLambertMaterial(color='red'),
                 position=value)
    scene.add(point)

click_picker.observe(f, names=['point'])

# Have marker follow picker point:
link((hover_point, 'position'), (hover_picker, 'point'))

# Show...


I am working on the file "2018-03-06-Polytechnique-Jupyter/notebooks/08 - PyThreejs.ipynb".
The first task is:
Design our own texture
Can you write Python code for it?

import numpy as np
from scipy import ndimage
import matplotlib
import matplotlib.pyplot as plt
from skimage import img_as_ubyte

jet = matplotlib.cm.get_cmap('jet')

np.random.seed(int(1)) # start random number generator
n = int(5) # starting points
size = int(32) # size of image
im = np.zeros((size,size)) # create zero image
points = size*np.random.random((2, n**2)) # locations of seed values
im[(points[0]).astype(np.int), (points[1]).astype(np.int)] = size # seed high values
im = ndimage.gaussian_filter(im, sigma=size/(float(4)*n)) # smooth high values into surrounding areas
im *= 1/np.max(im)# rescale to be in the range [0,1]
rgba_im = img_as_ubyte(jet(im)) # convert the values to rgba image using the jet colormap

t = DataTexture(data=rgba_im, format='RGBAFormat', width=size, height=size)

geometry = SphereGeometry(radius=1, widthSegments=16, heightSegments=10)#TorusKnotGeometry(radius=2, radialSegments=200)
material = MeshLambertMaterial(map=t)

myobject = Mesh(geometry=geometry, material=material)
c = PerspectiveCamera(position=[0, 3, 3], fov=40,
                      children=[DirectionalLight(color='#ffffff', position=[3, 5, 1], intensity=0.5)])
scene = Scene(children=[myobject, c, AmbientLight(color='#777777')])

renderer = Renderer(camera=c, scene = scene, controls=[OrbitControls(controlling=c)], width=400, height=400)
display(renderer)


I am working on the file "2018-03-06-Polytechnique-Jupyter/notebooks/08 - PyThreejs.ipynb".
The first task is:
Simple sphere and text
Can you write Python code for it?

ball = Mesh(geometry=SphereGeometry(radius=1),
            material=MeshLambertMaterial(color='red'),
            position=[2, 1, 0])

c = PerspectiveCamera(position=[0, 5, 5], up=[0, 1, 0],
                      children=[DirectionalLight(color='white', position=[3, 5, 1], intensity=0.5)])

scene = Scene(children=[ball, c, AmbientLight(color='#777777')])

renderer = Renderer(camera=c,
                    scene=scene,
                    controls=[OrbitControls(controlling=c)])
display(renderer)

ball.geometry.radius = 0.5

import time, math
ball.material.color = '#4400dd'
for i in range(1, 150, 2):
    ball.geometry.radius = i / 100.
    ball.position = [math.cos(i / 10.), math.sin(i / 50.), i / 100.]
    time.sleep(.05)


I am working on the file "2018-04_Stock_prediction/linear regression stock prediction project.ipynb".
The first task is:
Use PCA to reduce the number of features to two
Trying to use PCA to processing the features and test accuracy.
Can you write Python code for it?

# Use PCA to reduce the number of features to two, and test.
from sklearn.decomposition import PCA
#reduce 4 featrues to 2
pca = PCA(n_components=2)
predictor3=pca.fit_transform(predictor3)
print(predictor3.shape)
clf4 = linear_model.LinearRegression(n_jobs=-1)
X_train4, X_test4, y_train4, y_test4 =train_test_split(predictor3 , fin_price1['price'], test_size=0.1,shuffle= False)
clf4.fit(X_train4, y_train4)
forecast_set4 = clf4.predict(X_test4)
confidence3=clf4.score(X_test4, y_test4)
print('confident:',confidence3)
print("Mean squared error:",mean_squared_error(y_test4, forecast_set4))
print('After use PCA, the prediction is worse.')

num_samples4 = fin_price1.shape[0]
#add Forecase column to dataframe
fin_price1['Forecast2'] = np.nan
fin_price1['Forecast2'][int(0.9*num_samples3):num_samples3]=forecast_set4
style.use('ggplot')
fin_price1['price'].plot()
fin_price1['Forecast2'].plot()
plt.legend(loc=4)
plt.xlabel('Date')
plt.ylabel('Price')
plt.rcParams['figure.figsize'] = (20,20)
plt.show()


I am working on the file "2018-04_Stock_prediction/linear regression stock prediction project.ipynb".
The first task is:
Use simple stock market daily data as features
Can you write Python code for it?

#get stock basic data from quandl
df = quandl.get('WIKI/AAPL',start_date="1996-9-26",end_date='2017-12-31')

df = df[['Adj. Open','Adj. High','Adj. Low','Adj. Close','Adj. Volume']]
#calculate highest and lowest price change
df['HL_PCT']=(df['Adj. High']-df['Adj. Low'])/df['Adj. Close'] *100.0
#calculate return of stock price
df['PCT_change']= (df['Adj. Close']-df['Adj. Open'])/df['Adj. Open'] *100.0

df = df[['Adj. Close','HL_PCT','PCT_change','Adj. Volume']]
df_orig=df
date = df.index
df.head()

#plot heat map of corrlation
corr_stocks=df.corr()
corr_stocks=np.absolute(corr_stocks)
print(corr_stocks)
plt.figure(figsize=(12, 10))
plt.imshow(corr_stocks, cmap='RdYlGn', interpolation='none', aspect='auto')
plt.xticks(range(len(corr_stocks)), corr_stocks.columns, rotation='vertical')
plt.yticks(range(len(corr_stocks)), corr_stocks.columns);
plt.suptitle('Stock Correlations Heat Map', fontsize=15, fontweight='bold')
plt.show()
print('-------------------------------------------------')
print('From the correlation heat map, we can tell that the corrlation bewteen percentage change column and price is')
print('very low. So we need to get rid of this column to predict.')

#get rid of feature have least correlation
df = df[['Adj. Close','HL_PCT','Adj. Volume']]

from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
#use high low price change and volume as two features
predictor=df[['HL_PCT','Adj. Volume']]
#normalize the predictor
predictor=preprocessing.scale(predictor)
price=df['Adj. Close']
predictor=np.array(predictor)
price=np.array(price)
#using 90% as training data and 10% as testing data
X_train, X_test, y_train, y_test =train_test_split(predictor , price, test_size=0.1,shuffle= False)
clf = linear_model.LinearRegression(n_jobs=-1)
clf.fit(X_train, y_train)
y_pred1 = clf.predict(X_test)
print('the coefficient of determination R^2 of the prediction:',clf.score(X_test, y_test))
print("Mean squared error:",mean_squared_error(y_test, y_pred1))


I am working on the file "2018-04_Stock_prediction/linear regression stock prediction project.ipynb".
The first task is:
the first varible is negative because the model can be arbitrarily worse
Can you write Python code for it?

forecast_set = clf.predict(X_test)
num_samples = df.shape[0]
#add Forecase column to dataframe
df['Forecast'] = np.nan
df['Forecast'][int(0.9*num_samples):num_samples]=forecast_set

#plot graph for actual stock price and
style.use('ggplot')
df['Adj. Close'].plot()
df['Forecast'].plot()
plt.legend(loc=4)
plt.xlabel('Date')
plt.ylabel('Price')
plt.rcParams['figure.figsize'] = (20,20)
plt.show()
print('-------------------------')
print('from predicion graph we can see that the prediction does not work well')


I am working on the file "2018-04_Stock_prediction/linear regression stock prediction project.ipynb".
The first task is:
use price as one of the feature to predict price
We can see from previous prediction, even high related correlation featrues did not predict well. However, the highest related featrue is price itself. This time I want to use price as one of the feature to predict price.
Can you write Python code for it?

predictor2=df[['Adj. Close','HL_PCT','Adj. Volume']]
predictor2=preprocessing.scale(predictor2)
clf2 = linear_model.LinearRegression(n_jobs=-1)
X_train2, X_test2, y_train2, y_test2 =train_test_split(predictor2 , price, test_size=0.1,shuffle= False)
clf2.fit(X_train2, y_train2)
forecast_set2 = clf2.predict(X_test2)
print('the coefficient of determination R^2 of the prediction:',clf2.score(X_test2, y_test2))
print("Mean squared error:",mean_squared_error(y_test, forecast_set2))
print('Mean squared error is almost 0, the prediction is very well.')

num_samples = df.shape[0]
#add Forecase column to dataframe
df['Forecast'] = np.nan
df['Forecast'][int(0.9*num_samples):num_samples]=forecast_set2
style.use('ggplot')
df['Adj. Close'].plot()
df['Forecast'].plot()
plt.legend(loc=4)
plt.xlabel('Date')
plt.ylabel('Price')
plt.rcParams['figure.figsize'] = (20,20)
plt.show()
print('-------------------------')
print('from predicion graph we can see that prediction works well.')


I am working on the file "2018-06-18-IFPEN-Tech/notebooks/02 - Widgets.ipynb".
The first task is:
Interactive Jupyter widgets
Can you write Python code for it?

slider = FloatSlider(
    value=7.5,
    min=5.0,
    max=10.0,
    step=0.1,
    description='Input:',
)

slider

slider

slider.value

text = FloatText(description='Value')
dlink((slider, 'value'), (text, 'value'))
text

slider