I am working on the file "BDS_Lab09_FILL_IN-plots.ipynb".
The first task is:
1.2:  Clean the data! Make sure the station- and vehicle-identifiers are in the right format. A station identifier consists of 9 characters (prefix = '00') and a vehicle identifier consists of the concatentation of the vehicle type (IC/L/S/P/...) and the line identifier. Try to fix as much of the records as possible, drop only the unfixable ones. How many records did you drop?
2. Exploratory Data Analysis (EDA)
Let's create some visualisations of our data in order to gain some insights. Which features are useful, which ones aren't?
We will create 3 visualisations:
* Pie chart of the class distribution
* Stacked Bar Chart depicting the distribution for one aggregated variable (such as the weekday or the vehicle type)
* Scattter plot depicting the 'crowdiness' of the stations in Belgium
For each of the visualisations, code to generate the plot has already been handed to you. You only need to prepare the data (i.e. create a new dataframe or select certain columns) such that it complies with the input specifications. If you want to create your own plotting code or extend the given code, you are free to do so!
 2.1:  *Create a pie_chart with the distribution of the different classes. Have a look at our webscraping lab for plotting pie charts. TIP: the value_counts() does most of the work for you!
Can you write Python code for it?

fig, ax = plt.subplots(1,1, figsize=(5,5))
training['post.class'].value_counts().plot(kind='pie', ax=ax, autopct='%1.1f%%')
#we have a lot of null/undefined, especially in our test set, we can't simply throw them away


I am working on the file "BDS_Lab09_FILL_IN-plots.ipynb".
The first task is:
2.3:  *Create a stacked_bar_chart with the distribution of the three classes over an aggregated variable (group the data by weekday, vehicle_type, ...). More info on creating stacked bar charts can be found here: http://pandas.pydata.org/pandas-docs/stable/visualization.html#bar-plots

The dataframe you need will require your grouping variables as the index, and 1 column occupancy category, for example:

| Index | Occupancy_Low  | Occupancy_Medium | Occupancy_High | Sum_Occupancy |
|-------|----------------|------------------|----------------|---------------|
| IC    | 15             | 30               | 10             | 55 |
| S     | 20             | 10               | 30             | 60 |
| L     | 12             | 9                | 14             | 35 |

If you want the values to be relative (%), add a sum column and use it to divide the occupancy columns
Can you write Python code for it?

training[0:1]

occup = pd.crosstab(training['post.class'], training['post.occupancy'])
weekday = pd.crosstab(training['post.class'], training['weekday'])

occup = occup.drop(['BUS', 'EUR', 'EXT', 'ICE', 'ICT', 'P', 'TGV', 'THA', 'TRN', 'ic', 'null', ''])
occup = occup.apply(lambda r: r/r.sum(), axis=1)
occup[0:4]
weekday = weekday.drop(['BUS', 'EUR', 'EXT', 'ICE', 'ICT', 'P', 'TGV', 'THA', 'TRN', 'ic', 'null', ''])
weekday = weekday.apply(lambda r: r/r.sum(), axis=1)

df_occup = pd.DataFrame(occup)
df_occup.plot.bar(stacked=True);
df_weekday = pd.DataFrame(weekday)
df_weekday.plot.bar(stacked=True);


I am working on the file "BDS_Lab09_FILL_IN-plots.ipynb".
The first task is:
2.4:  * To have an idea about the hotspots in the railway network make a scatter plot that depicts the number of visitors per station. Aggregate on the destination station and use the GTFS dataset at iRail to find the geolocation of the stations (stops.txt): https://gtfs.irail.be/nmbs
Can you write Python code for it?

stops = stations_df[['URI','longitude','latitude']]
dest_count = training['post.to'].value_counts()
dest_count_df = pd.DataFrame({'id':dest_count.index, 'count':dest_count.values})
dest_loc = pd.merge(dest_count_df, stops, left_on = 'id', right_on = 'URI')
dest_loc = dest_loc[['id', 'count', 'latitude','longitude']]

fig, ax = plt.subplots(figsize=(12,10))
ax.scatter(dest_loc.longitude, dest_loc.latitude, s=dest_loc['count'] )


I am working on the file "BDS_Lab09_FILL_IN-plots.ipynb".
The first task is:
4. 'Advanced' predictive modeling: model selection & hyper-parameter tuning
Model evaluation and hyper-parameter tuning
In order to evaluate your model, K-fold cross-validation (https://en.wikipedia.org/wiki/Cross-validation_(statistics) ) is often applied. Here, the data is divided in K chunks, K-1 chunks are used for training while 1 chunk is used for testing. Different metrics exist, such as accuracy, AUC, F1 score, and more. For this lab, we will use accuracy.
Some machine learning techniques, supported by sklearn:

SVMs
Decision Trees
Decision Tree Ensemble: AdaBoost, Random Forest, Gradient Boosting
Multi-Level Perceptrons/Neural Networks
Naive Bayes
K-Nearest Neighbor
...

To tune the different hyper-parameters of a machine learning model, again different techniques exist:
* Grid search: exhaustively try all possible parameter combinations (Code to tune the different parameters of our LogReg model has been given)
* Random search: try a number of random combinations, it has been shown that this is quite equivalent to grid search
 4.1:  *Choose one or more machine learning techniques, different from Logistic Regression and apply them to our data, with tuned hyper-parameters! You will see that switching techniques in sklearn is really simple! Which model performs best on this data? *
Can you write Python code for it?

skf = StratifiedKFold(n_splits=5, random_state=1337)
X = training[['seconds_since_midnight', 'month']]
y = training['occupancy']
cms = []
accs = []
parameters = {#'penalty': ['l1', 'l2'],  # No penalty tuning, cause 'l1' is only supported by liblinear
              # It can be interesting to manually take a look at 'l1' with 'liblinear', since LASSO
              # provides sparse solutions (boils down to the fact that LASSO does some feature selection for you)
              'solver': ['newton-cg', 'lbfgs', 'liblinear', 'sag'],
              'tol': [1e-4, 1e-6, 1e-8],
              'C': [1e-2, 1e-1, 1.0, 1e1],
              'max_iter': [1e2, 1e3]
             }
for train_index, test_index in skf.split(X, y):
    X_train, X_test = X.iloc[train_index, :], X.iloc[test_index, :]
    y_train, y_test = y[train_index], y[test_index]

    tuned_log_reg = GridSearchCV(LogisticRegression(penalty='l2'), parameters, cv=3,
                       scoring='accuracy')
    tuned_log_reg.fit(X_train, y_train)
    print(tuned_log_reg.best_params_)

    predictions = tuned_log_reg.predict(X_test)
    cm = confusion_matrix(y_test, predictions)
    cms.append(cm)
    accs.append(accuracy_score(y_test, predictions))
    print(classification_report(y_test, predictions))

print('Confusion matrix:\n', np.mean(cms, axis=0))
print('Avg accuracy', np.mean(accs), '+-', np.std(accs))
print('Predict all lows', float(len(y[y == 0]))/float(len(y)))


I am working on the file "BDS_Lab09_FILL_IN-plots.ipynb".
The first task is:
5. Data augmentation with external data sources
There is a unlimited amount of factors that influence the occupancy of a train! Definitely more than the limited amount of data given in the feedback logs. Therefore, we will try to create new features for our dataset using external data sources. Examples of data sources include:

Weather APIs
A holiday calendar
Event calendars
Connection and delay information of the SpitsGidsAPI
Data from the NMBS/SNCB
Twitter and other social media
many, many more

In order to save time, a few 'prepared' files have already been given to you. Of course, you are free to scrape/generate your own data as well:

Hourly weather data for all stations in Belgium, from August till April weather_data.zip
A file which contains the vehicle identifiers and the stations where this vehicle stops  line_info.csv
Based on this line_info, you can construct a graph of the rail net in Belgium and apply some fancy graph features (pagerank, edge betweenness, ...) iGraph experiments.ipynb
A file containing the coordinates of a station, and the number of visitors during week/weekend for 2015  station_druktes.csv
A file with some of the holidays (this can definitely be extended) holidays.json
For event data, there is the Eventful API

 5.1:  Pick one (or more) external data source(s) and link your current data frame to that data source (requires some creativity in most cases). Extract features from your new, linked data source and re-train your model. How much gain did you achieve?
Als we kijken naar "training.id.value_counts()" dan zien we vooral dat het om studenten bestemmingen gaat, misschien komt dat omdat het vooral hen zijn die deze app gebruiken? We moeten dus nadenken wanneer zij de trein nemen, en wat dat kan beinvloeden. Misschien het aantal studenten per station incorporeren?
Can you write Python code for it?

holiday_pops = pd.read_json('data/holidays.json')
holidays = pd.read_json( (holiday_pops['holidays']).to_json(), orient='index')

holidays['date'] = pd.to_datetime(holidays['date'])
holidays.head(1)

training["date"] = training["querytime"].values.astype('datetime64[D]')
out_test["date"] = out_test["querytime"].values.astype('datetime64[D]')

training_holidays = pd.merge(training,holidays, how="left", on='date')
training_holidays.school = training_holidays.school.fillna(0)
training_holidays.name = training_holidays.name.fillna("geen")
training_holidays[0:1]

out_test_holidays = pd.merge(out_test,holidays, how="left", on='date')
out_test_holidays.school = out_test_holidays.school.fillna(0)
out_test_holidays.name = out_test_holidays.name.fillna("geen")
out_test_holidays[0:1]

from sklearn.preprocessing import LabelEncoder
encoder = LabelEncoder()
#encode the names from the holidays (Summer,Christmas...)
training_holidays["name_enc"] = encoder.fit_transform(training_holidays["name"])
out_test_holidays["name_enc"] = encoder.fit_transform(out_test_holidays["name"])
#encode the classes (IC,TGV,L...)
training_holidays["class_enc"] = encoder.fit_transform(training_holidays["post.class"])
out_test_holidays["class_enc"] = encoder.fit_transform(out_test_holidays["post.class"])

training_holidays=training_holidays.rename(columns = {'too':'destination'})
out_test_holidays=out_test_holidays.rename(columns = {'too':'destination'})

stations_df = pd.merge(stations_df,druktes_df.drop(['Unnamed: 0','station'],1), left_on = 'name', right_on = 'station_link')


I am working on the file "BDS_Lab09_FILL_IN-plots.ipynb".
The first task is:
6. Generating a Kaggle submission and comparing your methodology to others
 6.1:  Train your best performing model on train.nldjson and generate predictions for test.nldjson. Create a file called submission.csv with format listed below and submit it on the Kaggle competition! What's your position on the leaderboard?
Can you write Python code for it?

pickle.dump(training_holidays_druktes,open("temp_data/training_holidays_druktes.pkl","wb"))
pickle.dump(out_test_holidays_druktes,open("temp_data/out_test_holidays_druktes.pkl","wb"))

training_holidays_druktes = pd.read_pickle("temp_data/training_holidays_druktes.pkl")
out_test_holidays_druktes = pd.read_pickle("temp_data/out_test_holidays_druktes.pkl")

training_holidays_druktes[0:5]


I am working on the file "BDS_Lab09_FILL_IN-plots.ipynb".
The first task is:
Processing the data
Cleaning the variables, and adding station variables to our dataframe
Can you write Python code for it?

training['querytime'] = pd.to_datetime(training['querytime'])
out_test['querytime'] = pd.to_datetime(out_test['querytime'])

training = training.dropna()
training['post.occupancy'] = training['post.occupancy'].apply(lambda x: x.split("http://api.irail.be/terms/",1)[1])

training['post.vehicle'] = training['post.vehicle'].apply(lambda x: x.split("http://irail.be/vehicle/",1)[1])
out_test['post.vehicle'] = out_test['post.vehicle'].apply(lambda x: x.split("http://irail.be/vehicle/",1)[1])
#create class column, eg IC058 -> IC
training['post.class'] = training['post.vehicle'].apply(lambda x: " ".join(re.findall("[a-zA-Z]+", x)))
out_test['post.class'] = out_test['post.vehicle'].apply(lambda x: " ".join(re.findall("[a-zA-Z]+", x)))
#reset the index because you have duplicate indexes now because you appended DFs in a for loop
training = training.reset_index()

stations_df = pd.read_csv('data/stations.csv')
stations_df['from'] = stations_df.index
stations_df['destination'] = stations_df['from']
stations_df[0:4]
#post.from en post.to are in the some format of URI

stations_df["zoekterm"]=stations_df["name"]+" trein"

stations_df.loc[stations_df['zoekterm'].str.startswith("Zaventem"), "zoekterm"] = "Zaventem trein"
stations_df.loc[stations_df['zoekterm'].str.startswith("Charleroi"), "zoekterm"] = "Charleroi trein"
stations_df.loc[stations_df['zoekterm'].str.startswith("Brussel"), "zoekterm"] = "Brussel trein"
stations_df.loc[stations_df['zoekterm'].str.startswith("Gent"), "zoekterm"] = "Gent trein"
stations_df.loc[stations_df['zoekterm'].str.startswith("Liège"), "zoekterm"] = "Luik trein"
stations_df.loc[stations_df['zoekterm'].str.startswith("Antwerpen"), "zoekterm"] = "Antwerpen trein"

druktes_df = pd.read_csv('data/station_druktes.csv')
druktes_df[0:4]

training = pd.merge(training,stations_df[["URI","from"]], left_on = 'post.from', right_on = 'URI')
training = pd.merge(training,stations_df[["URI","destination"]], left_on = 'post.to', right_on = 'URI')
training = training.drop(['URI_y','URI_x'],1)

out_test =...


I am working on the file "BDS_Lab09_FILL_IN-plots.ipynb".
The first task is:
Since we have a lot of 'Null' (+-1/3th) values for our 'class' feature, and we don't want to throw that away, we can try to predict these labels based on the other features, we get +75% accuracy so that seems sufficient. But we can't forgot to do the same thing for the test set!
Can you write Python code for it?

training_class = training_holidays[training_holidays.class_enc != 0]
training_class = training_class[training_class.class_enc != 14]
test_class = training_holidays[(training_holidays.class_enc == 0)|(training_holidays.class_enc == 14)]

training_class["class_pred"]=training_class["class_enc"]
training_holidays_enc = pd.concat([training_class,test_class])

X_train = training_class[['seconds_since_midnight','weekday', 'month','id','id_2']]
X_test = test_class[['seconds_since_midnight','weekday', 'month','id','id_2']]
y_train = training_class['class_enc']

train.occupancy.value_counts()/train.shape[0]

test.occupancy.value_counts()/test.shape[0]

out_test_holidays_druktes.occupancy.value_counts()/out_test_holidays_druktes.shape[0]

from sklearn.linear_model import SGDClassifier
from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier
from sklearn.metrics import f1_score
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import VotingClassifier
from sklearn.model_selection import RandomizedSearchCV, GridSearchCV

from sklearn.cross_validation import train_test_split
train, test = train_test_split(training_holidays_druktes, test_size=0.2, random_state=42)

X_train = train[['seconds_since_midnight','drukte_from','drukte_to','school','name_enc','day__0','day__1','day__2','day__3','day__4','day__5','day__6','from_lat','from_lng','des_lat','des_lng','real_trend','class__0','class__1','class__2','class__3','class__4','class__5','class__6','class__7','class__8', 'temperature', 'humidity', 'windspeed', 'weather_type', 'visibility']]
X_test = test[['seconds_since_midnight','drukte_from','drukte_to','school','name_enc','day__0','day__1','day__2','day__3','day__4','day__5','day__6','from_lat','from_lng','des_lat','des_lng','real_trend','class__0','class__1','class__2','class__3','class__4','class__5','class__6','class__7','class__8', 'temperature', 'humidity', 'windspeed', 'weather_type', 'visibility']]
y_train = train['occupancy']
y_test = test['occupancy']
#month uit de set halen als...


I am working on the file "BDS_Lab09_FILL_IN-plots.ipynb".
The first task is:
Transform all null classes to one null class, maybe try to predict the class? Based on to and from and time
Can you write Python code for it?

def transform_druktes(row):
    start = row['from']
    destination = row['destination']
    day = row['weekday']
    row['from_lat']=stations_df[stations_df["from"] == start]["latitude"].values[0]
    row['from_lng']=stations_df[stations_df["destination"] == destination]["longitude"].values[0]
    row['des_lat']=stations_df[stations_df["from"] == start]["latitude"].values[0]
    row['des_lng']=stations_df[stations_df["destination"] == destination]["longitude"].values[0]
    row['zoekterm']=stations_df[stations_df["destination"] == destination]["zoekterm"].values[0]
    if day == 5:
        row['drukte_from']=stations_df[stations_df["from"] == start]["zaterdag"].values[0]
        row['drukte_to']=stations_df[stations_df["destination"] == destination]["zaterdag"].values[0]
    elif day == 6:
        row['drukte_from']=stations_df[stations_df["from"] == start]["zondag"].values[0]
        row['drukte_to']=stations_df[stations_df["destination"] == destination]["zondag"].values[0]
    elif day == 4:
        row['drukte_from']=stations_df[stations_df["from"] == start]["zondag"].values[0]/5.0*1.11
        row['drukte_to']=stations_df[stations_df["destination"] == destination]["zondag"].values[0]/5.0*1.11
    elif day == 3:
        row['drukte_from']=stations_df[stations_df["from"] == start]["zondag"].values[0]/5.0*1.21
        row['drukte_to']=stations_df[stations_df["destination"] == destination]["zondag"].values[0]/5.0*1.21
    elif day == 2:
        row['drukte_from']=stations_df[stations_df["from"] == start]["zondag"].values[0]/5.0*0.736
        row['drukte_to']=stations_df[stations_df["destination"] == destination]["zondag"].values[0]/5.0*0.736
    elif day == 1:
        row['drukte_from']=stations_df[stations_df["from"] == start]["zondag"].values[0]/5.0*0.92
        row['drukte_to']=stations_df[stations_df["destination"] == destination]["zondag"].values[0]/5.*0.92
    else:
        row['drukte_from']=stations_df[stations_df["from"] == start]["week"].values[0]/5.0*1.016
       ...


I am working on the file "BEL to Natural Language.ipynb".
The first task is:
Conversion
The PyBEL BELGraph instance is converted to INDRA statments with the function process_pybel_graph. It returns an instance of PybelProcessor, which stores the INDRA statments.
Can you write Python code for it?


I am working on the file "BEL to Natural Language.ipynb".
The first task is:
Data
The Sialic Acid graph is used as an example.
Can you write Python code for it?


I am working on the file "BEL to Natural Language.ipynb".
The first task is:
Dependencies
Can you write Python code for it?


I am working on the file "BEL to Natural Language.ipynb".
The first task is:
The list of INDRA statements is converted to plain english using the EnglishAssembler.
Can you write Python code for it?

asm = EnglishAssembler(stmts)
print(asm.make_model(), sep='\n')


I am working on the file "BEM_problem.ipynb".
The first task is:
BEM function solution
Generally, the influence of all the j panels on the i BE node can be expressed as follows:
\begin{matrix}
{{c}{ij}}{{p}{i}}+{{p}{i}}\int{{{s}{j}}}{{{H}{ij}}d{{s}{j}}}=({{v}{i}}\cdot \mathbf{n})\int_{{{s}{j}}}{{{G}{ij}}}d{{s}_{j}}
\end{matrix}
Applying boundary condition along the boundary on above equation, a linear systsem can be constructed as follows:
\begin{matrix}
\left[ {{{{H}'}}{ij}} \right]\left[ {{P}{i}} \right]=\left[ {{G}{ij}} \right]\left[ {{v}{i}}\cdot \mathbf{n} \right]
\end{matrix}
!!!!!MY IMPLEMENTATION MAY HAS SOME PROBLEM HERE!!!!!!
All the integration solution can be evaluated except on itself. Where,
$
\left[ {{{{H}'}}{ij}} \right]=\left{ \begin{matrix}
   \begin{matrix}
   {{H}{ij}} & i\ne j  \
\end{matrix}  \
   \begin{matrix}
   {{H}_{ij}}+\frac{1}{2} & i=j  \
\end{matrix}  \
\end{matrix} \right.
$

Figure 3. Representation of coordinate systems and the principle of superstition with well source and boundary element source
As shown in Fig.3, the pressure and velocity at any point i in the local gridblock can be determined using Eqs. below. Applying principle of superposition for each BE node along the boundary (Fig. 3), boundary condition can be written as follows:
\begin{matrix}
   {{P}{i}}(s)=\sum\limits{j=1}^{M}{{{B}{ij}}{{Q}{j}}} & \text{constant pressure boundary}  \
\end{matrix}
\begin{matrix}
   {{v}{i}}(s)\cdot {{\mathbf{n}}{i}}=\sum\limits_{j=1}^{M}{{{A}{ij}}{{Q}{j}}} & \text{constant flux boundary}  \
\end{matrix}
The Pi and v ·n are the konwn boundary codition. The flow rate(strength) of boundary elements in Hij and Gij are the only unknown terms.
So we could rearrange the matrix above as linear system:
$
{{\left[ \begin{matrix}
   {{A}{ij}}  \
   {{B}{ij}}  \
\end{matrix} \right]}{N\times N}}{{\left[ \begin{matrix}
   {{Q}{j}}  \
   {{Q}{j}}  \
\end{matrix} \right]}{N\times 1}}={{\left[ \begin{matrix}
   -{{u}{i}}\sin {{\alpha }{i}}+{{v}{i}}\cos {{\alpha }{i}} ...
Can you write Python code for it?

def build_matrix(panels):
    """Builds the source matrix.

    Arguments
    ---------
    panels -- array of panels.

    Returns
    -------
    A -- NxN matrix (N is the number of panels).
    """
    N = len(panels)
    A = numpy.empty((N, N), dtype=float)
    #numpy.fill_diagonal(A, 0.5)

    for i, p_i in enumerate(panels): #target nodes
        for j, p_j in enumerate(panels): #BE source
            #if i != j:   ###Matrix construction
                if i>=0 and i<Nbd or i>=3*Nbd and i<4*Nbd:
                    A[i,j] = -p_j.sinalpha*InflueceU(p_i.xc, p_i.yc, p_j)+p_j.cosalpha*InflueceV(p_i.xc, p_i.yc, p_j)
                    #A[i,j] = InflueceP(p_i.xc, p_i.yc, p_j)
                if i>=Nbd and i<2*Nbd or i>=2*Nbd and i<3*Nbd:
                    A[i,j] = -p_j.sinalpha*InflueceU(p_i.xc, p_i.yc, p_j)+p_j.cosalpha*InflueceV(p_i.xc, p_i.yc, p_j)
                    #A[i,j] = InflueceP(p_i.xc, p_i.yc, p_j)

    return A

def build_rhs(panels):
    """Builds the RHS of the linear system.

    Arguments
    ---------
    panels -- array of panels.

    Returns
    -------
    b -- 1D array ((N+1)x1, N is the number of panels).
    """
    b = numpy.empty(len(panels), dtype=float)

    for i, panel in enumerate(panels):
        V_well=( -panel.sinalpha*Qwell_1*InflueceU_W(panel.xc, panel.yc, wells[0])+panel.cosalpha*Qwell_1*InflueceV_W(panel.xc, panel.yc, wells[0]) )
        if i>=0 and i<Nbd:
            b[i]=0+V_well
            #b[i]=4000
            #b[i]=84
        if i>=Nbd and i<2*Nbd:
            b[i]=-V_well
            #b[i]=-42
        if i>=2*Nbd and i<3*Nbd:
            b[i]=-V_well
            #b[i]=-42
        if i>=3*Nbd and i<4*Nbd:
            b[i]=0+V_well
            #b[i]=84
    return b

#Qwell_1=300   #Flow rate of well 1
#Boundary_V=-227 #boundary velocity ft/day

A = build_matrix(panels)                    # computes the singularity matrix
b = build_rhs(panels)                       # computes the freestream RHS

# solves the linear...


I am working on the file "BEM_problem.ipynb".
The first task is:
BEM method
Can you write Python code for it?

#Q = 2000/3    #strength of the source-sheet,stb/d
h=25.26        #thickness of local gridblock,ft
phi=0.2        #porosity
kx=200         #pemerability in x direction,md
ky=200         #pemerability in y direction,md
kr=kx/ky       #pemerability ratio
miu=1          #viscosity,cp

Nw=1           #Number of well
Qwell_1=2000   #Flow rate of well 1
Boundary_V=-400 #boundary velocity ft/day


I am working on the file "BEM_problem.ipynb".
The first task is:
Boundary Discretization
we will create a discretization of the body geometry into panels (line segments in 2D). A panel's attributes are: its starting point, end point and mid-point, its length and its orientation. See the following figure for the nomenclature used in the code and equations below.

Figure 1. Nomenclature of the boundary element in the local coordinates
Create panel and well class
Can you write Python code for it?

class Panel:
    """Contains information related to a panel."""
    def __init__(self, xa, ya, xb, yb):
        """Creates a panel.

        Arguments
        ---------
        xa, ya -- Cartesian coordinates of the first end-point.
        xb, yb -- Cartesian coordinates of the second end-point.
        """
        self.xa, self.ya = xa, ya
        self.xb, self.yb = xb, yb

        self.xc, self.yc = (xa+xb)/2, (ya+yb)/2       # control-point (center-point)
        self.length = math.sqrt((xb-xa)**2+(yb-ya)**2)     # length of the panel

        # orientation of the panel (angle between x-axis and panel)
        self.sinalpha=(yb-ya)/self.length
        self.cosalpha=(xb-xa)/self.length

        self.Q = 0.                                # source strength
        self.U = 0.                                # velocity component
        self.V = 0.                                # velocity component
        self.P = 0.                                # pressure coefficient

class Well:
    """Contains information related to a panel."""
    def __init__(self, xw, yw,rw,Q):
        """Creates a panel.

        Arguments
        ---------
        xw, yw -- Cartesian coordinates of well source.
        Q      -- Flow rate of well source.
        rw     -- radius of well source.
        """
        self.xw, self.yw = xw, yw

        self.Q = Q                               # source strength
        self.rw = rw                                # velocity component


I am working on the file "BEM_problem.ipynb".
The first task is:
Boundary element implementation

Figure 2. Representation of a local gridblock with boundary elements
Generally, the influence of all the j panels on the i BE node can be expressed as follows:
\begin{matrix}
{{c}{ij}}{{p}{i}}+{{p}{i}}\int{{{s}{j}}}{{{H}{ij}}d{{s}{j}}}=({{v}{i}}\cdot \mathbf{n})\int_{{{s}{j}}}{{{G}{ij}}}d{{s}_{j}}
\end{matrix}
Where,
${{c}_{ij}}$  is the free term, cased by source position.
${{c}_{ij}}=\left{ \begin{matrix}
   \begin{matrix}
   1 & \text{source j on the internal domain}  \
\end{matrix}  \
   \begin{matrix}
   0.5 & \text{source j on the boundary}  \
\end{matrix}  \
   \begin{matrix}
   0 & \text{source j on the external domain}  \
\end{matrix}  \
\end{matrix} \right.$
$\int_{{{s}{j}}}{{{H}{ij}}d{{s}_{j}}\text{ }}$ is the integrated effect of the boundary element source i on the resulting normal flux at BE node j.
$\int_{{{s}{j}}}{{{G}{ij}}}d{{s}_{j}}$ is the is the integrated effect of the boundary element source i on the resulting pressure at BE node j
Line segment source solution for pressure and velocity (Derived recently)
The integrated effect can be formulated using line segment source solution, which givs:
\begin{equation}
\int_{{{s}{j}}}{{{G}{ij}}}d{{s}{j}}=B{{Q}{w}}=P({{{x}'}{i}},{{{y}'}{i}})=-\frac{70.60\mu }{h\sqrt{{{k}{x}}{{k}{y}}}}\int_{t=0}^{t={{l}{j}}}{\ln \left{ {{({x}'-t\cos {{\alpha }{j}})}^{2}}+\frac{{{k}{x}}}{{{k}{y}}}{{({y}'-t\sin {{\alpha }{j}})}^{2}} \right}dt}\cdot {{Q}{w}}
\end{equation}
\begin{equation}
\int_{{{s}{j}}}{{{H}{ij}}d{{s}{j}}\text{ }}={{v}{i}}(s)\cdot {{\mathbf{n}}{i}}=-{{u}{i}}\sin {{\alpha }{i}}+{{v}{i}}\cos {{\alpha }_{i}}
\end{equation}
Where,
\begin{equation}
u\left( {{{{x}'}}{i}},{{{{y}'}}{i}} \right)={{A}{u}}{{Q}{j}}=\frac{0.8936}{h\phi }\sqrt{\frac{{{k}{x}}}{{{k}{y}}}}\int_{t=0}^{t={{l}{j}}}{\frac{{{{{x}'}}{i}}-t\cos {{\alpha }{j}}}{{{\left( {{{{x}'}}{i}}-t\cos {{\alpha }{j}} \right)}^{2}}+\frac{{{k}{x}}}{{{k}{y}}}{{({{{{y}'}}{i}}-t\sin {{\alpha...
Can you write Python code for it?

#Panel infuence factor Bij
def InflueceP(x, y, panel):
    """Evaluates the contribution of a panel at one point.

    Arguments
    ---------
    x, y -- Cartesian coordinates of the point.
    panel -- panel which contribution is evaluated.

    Returns
    -------
    Integral over the panel of the influence at one point.
    """
#Transfer global coordinate point(x,y) to local coordinate
    x=x-panel.xa
    y=y-panel.ya
    L1=panel.length

#Calculate the pressure and velocity influence factor
    a=panel.cosalpha**2+kr*panel.sinalpha**2
    b=x*panel.cosalpha+kr*panel.sinalpha*y
    c=y*panel.cosalpha-x*panel.sinalpha
    dp=70.6*miu/h/math.sqrt(kx*ky)
    Cp = dp/a*(
                             (
                              b*math.log(x**2-2*b*L1+a*L1**2+kr*y**2)
                             -L1*a*math.log((x-L1*panel.cosalpha)**2+kr*(y-L1*panel.sinalpha)**2)
                             +2*math.sqrt(kr)*c*math.atan((b-a*L1)/math.sqrt(kr)/c)
                             )
                             -
                             (
                               b*math.log(x**2+kr*y**2)
                               +2*math.sqrt(kr)*c*math.atan((b)/math.sqrt(kr)/c)
                             )
                )
    #debug
    #print("a: %s b:%s c:%s  " % (a,b,c))
    #angle=math.atan((b-a*L1)/math.sqrt(kr)/c)*180/numpy.pi
    #print("Magic angle:%s"% angle)
    return Cp

def InflueceU(x, y, panel):
    """Evaluates the contribution of a panel at one point.

    Arguments
    ---------
    x, y -- Cartesian coordinates of the point.
    panel -- panel which contribution is evaluated.

    Returns
    -------
    Integral over the panel of the influence at one point.
    """
#Transfer global coordinate point(x,y) to local coordinate
    x=x-panel.xa
    y=y-panel.ya
    L1=panel.length

#Calculate the pressure and velocity influence factor
    a=panel.cosalpha**2+kr*panel.sinalpha**2
    b=x*panel.cosalpha+kr*panel.sinalpha*y
    c=y*panel.cosalpha-x*panel.sinalpha
   ...


I am working on the file "BEM_problem.ipynb".
The first task is:
We create a node distribution on the boundary that is refined near the corner with cosspace function
Can you write Python code for it?

def cosspace(st,ed,N):
    N=N+1
    AngleInc=numpy.pi/(N-1)
    CurAngle = AngleInc
    space=numpy.linspace(0,1,N)
    space[0]=st
    for i in range(N-1):
        space[i+1] = 0.5*numpy.abs(ed-st)*(1 - math.cos(CurAngle));
        CurAngle += AngleInc
    if ed<st:
        space[0]=ed
        space=space[::-1]
    return space


I am working on the file "BEM_problem.ipynb".
The first task is:
Well source function
Line source solution for pressure and velocity (Datta-Gupta, 2007)
\begin{equation}
P(x,y)=B{{Q}{w}}=-\frac{70.60\mu }{h\sqrt{{{k}{x}}{{k}{y}}}}\ln \left{ {{(x-{{x}{w}})}^{2}}+\frac{{{k}{x}}}{{{k}{y}}}{{(y-{{y}{w}})}^{2}} \right}{{Q}{w}}+{{P}_{avg}}
\end{equation}
\begin{equation}
\frac{\partial P}{\partial x}=u=\frac{0.8936}{h\phi }\sqrt{\frac{{{k}{x}}}{{{k}{y}}}}\sum\limits_{k=1}^{{{N}{w}}}{{{Q}{k}}}\frac{x-{{x}{k}}}{{{\left( x-{{x}{k}} \right)}^{2}}+\frac{{{k}{x}}}{{{k}{y}}}{{(y-{{y}_{k}})}^{2}}}
\end{equation}
\begin{equation}
\frac{\partial P}{\partial y}=v=\frac{0.8936}{h\phi }\sqrt{\frac{{{k}{x}}}{{{k}{y}}}}\sum\limits_{k=1}^{{{N}{w}}}{{{Q}{k}}}\frac{y-{{y}{k}}}{{{\left( x-{{x}{k}} \right)}^{2}}+\frac{{{k}{x}}}{{{k}{y}}}{{(y-{{y}_{k}})}^{2}}}
\end{equation}
Can you write Python code for it?

#Well influence factor
def InflueceP_W(x, y, well):
    """Evaluates the contribution of a panel at one point.

    Arguments
    ---------
    x, y -- Cartesian coordinates of the point.
    panel -- panel which contribution is evaluated.

    Returns
    -------
    Integral over the panel of the influence at one point.
    """
    dp=-70.6*miu/h/math.sqrt(kx*ky)
    Cp=dp*math.log((x-well.xw)**2+kr*(y-well.yw)**2)
    return Cp

def InflueceU_W(x, y, well):
    """Evaluates the contribution of a panel at one point.

    Arguments
    ---------
    x, y -- Cartesian coordinates of the point.
    panel -- panel which contribution is evaluated.

    Returns
    -------
    Integral over the panel of the influence at one point.
    """
    dv=0.8936/h/phi*math.sqrt(kx/ky)
    Cu=dv*(x-well.xw)/((x-well.xw)**2+kr*(y-well.yw)**2)
    return Cu

def InflueceV_W(x, y, well):
    """Evaluates the contribution of a panel at one point.

    Arguments
    ---------
    x, y -- Cartesian coordinates of the point.
    panel -- panel which contribution is evaluated.

    Returns
    -------
    Integral over the panel of the influence at one point.
    """
    dv=0.8936/h/phi*math.sqrt(kx/ky)
    Cv=dv*(y-well.yw)/((x-well.xw)**2+kr*(y-well.yw)**2)
    return Cv

#InflueceV(0.5,1,panels[3])
#InflueceP(0,0.5,panels[0])
#InflueceU(0,0.5,panels[0])


I am working on the file "BETA/TestCode/Tensorflow/Test/object_detection/object_detection_tutorial.ipynb".
The first task is:
Detection
Can you write Python code for it?

# For the sake of simplicity we will use only 2 images:
# image1.jpg
# image2.jpg
# If you want to test the code with your images, just add path to the images to the TEST_IMAGE_PATHS.
PATH_TO_TEST_IMAGES_DIR = 'test_images'
TEST_IMAGE_PATHS = [ os.path.join(PATH_TO_TEST_IMAGES_DIR, 'image{}.jpg'.format(i)) for i in range(1, 3) ]

# Size, in inches, of the output images.
IMAGE_SIZE = (12, 8)

with detection_graph.as_default():
  with tf.Session(graph=detection_graph) as sess:
    for image_path in TEST_IMAGE_PATHS:
      image = Image.open(image_path)
      # the array based representation of the image will be used later in order to prepare the
      # result image with boxes and labels on it.
      image_np = load_image_into_numpy_array(image)
      # Expand dimensions since the model expects images to have shape: [1, None, None, 3]
      image_np_expanded = np.expand_dims(image_np, axis=0)
      image_tensor = detection_graph.get_tensor_by_name('image_tensor:0')
      # Each box represents a part of the image where a particular object was detected.
      boxes = detection_graph.get_tensor_by_name('detection_boxes:0')
      # Each score represent how level of confidence for each of the objects.
      # Score is shown on the result image, together with the class label.
      scores = detection_graph.get_tensor_by_name('detection_scores:0')
      classes = detection_graph.get_tensor_by_name('detection_classes:0')
      num_detections = detection_graph.get_tensor_by_name('num_detections:0')
      # Actual detection.
      (boxes, scores, classes, num_detections) = sess.run(
          [boxes, scores, classes, num_detections],
          feed_dict={image_tensor: image_np_expanded})
      # Visualization of the results of a detection.
      vis_util.visualize_boxes_and_labels_on_image_array(
          image_np,
          np.squeeze(boxes),
          np.squeeze(classes).astype(np.int32),
          np.squeeze(scores),
          category_index,
          use_normalized_coordinates=True,
          line_thickness=8)
    ...


I am working on the file "BETA/TestCode/Tensorflow/Test/object_detection/object_detection_tutorial.ipynb".
The first task is:
Download Model
Can you write Python code for it?

opener = urllib.request.URLopener()
opener.retrieve(DOWNLOAD_BASE + MODEL_FILE, MODEL_FILE)
tar_file = tarfile.open(MODEL_FILE)
for file in tar_file.getmembers():
    file_name = os.path.basename(file.name)
    if 'frozen_inference_graph.pb' in file_name:
        tar_file.extract(file, os.getcwd())


I am working on the file "BETA/TestCode/Tensorflow/Test/object_detection/object_detection_tutorial.ipynb".
The first task is:
Helper code
Can you write Python code for it?

def load_image_into_numpy_array(image):
  (im_width, im_height) = image.size
  return np.array(image.getdata()).reshape(
      (im_height, im_width, 3)).astype(np.uint8)


I am working on the file "BETA/TestCode/Tensorflow/Test/object_detection/object_detection_tutorial.ipynb".
The first task is:
Load a (frozen) Tensorflow model into memory.
Can you write Python code for it?

detection_graph = tf.Graph()
with detection_graph.as_default():
    od_graph_def = tf.GraphDef()
    with tf.gfile.GFile(PATH_TO_CKPT, 'rb') as fid:
        serialized_graph = fid.read()
        od_graph_def.ParseFromString(serialized_graph)
        tf.import_graph_def(od_graph_def, name='')


I am working on the file "BETA/TestCode/Tensorflow/Test/object_detection/object_detection_tutorial.ipynb".
The first task is:
Model preparation
Variables
Any model exported using the export_inference_graph.py tool can be loaded here simply by changing PATH_TO_CKPT to point to a new .pb file.
By default we use an "SSD with Mobilenet" model here. See the detection model zoo for a list of other models that can be run out-of-the-box with varying speeds and accuracies.
Can you write Python code for it?

# What model to download.
MODEL_NAME = 'ssd_mobilenet_v1_coco_11_06_2017'
MODEL_FILE = MODEL_NAME + '.tar.gz'
DOWNLOAD_BASE = 'http://download.tensorflow.org/models/object_detection/'

# Path to frozen detection graph. This is the actual model that is used for the object detection.
PATH_TO_CKPT = MODEL_NAME + '/frozen_inference_graph.pb'

# List of the strings that is used to add correct label for each box.
PATH_TO_LABELS = os.path.join('data', 'mscoco_label_map.pbtxt')

NUM_CLASSES = 90


I am working on the file "BETA/TestCode/Tensorflow/Test/object_detection/object_detection_tutorial.ipynb".
The first task is:
Object detection imports
Here are the imports from the object detection module.
Can you write Python code for it?

from utils import label_map_util

from utils import visualization_utils as vis_util


I am working on the file "BETA/TestCode/Tensorflow/object_detection/object_detection_tutorial.ipynb".
The first task is:
Detection
Can you write Python code for it?

# For the sake of simplicity we will use only 2 images:
# image1.jpg
# image2.jpg
# If you want to test the code with your images, just add path to the images to the TEST_IMAGE_PATHS.
PATH_TO_TEST_IMAGES_DIR = 'test_images'
TEST_IMAGE_PATHS = [ os.path.join(PATH_TO_TEST_IMAGES_DIR, 'image{}.jpg'.format(i)) for i in range(1, 3) ]

# Size, in inches, of the output images.
IMAGE_SIZE = (12, 8)

with detection_graph.as_default():
  with tf.Session(graph=detection_graph) as sess:
    # Definite input and output Tensors for detection_graph
    image_tensor = detection_graph.get_tensor_by_name('image_tensor:0')
    # Each box represents a part of the image where a particular object was detected.
    detection_boxes = detection_graph.get_tensor_by_name('detection_boxes:0')
    # Each score represent how level of confidence for each of the objects.
    # Score is shown on the result image, together with the class label.
    detection_scores = detection_graph.get_tensor_by_name('detection_scores:0')
    detection_classes = detection_graph.get_tensor_by_name('detection_classes:0')
    num_detections = detection_graph.get_tensor_by_name('num_detections:0')
    for image_path in TEST_IMAGE_PATHS:
      image = Image.open(image_path)
      # the array based representation of the image will be used later in order to prepare the
      # result image with boxes and labels on it.
      image_np = load_image_into_numpy_array(image)
      # Expand dimensions since the model expects images to have shape: [1, None, None, 3]
      image_np_expanded = np.expand_dims(image_np, axis=0)
      # Actual detection.
      (boxes, scores, classes, num) = sess.run(
          [detection_boxes, detection_scores, detection_classes, num_detections],
          feed_dict={image_tensor: image_np_expanded})
      # Visualization of the results of a detection.
      vis_util.visualize_boxes_and_labels_on_image_array(
          image_np,
          np.squeeze(boxes),
          np.squeeze(classes).astype(np.int32),
          np.squeeze(scores),
        ...


I am working on the file "BETA/TestCode/Tensorflow/object_detection/object_detection_tutorial.ipynb".
The first task is:
Env setup
Can you write Python code for it?

# This is needed to display the images.
%matplotlib inline

# This is needed since the notebook is stored in the object_detection folder.
sys.path.append("..")


I am working on the file "BETA/TestCode/Tensorflow/object_detection/object_detection_tutorial.ipynb".
The first task is:
Load a (frozen) Tensorflow model into memory.
Can you write Python code for it?

detection_graph = tf.Graph()
with detection_graph.as_default():
  od_graph_def = tf.GraphDef()
  with tf.gfile.GFile(PATH_TO_CKPT, 'rb') as fid:
    serialized_graph = fid.read()
    od_graph_def.ParseFromString(serialized_graph)
    tf.import_graph_def(od_graph_def, name='')


I am working on the file "BETA/TestCode/Tensorflow/object_detection/object_detection_tutorial.ipynb".
The first task is:
Loading label map
Label maps map indices to category names, so that when our convolution network predicts 5, we know that this corresponds to airplane.  Here we use internal utility functions, but anything that returns a dictionary mapping integers to appropriate string labels would be fine
Can you write Python code for it?

label_map = label_map_util.load_labelmap(PATH_TO_LABELS)
categories = label_map_util.convert_label_map_to_categories(label_map, max_num_classes=NUM_CLASSES, use_display_name=True)
category_index = label_map_util.create_category_index(categories)


I am working on the file "BETA/TestCode/Tensorflow/object_detection/object_detection_tutorial.ipynb".
The first task is:
Model preparation
Variables
Any model exported using the export_inference_graph.py tool can be loaded here simply by changing PATH_TO_CKPT to point to a new .pb file.
By default we use an "SSD with Mobilenet" model here. See the detection model zoo for a list of other models that can be run out-of-the-box with varying speeds and accuracies.
Can you write Python code for it?


I am working on the file "BETA/TestCode/Tensorflow/object_detection/object_detection_tutorial.ipynb".
The first task is:
Object detection imports
Here are the imports from the object detection module.
Can you write Python code for it?


I am working on the file "BGC_Team/Facies Prediction_submit.ipynb".
The first task is:
2. Feature Selection
Log Plot of Facies
Filtered Data
Can you write Python code for it?

#count the number of unique entries for each facies, sort them by
#facies number (instead of by number of entries)
#facies_counts_filtered = training_data_filtered['Facies'].value_counts().sort_index()
facies_counts = training_data['Facies'].value_counts().sort_index()
#use facies labels to index each count
#facies_counts_filtered.index = facies_labels
facies_counts.index = facies_labels

#facies_counts_filtered.plot(kind='bar',color=facies_colors,
#                   title='Distribution of Filtered Training Data by Facies')
facies_counts.plot(kind='bar',color=facies_colors,
                   title='Distribution of Filtered Training Data by Facies')
#facies_counts_filtered
#training_data_filtered.columns
#facies_counts_filtered

training_data.columns
facies_counts


I am working on the file "BGC_Team/Facies Prediction_submit.ipynb".
The first task is:
3. Prediction Model
Accuracy
Can you write Python code for it?

def accuracy(conf):
    total_correct = 0.
    nb_classes = conf.shape[0]
    for i in np.arange(0,nb_classes):
        total_correct += conf[i][i]
    acc = total_correct/sum(sum(conf))
    return acc

adjacent_facies = np.array([[1], [0,2], [1], [4], [3,5], [4,6,7], [5,7], [5,6,8], [6,7]])

def accuracy_adjacent(conf, adjacent_facies):
    nb_classes = conf.shape[0]
    total_correct = 0.
    for i in np.arange(0,nb_classes):
        total_correct += conf[i][i]
        for j in adjacent_facies[i]:
            total_correct += conf[i][j]
    return total_correct / sum(sum(conf))


I am working on the file "BGC_Team/Facies Prediction_submit.ipynb".
The first task is:
5. Using Tensorflow
Filtered Data Model
Can you write Python code for it?

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import tensorflow as tf

# Specify that all features have real-value data
feature_columns_filtered = [tf.contrib.layers.real_valued_column("", dimension=7)]

# Build 3 layer DNN with 7, 17, 10 units respectively.
classifier_filtered = tf.contrib.learn.DNNClassifier(feature_columns=feature_columns_filtered,
                                            hidden_units=[7, 17, 10],
                                            n_classes=10)

# Fit model.
#classifier_filtered.fit(x=X_train_filtered,y=y_train_filtered,steps=5000)
#y_predict_filtered = []
#predictions = classifier_filtered.predict(x=X_test_filtered)

classifier_filtered.fit(x=X_train,y=y_train,steps=5000)
y_predict = []
predictions = classifier_filtered.predict(x=X_test)

for i, p in enumerate(predictions):
    y_predict.append(p)
    #print("Index %s: Prediction - %s, Real - %s" % (i + 1, p, y_test_filtered[i]))

# Evaluate accuracy.
#accuracy_score_filtered = classifier_filtered.evaluate(x=X_test_filtered, y=y_test_filtered)["accuracy"]
#print('Accuracy: {0:f}'.format(accuracy_score_filtered))
accuracy_score = classifier_filtered.evaluate(x=X_test, y=y_test)["accuracy"]
print('Accuracy: {0:f}'.format(accuracy_score))

cv_conf_dnn = confusion_matrix(y_test, y_predict)

print('Optimized facies classification accuracy = %.2f' % accuracy(cv_conf_dnn))
print('Optimized adjacent facies classification accuracy = %.2f' % accuracy_adjacent(cv_conf_dnn, adjacent_facies))
display_cm(cv_conf_dnn, facies_labels,display_metrics=True, hide_zeros=True)


I am working on the file "BGC_Team/Facies Prediction_submit.ipynb".
The first task is:
Add Missing PE by following AR4 Team
Can you write Python code for it?

#X = training_data_filtered[feature_names].values
# Testing without filtering
X = training_data[feature_names].values

reg = RandomForestRegressor(max_features='sqrt', n_estimators=50)
# DataImpAll = training_data_filtered[feature_names].copy()
DataImpAll = training_data[feature_names].copy()
DataImp = DataImpAll.dropna(axis = 0, inplace=False)
Ximp=DataImp.loc[:, DataImp.columns != 'PE']
Yimp=DataImp.loc[:, 'PE']
reg.fit(Ximp, Yimp)
X[np.array(DataImpAll.PE.isnull()),4] = reg.predict(DataImpAll.loc[DataImpAll.PE.isnull(),:].drop('PE',axis=1,inplace=False))


I am working on the file "BGC_Team/Facies Prediction_submit.ipynb".
The first task is:
Result from DNN
Can you write Python code for it?

classifier_filtered.fit(x=X_train_full,
               y=y_train_full,
               steps=10000)
predictions = classifier_filtered.predict(X_unknown)
y_predict_filtered = []
for i, p in enumerate(predictions):
    y_predict_filtered.append(p)
well_data['Facies'] = y_predict_filtered
well_data
well_data.to_csv('predict_result_dnn_full_data.csv')

#export_dir_path = 'd:\\'
#classifier_filtered.export(export_dir_path)


I am working on the file "BGC_Team/Facies Prediction_submit.ipynb".
The first task is:
SVM
Can you write Python code for it?

from sklearn.model_selection import KFold, cross_val_score,LeavePGroupsOut, LeaveOneGroupOut, cross_val_predict
from classification_utilities import display_cm, display_adj_cm

from sklearn import svm
clf_filtered = svm.LinearSVC(random_state=23)

#clf_filtered.fit(X_train_filtered, y_train_filtered)
clf_filtered.fit(X_train, y_train)

#predicted_labels_filtered = clf_filtered.predict(X_test_filtered)
predicted_labels = clf_filtered.predict(X_test)


I am working on the file "BGC_Team/Facies Prediction_submit.ipynb".
The first task is:
SVM for filtered data model
4. Result Analysis
Can you write Python code for it?

well_data = pd.read_csv('validation_data_nofacies.csv')
well_data['Well Name'] = well_data['Well Name'].astype('category')
well_features = well_data.drop(['Formation', 'Well Name', 'Depth'], axis=1)

X_unknown = scaler_filtered.transform(well_features)

# Using all data and optimize parameter to train the data
clf_filtered = svm.SVC(C=10, gamma=1)
clf_filtered.fit(X_train_full, y_train_full)
#clf_filtered.fit(X_train_filtered, y_train_filtered)
y_unknown = clf_filtered.predict(X_unknown)
well_data['Facies'] = y_unknown
well_data
well_data.to_csv('predict_result_svm_full_data.csv')


I am working on the file "BGC_Team/Facies Prediction_submit.ipynb".
The first task is:
Using Full data to train
Can you write Python code for it?

# 1=sandstone  2=c_siltstone   3=f_siltstone # 4=marine_silt_shale
#5=mudstone 6=wackestone 7=dolomite 8=packstone 9=bafflestone
facies_colors = ['#F4D03F', '#F5B041', '#DC7633','#A569BD',
       '#000000', '#000080', '#2E86C1', '#AED6F1', '#196F3D']
feature_names = ['GR', 'ILD_log10', 'DeltaPHI', 'PHIND', 'PE', 'NM_M', 'RELPOS']

facies_labels = ['SS', 'CSiS', 'FSiS', 'SiSh', 'MS',
                 'WS', 'D','PS', 'BS']
#facies_color_map is a dictionary that maps facies labels
#to their respective colors

training_data = pd.read_csv(facies_vector_path)

facies_color_map = {}
for ind, label in enumerate(facies_labels):
    facies_color_map[label] = facies_colors[ind]

def label_facies(row, labels):
    return labels[ row['Facies'] -1]

training_data.loc[:,'FaciesLabels'] = training_data.apply(lambda row: label_facies(row, facies_labels), axis=1)
training_data.describe()

# Fitering out some outliers
j = []
for i in range(len(training_data)):
    if ((training_data['NM_M'].values[i]==2)and ((training_data['Facies'].values[i]==1)or(training_data['Facies'].values[i]==2)or(training_data['Facies'].values[i]==3))):
        j.append(i)
    elif((training_data['NM_M'].values[i]==1)and((training_data['Facies'].values[i]!=1)and(training_data['Facies'].values[i]!=2)and(training_data['Facies'].values[i]!=3))):
        j.append(i)

training_data_filtered = training_data.drop(training_data.index[j])
print(np.shape(training_data_filtered))


I am working on the file "BGC_Team/Facies Prediction_submit6.ipynb".
The first task is:
1. Data Prepocessing
1) Filtered data preparation
After the initial data validation, we figure out the NM_M input is a key differentiator to group non-marine stones (sandstone, c_siltstone, and f_siltstone) and marine stones (marine_silt_shale, mudstone, wakestone, dolomite, packstone, and bafflestone) in the current field. Our team decides to use this classifier aggressively and prepare a filtered dataset which cleans up the outliers.
Can you write Python code for it?

# Input file paths
facies_vector_path = 'facies_vectors.csv'
train_path = 'training_data.csv'
test_path = 'validation_data_nofacies.csv'
# Read training data to dataframe
#training_data = pd.read_csv(train_path)


I am working on the file "BGC_Team/Facies Prediction_submit6.ipynb".
The first task is:
2. Feature Selection
Log Plot of Facies
Filtered Data
Can you write Python code for it?


I am working on the file "BGC_Team/Facies Prediction_submit6.ipynb".
The first task is:
4. Result Analysis
Prepare test data
Can you write Python code for it?

well_data = pd.read_csv('validation_data_nofacies.csv')
well_data['Well Name'] = well_data['Well Name'].astype('category')
well_features = well_data.drop(['Formation', 'Well Name', 'Depth'], axis=1)
# Prepare test data
well_ts = well_data['Well Name'].values
depth_ts = well_data['Depth'].values
X_ts = well_data[feature_names].values
X_ts = scaler.transform(X_ts)
# Augment features
X_ts, padded_rows = augment_features(X_ts, well_ts, depth_ts)

# Using all data and optimize parameter to train the data
clf_filtered = svm.SVC(C=10, gamma=1)
clf_filtered.fit(X_train_full, y_train_full)
#clf_filtered.fit(X_train_filtered, y_train_filtered)
y_pred = clf_filtered.predict(X_ts)
well_data['Facies'] = y_pred
well_data
well_data.to_csv('predict_result_svm_full_data.csv')


I am working on the file "BGC_Team/Facies Prediction_submit6.ipynb".
The first task is:
Add Missing PE by following AR4 Team
Can you write Python code for it?


I am working on the file "BGC_Team/Facies Prediction_submit6.ipynb".
The first task is:
Filtered facies
Can you write Python code for it?

correct_facies_labels = training_data['Facies'].values
feature_vectors = training_data.drop(['Formation', 'Well Name', 'Depth','Facies','FaciesLabels'], axis=1)


I am working on the file "BGC_Team/Facies Prediction_submit6.ipynb".
The first task is:
Find best model by setting different parameters
Can you write Python code for it?

def dnn_prediction(dnn,s_num):
    dnn.fit(x=X_train,y=y_train,steps=s_num)
    y_predict = []
    predictions = classifier_filtered.predict(x=X_test)

    for i, p in enumerate(predictions):
        y_predict.append(p)

    score = classifier_filtered.evaluate(x=X_test, y=y_test)["accuracy"]
    print('Accuracy: {0:f}'.format(accuracy_score))

    cv_conf = confusion_matrix(y_test, y_predict)
    return score, cv_conf


I am working on the file "BGC_Team/Facies Prediction_submit6.ipynb".
The first task is:
Normailization
Can you write Python code for it?

from sklearn import preprocessing
from sklearn.cross_validation import train_test_split
scaler = preprocessing.StandardScaler().fit(X)
scaled_features = scaler.transform(X)


I am working on the file "BGC_Team/Facies Prediction_submit6.ipynb".
The first task is:
Post Processing
Can you write Python code for it?

well_data["Well Name"].value_counts()
idx_s = []
idx_c = []
for i in range(len(well_data)):
    if (well_data["Well Name"].values[i] == "STUART"):
        idx_s.append(i)
    else:
        idx_c.append(i)
well_s = well_data.drop(well_data.index[idx_c])
well_c = well_data.drop(well_data.index[idx_s])

def smooth_results(w_data):
    data = w_data.copy()
    first_face = 0
    next_face = 0
    face_len = 0
    last_face = 0
    for i in range(len(data)):
        if(i==0):
            first_face = data["Facies"].values[i]
            continue
        next_face = data["Facies"].values[i]
        if (first_face == next_face):
            face_len = face_len+1
        if (face_len >=4):
            last_face = first_face
        if (first_face != next_face):
            if(last_face == next_face) and (face_len <4):
                for j in range(i-face_len, i):
                    data["Facies"].values[j]=last_face
                face_len = 1
            else:
                face_len = 1
                first_face = next_face
    return data

well_s_s = smooth_results(well_s)
well_c_s = smooth_results(well_c)

smooth_result = well_s_s
smooth_result = smooth_result.append(well_c_s)
smooth_result.to_csv('predict_result_dnn_full_data_smooth.csv')


I am working on the file "BGC_Team/Facies Prediction_submit6.ipynb".
The first task is:
SVM
Can you write Python code for it?

from classification_utilities import display_cm, display_adj_cm
import sklearn.svm as svm

clf_filtered = svm.SVC(C=10, gamma=1)
clf_filtered.fit(X_train, y_train)

#predicted_labels_filtered = clf_filtered.predict(X_test_filtered)
predicted_labels = clf_filtered.predict(X_test)
cv_conf_svm = confusion_matrix(y_test, predicted_labels)
print('Optimized facies classification accuracy = %.2f' % accuracy(cv_conf_svm))
print('Optimized adjacent facies classification accuracy = %.2f' % accuracy_adjacent(cv_conf_svm, adjacent_facies))
display_cm(cv_conf_svm, facies_labels,display_metrics=True, hide_zeros=True)


I am working on the file "BGC_Team/Facies Prediction_submit6.ipynb".
The first task is:
Using Full data to train
Can you write Python code for it?

# 1=sandstone  2=c_siltstone   3=f_siltstone # 4=marine_silt_shale
#5=mudstone 6=wackestone 7=dolomite 8=packstone 9=bafflestone
facies_colors = ['#F4D03F', '#F5B041', '#DC7633','#A569BD',
       '#000000', '#000080', '#2E86C1', '#AED6F1', '#196F3D']
feature_names = ['GR', 'ILD_log10', 'DeltaPHI', 'PHIND', 'PE', 'NM_M', 'RELPOS']

facies_labels = ['SS', 'CSiS', 'FSiS', 'SiSh', 'MS',
                 'WS', 'D','PS', 'BS']
#facies_color_map is a dictionary that maps facies labels
#to their respective colors

training_data = pd.read_csv(facies_vector_path)
X = training_data[feature_names].values
y = training_data['Facies'].values
well = training_data['Well Name'].values
depth = training_data['Depth'].values

facies_color_map = {}
for ind, label in enumerate(facies_labels):
    facies_color_map[label] = facies_colors[ind]

def label_facies(row, labels):
    return labels[ row['Facies'] -1]

training_data.loc[:,'FaciesLabels'] = training_data.apply(lambda row: label_facies(row, facies_labels), axis=1)
training_data.describe()

# Fitering out some outliers
j = []
for i in range(len(training_data)):
    if ((training_data['NM_M'].values[i]==2)and ((training_data['Facies'].values[i]==1)or(training_data['Facies'].values[i]==2)or(training_data['Facies'].values[i]==3))):
        j.append(i)
    elif((training_data['NM_M'].values[i]==1)and((training_data['Facies'].values[i]!=1)and(training_data['Facies'].values[i]!=2)and(training_data['Facies'].values[i]!=3))):
        j.append(i)

training_data_filtered = training_data.drop(training_data.index[j])
print(np.shape(training_data_filtered))


I am working on the file "BMA data extraction notebook.ipynb".
The first task is:
Now we extract the result pages against each of the id(1 to 66000) and store the strings in a pandas Dataframe. We will tokenize the resultant string later.
Can you write Python code for it?

start = 'doctor_info'
finish="</div"
extracted_strings = []
extracted_df = pd.DataFrame(columns=['extracted'])

for reg_no in xrange(1,66001):
    browser = Browser()
    browser.open("http://bmdc.org.bd/doctors-info/")
    for form in browser.forms():
        pass
    # We have 2 forms in this page and we going to select the second form
    browser.select_form(nr=1)
    # This form has 2 input fields, first field, search_doc_id takes an number and second field type indicates if the
    # id is assocated to a medical doctor or dentist
    form['search_doc_id']=str(reg_no)
    form['type']=['1']
    # Submit the form and read the result
    response = browser.submit()
    content = response.read()
    str_content = str(content)
    #Extract only the relevant portion
    extracted_str = extract_sub_string(str_content, start, finish)
    extracted_strings.append(extracted_str)
    # Originally these commnted out snipppets were run so that each group of 100 doctors are recorded at a time in
    # seperate csv files. for testing and stability purpose. Each 100 doctors took around 6-7 minutes to record.
    #if reg_no%100==0:
    #    file_number = reg_no/100
    #    extracted_df = pd.DataFrame(columns=['extracted'])
    #    extracted_df.extracted = extracted_strings
    #    extracted_df.to_csv(str(file_number)+'.csv')
    #    extracted_strings = []
extracted_df.extracted = extracted_strings
extracted_df.to_csv('all_bma_doctor.csv')


I am working on the file "BMI-Meteorology-P-Scalar.ipynb".
The first task is:
Despite setting a value of 20 mm/hr for P in the cfg file, if I call get_value at this point, the model precip values are zeroed out:
Can you write Python code for it?

precip = m.get_value('atmosphere_water__precipitation_leq-volume_flux')  # `P` internally
print type(precip)
print precip.size
precip


I am working on the file "BMI-Meteorology-P-Scalar.ipynb".
The first task is:
Import the Meteorology component and create an instance:
Can you write Python code for it?

from topoflow.components.met_base import met_component
m = met_component()


I am working on the file "BMI-Meteorology-P-Scalar.ipynb".
The first task is:
Is precipitation being produced?
Can you write Python code for it?

precip = m.get_value('atmosphere_water__precipitation_leq-volume_flux')
print precip


I am working on the file "BMI-Meteorology-P-Scalar.ipynb".
The first task is:
Locate the cfg file and initialize the component:
Can you write Python code for it?

cfg_file = './input/meteorology.cfg'
m.initialize(cfg_file)


I am working on the file "BMI-Meteorology-P-Scalar.ipynb".
The first task is:
Maybe this will change after the first update?
Show the model start, current, and stop times:
Can you write Python code for it?

print m.get_start_time(), m.get_current_time(), m.get_end_time()


I am working on the file "BMI-Meteorology-P-Scalar.ipynb".
The first task is:
Note that it hasn't precipitated:
Can you write Python code for it?


I am working on the file "BMI-vs-Distance.ipynb".
The first task is:
1. Retrieve an clean up the data
Athlete stats from The Guardian. We're only interested in Athletics here, so drop the other sports/events, and change a few names to get the distance in metres.
Can you write Python code for it?

df=pd.read_csv("london-2012-athletes.csv")
df=df[df.Sport=="Athletics"]
df.replace(to_replace={'Event':{r'0,000m' : "000m",
                                r'km' : "000m",
                                r'Marathon' : "42195m",
                                r'4 x 100m' : "100m",
                                r'4 x 400m' : "400m",
                                }
                      }, inplace=True, regex=True)
df=df.reset_index()


I am working on the file "BMI-vs-Distance.ipynb".
The first task is:
2. Calculate the Body Mass Index (BMI) of each athlete
The BMI is by no means a perfect descriptor of the athlete's "shape". we can however guess (and confirm below) that sprint runners tend to have a high BMI compared to long-distance runners.
Can you write Python code for it?

df['Distance']=df.apply(lambda row: int("".join(filter(str.isdigit, row['Event']))),
                        axis=1)
df['BMI']=df['Weight']/df['Height, cm']*100
df[['Name', 'Distance', 'BMI', 'Sex']].head()


I am working on the file "BMI-vs-Distance.ipynb".
The first task is:
Drop events that aren't runs (throws and jumps) as well as hurdles. Relays are kept in the dataset.
Can you write Python code for it?

sports_to_ignore = ['Hammer', 'Shot', 'Hurdles', 'Javelin', 'Vault', 'Decathlon', 'Heptathlon', 'Jump', 'Discus', 'Steeplechase', 'Race Walk']
ignore_pattern = '|'.join(sports_to_ignore)
df=df[~df.Event.str.contains(ignore_pattern, flags=re.IGNORECASE)]
df=df.reset_index()

_=df.Event.groupby(df.Event).count().sort_index().plot(kind="bar", title="Number of Athletes per Event")


I am working on the file "BMI-vs-Distance.ipynb".
The first task is:
Some athletes take part in multiple events; split each event into its own row.
Can you write Python code for it?

s = pd.DataFrame(df.Event.str.split(',').tolist()).stack()
s.index = s.index.droplevel(-1)
s.name='Event'
df.drop('Event', axis=1, inplace=True)

df=df.join(s)


I am working on the file "BMLSwPython/01_GettingStarted_withPython.ipynb".
The first task is:
$$  f(x) = 0.0105322215 * x^2 - 5.26545650 * x + 1974.6082 $$
What if we want to regress two response output instead of one, As we can see in the graph that there is a steep change in data between week 3 and 4, so let's draw two reponses line, one for the data between week0 and week3.5 and second for week3.5 to week5
Can you write Python code for it?

# we are going to divide the data on time so
div = 3.5*7*24

X1 = X[X<=div]
Y1 = y[X<=div]

X2 = X[X>div]
Y2 = y[X>div]

# now plotting the both data

fa = sp.poly1d(sp.polyfit(X1, Y1, 1))
fb = sp.poly1d(sp.polyfit(X2, Y2, 1))

fa_error = error(fa, X1, Y1)
fb_error = error(fb, X2, Y2)
print("Error inflection = %f" % (fa_error + fb_error))

x1 = np.linspace(-100, X1[-1]+100, 1000)
x2 = np.linspace(X1[-10], X2[-1]+100, 1000)

ya = fa(x1)
yb = fb(x2)

ax.plot(x1, ya, c='#800000', linewidth=2)  # brown
ax.plot(x2, yb, c='#FFA500', linewidth=2)  # orange
ax.grid(True)

fig


I am working on the file "BMLSwPython/01_GettingStarted_withPython.ipynb".
The first task is:
$$ f(x) = 2.59619213 * x + 989.02487106 $$
 Polynomial 2-d
Can you write Python code for it?

# sp's polyfit func do the same
fp2 = sp.polyfit(X, y, 2)

print(fp2)

# generating the 2 order function
f2= sp.poly1d(fp2)

# checking error
print("Error : ",error(f2, X, y))

x1= np.linspace(-100, np.max(X)+100, 2000)
y2= f2(x1)

ax.plot(x1, y2, c='r', linewidth=2)
ax.legend(["data", "d = %i" % f1.order, "d = %i" % f2.order], loc='best')
fig


I am working on the file "BMLSwPython/01_GettingStarted_withPython.ipynb".
The first task is:
Choosing the right model and learning algorithm
Can you write Python code for it?

# creating a error calc fuction
def error(f, x, y):
    return np.sum((f(x) - y)**2)


I am working on the file "BMLSwPython/01_GettingStarted_withPython.ipynb".
The first task is:
Comparing the time
Can you write Python code for it?

start = timeit.timeit()

X = range(1000)

pySum = sum([n*n for n in X])

end = timeit.timeit()

print("Total time taken: ", end-start)


I am working on the file "BMLSwPython/01_GettingStarted_withPython.ipynb".
The first task is:
Filtering the nan data
Can you write Python code for it?

X = X[~np.isnan(y)]
y = y[~np.isnan(y)]

# checking for nan values
print(sum(np.isnan(X)))
print(sum(np.isnan(y)))

fig, ax = plt.subplots(figsize=(8,6))

ax.plot(X, y, '.b')
ax.margins(0.2)
plt.xticks([w*24*7 for w in range(0, 6)], ["week %d" %w for w in range(0, 6)])
ax.set_xlabel("Week")
ax.set_ylabel("Hits / Week")
ax.set_title("Web Traffic over weeks")


I am working on the file "BMLSwPython/01_GettingStarted_withPython.ipynb".
The first task is:
Learning Scipy
Can you write Python code for it?

# reading the web data

data = sp.genfromtxt("data/web_traffic.tsv", delimiter="\t")
print(data[:3])
print(len(data))


I am working on the file "BMLSwPython/01_GettingStarted_withPython.ipynb".
The first task is:
Preprocessing and Cleaning the data
Can you write Python code for it?

X = data[:, 0]
y = data[:, 1]

# checking for nan values
print(sum(np.isnan(X)))
print(sum(np.isnan(y)))


I am working on the file "BMLSwPython/01_GettingStarted_withPython.ipynb".
The first task is:
Suppose we choose that function with degree 2 is best fit for our data and want to predict that if everything will go same then when we will hit the 100000 count ??
$$  0 = f(x) - 100000 = 0.0105322215 * x^2 - 5.26545650 * x + 1974.6082 - 100000 $$
SciPy's optimize module has the function
fsolve that achieves this, when providing an initial starting position with parameter
x0. As every entry in our input data file corresponds to one hour, and we have 743 of
them, we set the starting position to some value after that. Let fbt2 be the winning
polynomial of degree 2.
Can you write Python code for it?

print(f2)

print(f2 - 100000)

# import
from scipy.optimize import fsolve

reached_max = fsolve(f2-100000, x0=800)/(7*24)
print("100,000 hits/hour expected at week %f" % reached_max[0])


I am working on the file "BMLSwPython/CH01Learn.ipynb".
The first task is:
As we can see that between week3 and week4, there is drastic change in data behaviour, so instead of plotting one line for all the data, we are going to split the data into two parts based on data behaviour
Can you write Python code for it?

# taking n=3.5
split=int(3.5*7*24)
X1 = X[:split]
X2 = X[split:]
y1 = y[:split]
y2 = y[split:]


I am working on the file "BMLSwPython/CH01Learn.ipynb".
The first task is:
Fitting a linear model fp1
Can you write Python code for it?

# starting with linear model where degree is 1
# polyfit() - best put that line into the chart so that it results in the smallest
# approximation error

fp1, residuals, rank, sv, rcond = sp.polyfit(X, y, 1, full=True)
fp1


I am working on the file "BMLSwPython/CH01Learn.ipynb".
The first task is:
Fitting the model with polynomial degree of 2
Can you write Python code for it?

fp2 = sp.polyfit(X, y, 2)
fp2

# fitting the model
f2 = sp.poly1d(fp2)
f2

# checking put the error for fp1 model
rssErr(f2, X, y)

# plotting it
#  setting the plot
plt.figure(figsize=(8,6), dpi=80)

# plot type and color
plt.scatter(X,y, color='green')
l1, = plt.plot(X, f1(X), color='blue', linewidth=3)   #plotting the fucntions
l2, = plt.plot(X, f2(X), color='red', linewidth=3)   #plotting the fucntions

# adding legends
plt.legend([l1, l2], ["d=%i" % f1.order, "d=%i" % f2.order], loc="upper left")
#plt.legend(["d=%i" % f1.order], loc="upper left")
#plt.legend(["d=%i" % f2.order], loc="upper left")

# plot labels
plt.xlabel("X (week) ")
plt.ylabel("y (traffic in thousands) ")
plt.title("Weekly web traffic data")

# grid and autoscale
plt.grid(True, linestyle='-', color='0.75')
plt.autoscale(True)

# ticks renaming
# plt.xticks(x, labels, rotation='vertical')
v1 = [w*7*24 for w in range(10)]
lbl = ["week %i" % i for i in range(10) ]
plt.xticks(v1, lbl, rotation='vertical')
plt.yticks([i*1000 for i in range(10)], ["%i" % i for i in range(10)])

# Pad margins so that markers don't get clipped by the axes
plt.margins(0.2)

# display plot
plt.show()


I am working on the file "BMLSwPython/CH01Learn.ipynb".
The first task is:
Plotting the linear model with these 2 datasets
Can you write Python code for it?

fps1 = sp.polyfit(X1, y1, 1)
fs1 = sp.poly1d(fps1)

fps2 = sp.polyfit(X2, y2, 1)
fs2 = sp.poly1d(fps2)

# error in both
print(rssErr(fs1, X1, y1))
print(rssErr(fs2, X2, y2))

# plotting the data into
plt.figure(figsize=(12,9), dpi=120)

# grid style and autoscale properties
plt.grid(True)
plt.autoscale(True)
plt.margins(0.15)

# assigning labels
plt.title("Web traffic data")
plt.xlabel("Week")
plt.ylabel("Hits")

# re-marking ticks
plt.xticks([w*7*24 for w in range(0, 9)], ["week %d" %i for i in range(0, 9)])

# plotting the data
plt.scatter(X, y, color='green')
ls1, = plt.plot(X1, fs1(X1), color='blue', linewidth=3)
plt.plot(X2, fs1(X2), color='blue', linewidth=3, linestyle='--')
ls2, = plt.plot(X2, fs2(X2), color='red', linewidth=3)
plt.plot(X1[split-40:split], fs2(X1[split-40:split]), color='red', linewidth=3, linestyle='--')

# adding legeds
plt.legend([ls1, ls2], ["sub-dataset1 d=%d " %fs1.order, "subdataset2 d=%d" %fs2.order], loc="upper left")

# display plot
plt.show()


I am working on the file "BMLSwPython/CH01Learn.ipynb".
The first task is:
Trying to fir the model with 53 polynomial
Can you write Python code for it?

fp53 = sp.polyfit(X, y, 53)
fp53

# fitting the model
f53 = sp.poly1d(fp53)
f53

# checking put the error for fp1 model
rssErr(f53, X, y)

# plotting it
#  setting the plot
plt.figure(figsize=(12,9), dpi=80)

# plot type and color
plt.scatter(X,y, color='green')
l1, = plt.plot(X, f1(X), color='blue', linewidth=3)   #plotting the fucntions
l53, = plt.plot(X, f53(X), color='red', linewidth=3)   #plotting the fucntions

# adding legends
plt.legend([l1, l53],["d=%i" % f1.order, "d=%i" % f53.order], loc="upper left")

# plot labels
plt.xlabel("X (week) ")
plt.ylabel("y (traffic in thousands) ")
plt.title("Weekly web traffic data")

# grid and autoscale
plt.grid(True, linestyle='-', color='0.75')
plt.autoscale(True)

# ticks renaming
# plt.xticks(x, labels, rotation='vertical')
v1 = [w*7*24 for w in range(10)]
lbl = ["week %i" % i for i in range(10) ]
plt.xticks(v1, lbl, rotation='vertical')
plt.yticks([i*1000 for i in range(10)], ["%i" % i for i in range(10)])

# Pad margins so that markers don't get clipped by the axes
plt.margins(0.2)

# display plot
plt.show()


I am working on the file "BMLSwPython/CH02.ipynb".
The first task is:
As we have used all the data to train our model, so training accuracy will be > ~95, but this is not what we want, before procceding futher let's check the accuracy
Can you write Python code for it?

sy_pred = knn1.predict(sX)
metrics.accuracy_score(sy, sy_pred)  # it will always be around 1


I am working on the file "BMLSwPython/CH02.ipynb".
The first task is:
But this way is easy with small datasets which has few features. We need to use advance classification technique to classify large and complex data
Can you write Python code for it?

# import
from sklearn.neighbors import KNeighborsClassifier
from sklearn.cross_validation import cross_val_score, train_test_split, KFold
from sklearn import metrics


I am working on the file "BMLSwPython/CH02.ipynb".
The first task is:
KNN Classifier with 1 neighbors
Can you write Python code for it?

# reading SEED dataset
header = ["area", "perimeter", "compactness", "kernel_length", "kernel_width", "asymmetry coefficient",
          "length of kernel groove", "variety"]
seed = sp.genfromtxt("data/seeds_dataset.txt", delimiter="|")
seed

sX = seed[:,0:7]
sy = seed[:,7]

print(sX[:2], sy[:2])

# instantiate the Classifier with neighbors = 1
knn1 = KNeighborsClassifier(n_neighbors=1)

# fitting the data
knn1.fit(sX, sy)

# predicting the type
knn1.predict([[15.26,14.84,0.871,5.763,3.312,2.221,5.22]])


I am working on the file "BMLSwPython/CH02.ipynb".
The first task is:
KNN with N=1, splitting the data into train and test
Can you write Python code for it?

sX_train, sX_test, sy_train, sy_test = train_test_split(sX, sy, test_size=0.4, random_state=4)

knn1 = KNeighborsClassifier(n_neighbors=1)

knn1.fit(sX_train, sy_train)
sy_test_pred = knn1.predict(sX_test)

metrics.accuracy_score(sy_test, sy_test_pred)  # 92% accuracy


I am working on the file "BMLSwPython/CH02.ipynb".
The first task is:
Let's build a model which can identify the type of IRIS flower using simple mathematics
Can you write Python code for it?

# creating boolean filtes
setosa = y==0
versi = y==1
virgi = y==2

#setosa

X_setosa, y_setosa = X[setosa,], y[setosa]
X_versi, y_versi = X[versi,], y[versi]
X_virgi, y_virgi = X[virgi,], y[virgi]

# creating petal length condition
print(X_setosa[:,2].max())
print(X_setosa[:,2].min())

# let's check the same for others but not setosa
print(X[~setosa,2].max())
print(X[~setosa,2].min())

# let's check the same for others individually
print(X_versi[:,2].max())
print(X_versi[:,2].min())

print(X_virgi[:,2].max())
print(X_virgi[:,2].min())


I am working on the file "BOLFI.ipynb".
The first task is:
Although BOLFI is best used with complicated simulators, for demonstration purposes we will use the familiar MA2 model introduced in the basic tutorial, and load it from ready-made examples:
Can you write Python code for it?

from elfi.examples import ma2
model = ma2.get_model(seed_obs=seed)
elfi.draw(model)


I am working on the file "BOLFI.ipynb".
The first task is:
As BOLFI is a more advanced inference method, its interface is also a bit more involved as compared to for example rejection sampling. But not much: Using the same graphical model as earlier, the inference could begin by defining a Gaussian process (GP) model, for which ELFI uses the GPy library. This could be given as an elfi.GPyRegression object via the keyword argument target_model. In this case, we are happy with the default that ELFI creates for us when we just give it each parameter some bounds as a dictionary.
Other notable arguments include the initial_evidence, which gives the number of initialization points sampled straight from the priors before starting to optimize the acquisition of points, update_interval which defines how often the GP hyperparameters are optimized, and acq_noise_var which defines the diagonal covariance of noise added to the acquired points. Note that in general BOLFI does not benefit from a batch_size higher than one, since the acquisition surface is updated after each batch (especially so if the noise is 0!).
Can you write Python code for it?

bolfi = elfi.BOLFI(log_d, batch_size=1, initial_evidence=20, update_interval=10,
                   bounds={'t1':(-2, 2), 't2':(-1, 1)}, acq_noise_var=[0.1, 0.1], seed=seed)


I am working on the file "BOLFI.ipynb".
The first task is:
Fitting the surrogate model
Now we can immediately proceed with the inference. However, when dealing with a Gaussian process, it may be beneficial to take a logarithm of the discrepancies in order to reduce the effect that high discrepancies have on the GP. (Sometimes you may want to add a small constant to avoid very negative or even -Inf distances occurring especially if it is likely that there can be exact matches between simulated and observed data.) In ELFI such transformed node can be created easily:
Can you write Python code for it?

log_d = elfi.Operation(np.log, model['d'])


I am working on the file "BOLFI.ipynb".
The first task is:
Sampling
Finally, samples from the posterior can be acquired with an MCMC sampler. By default it runs 4 chains, and half of the requested samples are spent in adaptation/warmup. Note that depending on the smoothness of the GP approximation, the number of priors, their gradients etc., this may be slow.
Can you write Python code for it?

%time result_BOLFI = bolfi.sample(1000, info_freq=1000)


I am working on the file "BOLFI.ipynb".
The first task is:
Sometimes you may have some samples readily available. You could then initialize the GP model with a dictionary of previous results by giving initial_evidence=result.outputs.
The BOLFI class can now try to fit the surrogate model (the GP) to the relationship between parameter values and the resulting discrepancies. We'll request only 100 evidence points (including the initial_evidence defined above).
Can you write Python code for it?


I am working on the file "BOLFI.ipynb".
The first task is:
The black vertical lines indicate the end of warmup, which by default is half of the number of iterations.
Can you write Python code for it?


I am working on the file "BOLFI.ipynb".
The first task is:
There could be an unnecessarily high number of points at parameter bounds. These could probably be decreased by lowering the covariance of the noise added to acquired points, defined by the optional acq_noise_var argument for the BOLFI constructor. Another possibility could be to add virtual derivative observations at the borders, though not yet implemented in ELFI.
BOLFI Posterior
Above, the fit method returned a BolfiPosterior object representing a BOLFI posterior (please see the paper for details). The fit method accepts a threshold parameter; if none is given, ELFI will use the minimum value of discrepancy estimate mean. Afterwards, one may request for a posterior with a different threshold:
Can you write Python code for it?


I am working on the file "BPSK postprocessing.ipynb".
The first task is:
For clock recovery we use a very simple approach. We use a constant clock frequency of 5 samples per clock cycle and we guess the phase of the clock by trial and error (if we don't do any interpolation or polyphase filtering of the signal, there are only 5 possible phases).
Can you write Python code for it?

bpsk = zz.real
bpsk_f = scipy.signal.convolve(bpsk, h)
start = 0
end = 300
shift = 4
plt.plot(bpsk_f[start+30:end+30])
plt.plot(bpsk[start:end])
plt.plot(np.arange(shift,shift+end-start,5), bpsk_f[np.arange(shift+start+30,shift+end+30,5)], 'x', color='red')
plt.title('Clock recovery')
plt.xlabel('Samples')
plt.legend(['RRC-filtered', 'Unfiltered']);


I am working on the file "BPSK postprocessing.ipynb".
The first task is:
Now we do the same procedure with the second packet.
Can you write Python code for it?

x = np.fromfile('bpsk/packet2.f32', dtype='float32')
t = np.arange(x.size)/samp_rate
plt.plot(t, x)
plt.xlabel('Time (s)')
plt.title('Samples');
z = scipy.signal.hilbert(x)*np.exp(-1j*2*np.pi*(np.arange(x.size))*(1-0.023)/4)
ff = np.fft.fft(z**2)
plt.figure()
plt.plot(np.fft.fftshift(np.fft.fftfreq(z.size, 1/48e3)), np.abs(np.fft.fftshift(ff)))
plt.title('Baseband signal squared (frequency domain)')
plt.xlabel('Frequency (Hz)');
bins = 500
lowpass_f = np.concatenate((np.ones(bins//2), np.zeros(ff.size-bins), np.ones(bins//2)))
carrier_sq = np.fft.ifft(ff * lowpass_f)
phase = np.unwrap(np.angle(carrier_sq))/2
plt.figure()
plt.plot(t, phase)
plt.xlabel('Time (s)')
plt.ylabel('Phase (rad)')
plt.title('Suppressed carrier phase');
ff = np.fft.fft(z**2)
zz = z*np.exp(-1j * phase)
plt.figure()
plt.plot(t, zz.real)
plt.plot(t, zz.imag, alpha=0.5)
plt.title('IQ (suppresed carrier removed)')
plt.legend(['I','Q'])
plt.xlabel('Time (s)');
bpsk = zz.real
bpsk_f = scipy.signal.convolve(bpsk, h)
start = 0
end = 300
shift = 1
plt.figure()
plt.plot(bpsk_f[start+30:end+30])
plt.plot(bpsk[start:end])
plt.plot(np.arange(shift,shift+end-start,5), bpsk_f[np.arange(shift+start+30,shift+end+30,5)], 'x', color='red')
plt.title('Clock recovery')
plt.xlabel('Samples')
plt.legend(['RRC-filtered', 'Unfiltered']);
bits2 = bpsk_f[shift+30::5]
plt.figure()
plt.plot(bits2,'o', alpha=0.1)
plt.title('Soft bits (RRC-filtered)')
plt.figure()
plt.title('Soft bits (unfiltered)')
plt.plot(bpsk[shift::5], 'o', alpha=0.1);


I am working on the file "BPSK postprocessing.ipynb".
The first task is:
The phase of the suppressed carrier varies almost linearly, since the frequency of the signal is not exactly 0Hz. It is interesting to remove the linear trend. The remaining changes in the phase are mostly caused by the satellite's accelaration (from the point of view of the receiver).
Can you write Python code for it?

plt.plot(t, scipy.signal.detrend(phase))
plt.title('Suppresed carrier phase (linear detrend)')
plt.xlabel('Time (s)')
plt.ylabel('Phase (rad)')
plt.title('Suppressed carrier phase');


I am working on the file "BPSK postprocessing.ipynb".
The first task is:
The self-correlation peaks are spaced 1920 bits apart, meaning that there is some structure inside the packet that repeats every 240 bytes.
Can you write Python code for it?

np.diff(np.flatnonzero(corr1 > 500))

np.diff(np.flatnonzero(corr2 > 1000))


I am working on the file "BPSK postprocessing.ipynb".
The first task is:
To investigate the repeating sections, we compute the XOR (product) of each packet with the same packet delayed 1920 bits. We get a section of 1's where the bits coincide.
Can you write Python code for it?

plt.plot(np.sign(bits[:-1920])*np.sign(bits[1920:]))
plt.ylim([-2,2])
plt.title('XOR of packet 1 with packet 1 delayed 1920 bits');

plt.plot(np.sign(bits2[:-1920])*np.sign(bits2[1920:]))
plt.ylim([-2,2])
plt.title('XOR of packet 2 with packet 2 delayed 1920 bits');


I am working on the file "BPSK postprocessing.ipynb".
The first task is:
To obtain the phase of the suppresed carrier, we low-pass filter the signal squared in the frequency domain, then get its phase and divide by 2.
Can you write Python code for it?

bins = 500
lowpass_f = np.concatenate((np.ones(bins//2), np.zeros(ff.size-bins), np.ones(bins//2)))
carrier_sq = np.fft.ifft(ff * lowpass_f)
phase = np.unwrap(np.angle(carrier_sq))/2

plt.plot(t, phase)
plt.xlabel('Time (s)')
plt.ylabel('Phase (rad)')
plt.title('Suppressed carrier phase');


I am working on the file "BPSK postprocessing.ipynb".
The first task is:
To recover the supressed carrier, we compute the square of the signal to remove the BPSK modulation. This gives us a carrier at twice the supressed carrier's frequency.
Can you write Python code for it?

ff = np.fft.fft(z**2)
plt.plot(np.fft.fftshift(np.fft.fftfreq(z.size, 1/48e3)), np.abs(np.fft.fftshift(ff)))
plt.title('Baseband signal squared (frequency domain)')
plt.xlabel('Frequency (Hz)');


I am working on the file "BPSK postprocessing.ipynb".
The first task is:
We open the first packet and plot the samples. The fading in the signal is apparent.
Can you write Python code for it?

samp_rate = 48000
x = np.fromfile('bpsk/packet1.f32', dtype='float32')
t = np.arange(x.size)/samp_rate
plt.plot(t, x)
plt.xlabel('Time (s)')
plt.title('Samples');


I am working on the file "BPSK postprocessing.ipynb".
The first task is:
We use the phase of the suppressed carrier to bring the BPSK signal to the I component.
Can you write Python code for it?

zz = z*np.exp(-1j * phase)
plt.plot(t, zz.real)
plt.plot(t, zz.imag, alpha=0.5)
plt.title('IQ (suppresed carrier removed)')
plt.legend(['I','Q'])
plt.xlabel('Time (s)');


I am working on the file "BSEC2015 Thunder large-scale image SVD demo.ipynb".
The first task is:
As we see from the shape of the resulting components, we have 10 components, each 384 by 256 pixels. We'll plot each of those now.
Can you write Python code for it?

for img in range(components.shape[0]):
    plt.subplot(2, 5, img)
    image(components[img])


I am working on the file "BSEC2015 Thunder large-scale image SVD demo.ipynb".
The first task is:
Perform the SVD
We have our image dataset of faces. Let's now compute the first 10 principal components, using a distributed SVD solver. We can do this by "stacking" all 98,304 images into a matrix, which is then decomposed. In practice, the images never leave whatever machine they're on--rather, the code is sent over the network and each image is annotated with its effective "row" in the matrix, which is used later to perform the actual SVD. All these details are hidden from the user.
Can you write Python code for it?

# Import the PCA method. (http://thunder-project.org/thunder/docs/tutorials/factorization.html#pca)
from thunder import PCA

# Initialize our PCA model with k = 10 components, and then fit that model to our dataset.
model = PCA(k = 10).fit(images)


I am working on the file "BSEC2015 Thunder large-scale image SVD demo.ipynb".
The first task is:
Read in the data
Now that all the setup is done, we're going to read in the image data we deposited earlier on S3 into a distributed data structure. If we were using vanilla Spark, this might be an entire operation unto itself, despite Spark's relatively high level of abstraction. However, since we're using thunder-python, it is built specifically to handle image and time series data, so it implements utility methods that make it very simple to read large image datasets into a robust distributed data structure.
Can you write Python code for it?

# This pulls in all the .tif files at the S3 path and puts them into a Series data structure--an
# array-like entity that is distributed across the cluster.
images = tsc.loadImagesAsSeries("s3://squinn-bsec2015/feret", inputFormat = "tif")
images.cache()


I am working on the file "BSEC2015 Thunder large-scale image SVD demo.ipynb".
The first task is:
That took about an hour to run on all the images with 15 AWS instances. Here we'll reproduce the top 10 principal components.
Can you write Python code for it?

components = model.scores.pack()
components.shape


I am working on the file "BSEC2015 Thunder large-scale image SVD demo.ipynb".
The first task is:
That's it! We've read all the images into a Series, a distributed 1-dimensional array of heterogeneous data. Now we can ask basic questions such as, how many images are we looking at? What does an image look like?
Can you write Python code for it?

# How many images are we looking at?
print images.nrecords

# Let's take a look at the first image.
i = images.toImages().first()[1]
image(i)

# Maybe another image, too?
i = images.toImages().get(2)
image(i)


I am working on the file "BSRTExample.ipynb".
The first task is:
... and we can also perform a moving average over the data
Can you write Python code for it?


I am working on the file "BSRTExample.ipynb".
The first task is:
... or plot only specific slots
Can you write Python code for it?


I am working on the file "BSRTExample.ipynb".
The first task is:
Dictionary with emittances and timestamps for each slot
Can you write Python code for it?


I am working on the file "BSRTExample.ipynb".
The first task is:
Fitting the emittance
fit the emittance between tstart and tend
Can you write Python code for it?


I am working on the file "BSRTExample.ipynb".
The first task is:
For each (tstart,tend) the fitting data is stored in this dictionary. This means you can e.g. fit your data from [t0,t1], [t1,t2], etc., plot the complete data and then plot the individual fits on top.
Can you write Python code for it?


I am working on the file "BSRTExample.ipynb".
The first task is:
Getting the BSRT data from timber
generate the BSRT instance, which automatically calculates the emittances using the data stored in timber.
Can you write Python code for it?


I am working on the file "BSRTExample.ipynb".
The first task is:
Here we perform the fit. The function bsrt.plot(..., fit=True) and bsrt.plot_fit() automatically generate this data, if no entry for the desired timestamp is found in bsrt.emitfit. Just to show it, we use here the underlying function bsrt.fit.
Can you write Python code for it?

for ts,te in [[t0fit,t1fit],[t1fit,t2fit],[t2fit,t3fit],[t3fit,t4fit]]:
    bsrt.fit(ts,te,force=True)


I am working on the file "BSRTExample.ipynb".
The first task is:
The fit data from tstart to tend ist then stored in bsrt.emitfit.
Can you write Python code for it?


I am working on the file "BSRTExample.ipynb".
The first task is:
and now we can do the plot
Can you write Python code for it?

plt.figure()
for slot,c in zip([1062,1074],['b','r']):
    # plot the averaged data
    bsrt.plot(plane='v',t1=t0fit,t2=t4fit,slots=slot,avg=None,fit=False,color=c)
    # now add the fit data with a black line
    for ts,te in [[t0fit,t1fit],[t1fit,t2fit],[t2fit,t3fit],[t3fit,t4fit]]:
        bsrt.plot_fit(plane='v',t1=ts,t2=te,slots = slot, color=c)


I am working on the file "BSRTExample.ipynb".
The first task is:
what slots do we have?
Can you write Python code for it?


I am working on the file "BSidesLV -- your model isn't that special -- (1) MLP.ipynb".
The first task is:
Multilayer perceptron
We'll use the features we extracted to train a multilayer perceptron (MLP).  An MLP is an artificial neural network with at least one hidden layer.  Is a multilayer perceptron "deep learning"?  Well, it's a matter of semantics, but "deep learning" may imply that the features and model are optimized together, end-to-end.  So, it that sense, no: since we're using domain knowledge to extract features, then pass it to an artificial neural network, we'll remain conservative and call this an MLP. (As we'll see, don't get fooled just because we're not calling this "deep learning": this MLP is no slouch.) The network architecture is defined in classifier/simple_multilayer.py.
Can you write Python code for it?

# StandardScaling the data can be important to multilayer perceptron
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler().fit(X_train)

# Note that we're using scaling info form X_train to transform both
X_train = scaler.transform(X_train) # scale for multilayer perceptron
X_test = scaler.transform(X_test)

from classifier import simple_multilayer
from keras.callbacks import LearningRateScheduler, EarlyStopping, ReduceLROnPlateau, ModelCheckpoint
model = simple_multilayer.create_model(
    input_shape=(X_train.shape[1], ),     # input dimensions
    input_dropout=0.05,                   # this prevents the model becoming a fanboy of (overfitting to) any particular input feature
    hidden_dropout=0.1,                   # same, but for hidden units.  Dropping out hidden layers can create a sort of ensemble learner
    hidden_layers=[4096, 2048, 1024, 512] # this is "art". making up # of hidden layers and width of each. don't be afraid to change this
)
model.fit(X_train, y_train,
          batch_size=128,
          epochs=200,
          verbose=1,
          callbacks=[
              EarlyStopping( patience=20 ),
              ModelCheckpoint( 'multilayer.h5', save_best_only=True),
              ReduceLROnPlateau( patience=5, verbose=1)],
          validation_data=(X_test, y_test))

from keras.models import load_model
# we'll load the "best" model (in this case, the penultimate model) that was saved
# by our ModelCheckPoint callback
model = load_model('multilayer.h5')

y_pred = model.predict(X_test)
common.summarize_performance(y_pred, y_test, "Multilayer perceptron")
# The astute reader will note we should be doing this on a separate holdout, since we've explicitly
# saved the model that works best on X_test, y_test...an exercise for left for the reader...


I am working on the file "BSidesLV -- your model isn't that special -- (2) End-to-End.ipynb".
The first task is:
End-to-end deep learning for malware
So, let's move to "real" end-to-end deep learning, because deep learning does everything better, right?  For this, there's a few things to note.
1. Images have a notion of pixel intensity which has a natural ordering: black < gray < white.  But binaries are made of bytes, some of which represents instructions, some ASCII or unicode text (depending on placement and context in the file), and the byte value has no real logical ordering.  In our model, we'll let the model choose a mapping from byte value to "color", where you can specify how many dimensions make up each "color".  The layer that does this is called an embedding layer.
2. Even hefty GPUs may have a hard time holding lots of embedded binary files in contiguous memory.  For this reason, we'll set a cap on the maximum number of bytes we'll read in (hint: the median size of malware on VirusTotal is close to 1MB), and furthermore, break the file into chunks that the memory manager can conveniently place into contiguous memory on the GPU.  (Even though you may have 4GB of memory on your GPU, you may not be able to hold a 1MB file with its embedded/"colored" bytes in contiguous memory.)
3. Speaking of memory usage, we can limit the memory usage by the batch size.
But, think of it!  End-to-end deep learing for static malware detection. No PE parsing required! No feature engineering required! No work required!  Right?
You can find code that defines the end-to-end model architecture at classifier/endtoend.py.
Can you write Python code for it?

# for this demo, will slurp in only the first 256K (2**18) bytes of the file
# for a nice GPU like a Titan X, you should be able to squeeze in > 2MB ...
# ...but warning! this makes training more difficult...larger haystack to find needles
max_file_length = int(2**18) # powers of 2 FTW
file_chunks = 8  # break file into this many chunks
file_chunk_size = max_file_length // file_chunks

batch_size = 8

# That this is very long running cell, and we're going
# it may appear that the output is truncated before training completes

# let's train this puppy
from classifier import endtoend
import math
from keras.callbacks import LearningRateScheduler, EarlyStopping, ModelCheckpoint, ReduceLROnPlateau

# create_model(input_shape, byte_embedding_size=2, input_dropout=0.05, hidden_dropout=0.05, kernel_size=16, n_filters_per_layer=[64,256,1024], n_mlp_layers=2 )
model_e2e = endtoend.create_model(input_shape=(file_chunks, file_chunk_size))
train_generator = common.generator(list(zip(sha256_train, y_train)), batch_size, file_chunks, file_chunk_size)
test_generator = common.generator(list(zip(sha256_test, y_test)), 1, file_chunks, file_chunk_size)
training_history = model_e2e.fit_generator(train_generator,
                        steps_per_epoch=math.ceil(len(sha256_train) / batch_size),
                        epochs=100,
                        callbacks=[
                            EarlyStopping( patience=10 ),
                            ModelCheckpoint( 'endtoend.h5', save_best_only=True),
                            ReduceLROnPlateau( patience=5)],
                        validation_data=test_generator,
                        validation_steps=len(sha256_test))


I am working on the file "BSidesLV -- your model isn't that special -- (3) MalwaResNet.ipynb".
The first task is:
MalwaResNet
Clearly, my model needs a heavy dose of special fanciness.  We'll turn to ResNet architecture!  This builds on the end-to-end model in the previous notebook, but stacks things higher and deeper, thanks to our hand-crafted residual cell.
This model may require a hefty GPU.  I ran it successfully on a single TITAN X (Pascal).  If yours doesn't work, try enabling the lite=True options in create_model.
You can find code that defines the MalwaResNet model architecture at classifier/malwaresnet.py.
Can you write Python code for it?

# for this demo, will slurp in only the first 256K (2**18) bytes of the file
max_file_length = int(2**18)
file_chunks = 16  # break file into this many chunks
file_chunk_size = max_file_length // file_chunks
batch_size = 4

# That this is very long running cell, and
# it may appear that the output is truncated before training completes

# let's train this puppy
from classifier import malwaresnet
import math
from keras.callbacks import LearningRateScheduler, EarlyStopping, ModelCheckpoint

# create_model(input_shape, byte_embedding_size=2, lite=False)
model_malwaresnet = malwaresnet.create_model(input_shape=(file_chunks, file_chunk_size), byte_embedding_size=2)
train_generator = common.generator(list(zip(sha256_train, y_train)), batch_size, file_chunks, file_chunk_size)
test_generator = common.generator(list(zip(sha256_test, y_test)), 1, file_chunks, file_chunk_size)
training_history = model_malwaresnet.fit_generator(train_generator,
                        steps_per_epoch=math.ceil(len(sha256_train) / batch_size),
                        epochs=20,
                        callbacks=[
                            EarlyStopping( patience=1 ),
                            ModelCheckpoint( 'malwaresnet.h5', save_best_only=True),
                            LearningRateScheduler(lambda epoch: common.schedule(epoch, start=0.1, decay=0.5, every=1))],
                        validation_data=test_generator,
                        validation_steps=len(sha256_test))


I am working on the file "BSidesLV -- your model isn't that special -- (3) MalwaResNet.ipynb".
The first task is:
Okay, no more snarkiness about impatient millenials from me.  Each epoch is taking 31000 s = 517 min = 8hrs 36 minutes.  But, we're going to remain optimistic that this is going to be awesome.  With a name like MalwaResNet, how could it not be?
Can you write Python code for it?

from keras.models import load_model
# we'll load the "best" model (in this case, the penultimate model) that was saved
# by our ModelCheckPoint callback
model_malwaresnet = load_model('malwaresnet.h5')
# we could load the "best" model, but in this case, the "best" model is the penultimate, and not much better
# than the model we have in hand
y_pred = []
for sha256, lab in zip(sha256_test, y_test):
    y_pred.append(
        model_malwaresnet.predict_on_batch(
            np.asarray([common.get_file_data(sha256, lab, max_file_length)]).reshape(
                (-1, file_chunks, file_chunk_size))
        )
    )
common.summarize_performance(np.asarray(y_pred).flatten(), y_test, "End-to-end convnet")


I am working on the file "BString-Copy1.ipynb".
The first task is:
3) All derivatives of AString can be represented via AString itself:
AString'(x) = AString(2x + 1) - AString(2x - 1)
4) Combination of Atomic Strings can exactly represent a straight line:
x = AString(x) + Astring(x+1) + Astring(x+2)...
Can you write Python code for it?

x = np.arange(-40.0, 40.0, 0.01)

#pl.plot(x, ABline (x, 1, 0),            label='ABLine 1*x')
pl.plot(x, AString(x, 10.0,-15, 10, -15),   '--',  linewidth=1, label='AString 1')
pl.plot(x, AString(x, 10.0, -5, 10, -5),    '--',  linewidth=1, label='AString 2')
pl.plot(x, AString(x, 10.0, +5, 10, +5),    '--',  linewidth=1, label='AString 3')
pl.plot(x, AString(x, 10.0,+15, 10, +15),   '--',  linewidth=1, label='AString 4')

AS2 = Sum(AString(x, 10.0, -15, 10, -15), AString(x, 10., -5, 10, -5))
AS3 = Sum(AS2, AString(x, 10, +5, 10, +5))
AS4 = Sum(AS3, AString(x, 10,+15, 10, +15))
pl.plot(x, AS4,  label='AStrings Joins', linewidth=2)

pl.title('Atomic Strings Combinations')
pl.legend(loc='best', numpoints=1)
pl.grid(True)
pl.show()