I am working on the file "FBA_tutorials/2_introduction_to_FBA_FVA_RECON2.ipynb".
The first task is:
You may ignore the biomass_other reaction.
Assignment (3 min): Would you say this is a reasonable medium for a human cell to live in? What are the limiting medium components currently? Is there an oxygen or carbon limitation? What is the carbon source?
Performing Flux Balance Analysis
What is FBA again?
We will use the constraint-based modeling technique called flux balance analysis (FBA) (http://doi.org/10.1038/nbt.1614). Given the metabolic network this method computationally calculates the flux distribution, meaning the fluxes through all reactions in the model at steady-state, that is optimal according to a specified objective function. In our case, the objective function will usually be the flux through the biomass reaction. This reaction contains (when known) experimentally determined ratio's of metabolites that make of the composition of a cell.
FBA has traditionally been applied to metabolism as we are doing here. FBA primarily makes use of the stoichiometry matrix $S$. This matrix is of size $m \times r$ where $m$ is the number of species or metabolites in the system and $r$ is the number of reactions. Every row of $S$ specifies for a specific metabolite in what quantity it partakes in each reaction. Therefore, each element $(i,j)$ of $S$ contains the stoichiometric coefficient of metabolite $i$ in reaction $j$.
The word flux is used to describe a reaction rate at steady state. In FBA the aim is to find the flux distributions of the network that satisfy (1) the steady-state condition, this is based on the assumption that metabolism occurs on a fast time-scale compared to gene regulatory events and thus reaction rates are constant, (2) thermodynamic feasibility , i.e. some reactions are known to be irreversible, (3) maximal flux constraints when these are known and (4) maximize some objective like growth.
For the brave
Mathematically, this can be concisely summarized as
\begin{align}
\text{Maximize } Z &=c^T v, \text{ such that } \
Sv &= 0 \
\alpha_k \leq & ~ v_k \leq...
Can you write Python code for it?


I am working on the file "FBA_tutorials/3_Phenylketonuria_network_illustration.ipynb".
The first task is:
Assignment (2 min):
* How is it possible that there is still equal biomass reaction flux?
* The flux variability result shows us the interval for EX_phe_L_e is entirely negative. What does this mean?
* Why is PHETHPTOX2 not essential here? I.e. why does FVA allow its minimum to be zero?
Note: The recycling pathway of biopterin.
Can we knock  out one of the two conversion enzymes and get the other one?
Can you write Python code for it?

model.reactions.r0399.upper_bound = 0;

fbasol = pfba(model)
fvasol = cobra.flux_analysis.flux_variability_analysis(model,reaction_list=rxnsOfInterest,fraction_of_optimum=0,remove_cycles=True)
fvasol

b = show_map(fbasol,map_loc)
b.display_in_notebook()


I am working on the file "FBA_tutorials/3_Phenylketonuria_network_illustration.ipynb".
The first task is:
Assignment (2 min): In the cell below add the code to knockout the gene and see what happens. Does the result make sense?
Can you write Python code for it?

### KNOCKOUT THE GENE HERE

fbasol = pfba(model)
fvasol = cobra.flux_analysis.flux_variability_analysis(model,reaction_list=rxnsOfInterest,fraction_of_optimum=0,remove_cycles=True)
fvasol

b = show_map(fbasol,map_loc)
b.display_in_notebook()


I am working on the file "FBA_tutorials/3_Phenylketonuria_network_illustration.ipynb".
The first task is:
Assignment (2 min):Is PHETHPTOX2 now essential? ie. is it required to carry flux to get growth?
knockout PAH and its alternative reaction by knocking out the gene
Can you write Python code for it?


I am working on the file "FBA_tutorials/3_Phenylketonuria_network_illustration.ipynb".
The first task is:
Phenylalanine Catabolism Is Genetically Defective in Some People}
Assignment (10 min): Read the text below and study the images.
The following is taking from Lehninger's principles of biochemistry.
Given that many amino acids are either neuro-transmitters or precursors or antagonists of neutrotransmitters, genetic defects of amino acid metabolism can cause defective neural development and mental retardation. In most such diseases specific intermediates accumulate. For example, a genetic defect in phenylalanine hydroxylase, the first enzyme in the catabolic pathway for phenylalanine, is responsible for the disease phenylketonuria (PKU), the most common cause of elevated levels of phenylalanine (hyperphenylalaninemia). Phenylalanine hydroxylase (also called phenylalanine-4-monooxygenase) is one of a general class of enzymes called mixed-function oxidases, all of which catalyze simultaneous hydroxylation of a substrate by an oxygen atom of $O_2$ and reduction of the other oxygen atom to $H_2O$. Phenylalanine hydroxylase requires the cofactor tetrahydrobiopterin, which carries electrons from NADH to $O_2$ and becomes oxidized to dihydrobiopterin in the process. It is subsequently reduced by the enzyme dihydrobiopterin reductase in a reaction that requires NADH.
In individuals with PKU, a secondary, normally little-used pathway of phenylalanine metabolism comes into play. In this pathway phenylalanine undergoes transamination with pyruvate to yield phenylpyruvate. Phenylalanine and phenylpyruvate accumulate in the blood and tissues and are excreted in the urine—hence the name “phenylketonuria.” Much of the phenylpyruvate, rather than being excreted as such, is either decarboxylated to phenylacetate or reduced to phenyllactate. Phenylacetate imparts a characteristic odor to the urine, which nurses have traditionally used to detect PKU in infants. The accumulation of phenylalanine or its metabolites in early life impairs normal development of the brain, causing severe mental...
Can you write Python code for it?

# write a for loop here.
# tip: make use of model.reactions and of the .reaction attribute of each reaction


I am working on the file "FBA_tutorials/4_biomarker_prediction_Schlomi_example.ipynb".
The first task is:
Reproducing Figure 1B
We have written a python function that implements the entire method proposed by Shlomi et al in the "findBiomarkers()" function. For now we will simply check to see if it produces the same qualitative results as the original paper.
We have reconstructed the network in Figure 1A. We will now block reaction 1 and show that we predict what is in Figure 1B.
Can you write Python code for it?

exchanges = [ rxn for rxn in model.reactions if rxn.products == [] ]
exchangesIds = [rxn.id for rxn in exchanges]

findBiomarkers(model,fvaRxns=exchangesIds,mods=[model.reactions.R1.id],cutoff=0)


I am working on the file "FBA_tutorials/5_biomarker_prediction_PKU.ipynb".
The first task is:
Assignment (3 min): If you see the prediction of the biofluids/tissue as the brain tissue: does the model correctly predict issues with neurotransmitters in the brain?
Assignment (10 min):
What biomarkers are predicted when you focus on blocking the cofactor biopterin recycling reactions that also produce PKU? To get you started we included some code below.
Can you write Python code for it?

model.reactions.DHPR.gene_reaction_rule, model.reactions.DHPR.reaction

model.reactions.DHPR2.gene_reaction_rule,model.reactions.DHPR2.reaction

model.reactions.r0398.gene_reaction_rule,model.reactions.r0398.reaction

# UNCOMMENT & FIX THE LINE BELOW
# findBiomarkers(model,fvaRxns=biomarkers_to_check,mods=[ADD THE RELEVANT GENES HERE],synchronous=True)


I am working on the file "FBA_tutorials/extra_exploring_ecoli_core.ipynb".
The first task is:
Assignment: Examine the network.
Can you write Python code for it?

b = show_map([],map_loc)
b.display_in_notebook()

sol = model.optimize()
b = show_map(sol,map_loc)
b.display_in_notebook()


I am working on the file "FBA_tutorials/extra_generate_COBRA_model_geenen_2012.ipynb".
The first task is:
Fix reaction reversibility
Can you write Python code for it?

model.reactions.v_v1.lower_bound = 0 # met => SAM
model.reactions.v_v2.lower_bound = 0 # met => SAM
model.reactions.v_v3.lower_bound = 0 # SAM => SAH
model.reactions.v_v4.lower_bound = 0 #
model.reactions.v_v6.lower_bound = 0 # hcy => met
model.reactions.v_v12.lower_bound = 0 # 2 GSH -> GSSG
model.reactions.v_v13.lower_bound = 0 # GSSG -> 2 GSH
model.reactions.v_v14.lower_bound = 0 # cGSSG => bGSSG
model.reactions.v_v15.lower_bound = 0 # cGSSG => bGSSG
model.reactions.v_v16.lower_bound = 0 #
model.reactions.v_v17.lower_bound = 0 #
model.reactions.v_v23.lower_bound = 0 # cTHF => cCH2THF
model.reactions.v_v25.lower_bound = 0 # oxo -> cglut
model.reactions.v_v33.lower_bound = 0 # cGSH + para => ASG
model.reactions.v_v34.lower_bound = 0 # cGSH + para => ASG
model.reactions.v_v40.lower_bound = 0 # bGSSG -> 2.0 bcys
model.reactions.v_v41.lower_bound = 0 #


I am working on the file "FBA_tutorials/extra_generate_COBRA_model_geenen_2012.ipynb".
The first task is:
Generate an updated JSON model for (Geenen,2012)
Can you write Python code for it?


I am working on the file "FBA_tutorials/extra_generate_COBRA_model_geenen_2012.ipynb".
The first task is:
Modify the published SBML model and export it as a json for use with Escher
* Clearly mark exchange reactions with EX_ in the ID
* Exchange reactions should always carry negative flux to take up a metabolite
* Add paracetamol explictly to the model and FORCE its uptake
Can you write Python code for it?

exchanges = [rxn for rxn in model.reactions if rxn.products == [] or rxn.reactants == []]
for r in exchanges:
    if 'EX_' not in r.id:
        r.id = 'EX_'+r.id
    model.repair()
    if list(r.metabolites.values())[0] == 1 :
        r.add_metabolites({list(r.metabolites.keys())[0]:-2})
        model.repair()
    r.lower_bound = -1; r.upper_bound = 1000
    print(r.id,r.reaction)

# Add paracetamol
para = Metabolite('para')
model.add_metabolites(para)
EX_para = Reaction('EX_para')
model.add_reaction(EX_para)
model.reactions.EX_para.add_metabolites({'para':-1})
model.reactions.EX_para.lower_bound = -1; model.reactions.EX_para.upper_bound = -1
model.reactions.v_v33.add_metabolites({'para':-1})
model.reactions.v_v34.add_metabolites({'para':-1})

# fix reaction 29 bug
# In the kinetic model bgly is considered to be a reservoir
# Upon importing into FBA this leads to this imbalanced reaction where cys = cysgly
# We solve this by making this reaction produce gly in the cytosol
# this is biochemically a bit strange but topologically does not matter
model.reactions.v_v29.add_metabolites({'cgly':1})

cobra.io.save_json_model(model,'models/Geenen_cobra_model.json')
exchanges


I am working on the file "FBA_tutorials/extra_generate_COBRA_model_geenen_2012.ipynb".
The first task is:
Show all reactions and their bounds
Can you write Python code for it?

df = pd.DataFrame.from_dict({'ID':[r.id for r in model.reactions],
                             'Reaction': [r.reaction for r in model.reactions],
                             'LB':[r.lower_bound for r in model.reactions],
                             'UB':[r.upper_bound for r in model.reactions] })
df[['ID','Reaction','LB','UB']]


I am working on the file "FBA_tutorials/minibook-2nd-code-master/chapter2/cleaning/cleanup.ipynb".
The first task is:
Cleanup headers
First, we remove extra spaces in the headers.
Can you write Python code for it?

fare.columns = [col.strip() for col in fare.columns]


I am working on the file "FBA_tutorials/minibook-2nd-code-master/chapter2/cleaning/cleanup.ipynb".
The first task is:
Reorder rows by date
Then, we reorder the rows by date.
Can you write Python code for it?

data.pickup_datetime = pd.to_datetime(data.pickup_datetime)
data.dropoff_datetime = pd.to_datetime(data.dropoff_datetime)
fare.pickup_datetime = pd.to_datetime(fare.pickup_datetime)

sorted_index = data.sort('pickup_datetime').index

data = data.ix[sorted_index]
fare = fare.ix[sorted_index]


I am working on the file "FBA_tutorials/minibook-2nd-code-master/chapter2/cleaning/cleanup.ipynb".
The first task is:
Save the cleaned datasets
Finally, we save the cleaned-up datasets.
Can you write Python code for it?

data.to_csv('../data/nyc_data.csv', index=False)
fare.to_csv('../data/nyc_fare.csv', index=False)


I am working on the file "FBA_tutorials/minibook-2nd-code-master/chapter2/cleaning/cleanup.ipynb".
The first task is:
We get rid of values outside this area.
Can you write Python code for it?

lon_p, lat_p = data.pickup_longitude, data.pickup_latitude
lon_d, lat_d = data.dropoff_longitude, data.dropoff_latitude

to_keep = ((lon_p > lon_min) &
           (lon_p < lon_max) &
           (lat_p > lat_min) &
           (lat_p < lat_max) &
           (lon_d > lon_min) &
           (lon_d < lon_max) &
           (lat_d > lat_min) &
           (lat_d < lat_max))


I am working on the file "FBA_tutorials/minibook-2nd-code-master/chapter2/cleaning/cleanup.ipynb".
The first task is:
We keep those rows where the coordinates are within the rectangle.
Can you write Python code for it?

data = data.ix[to_keep]
fare = fare.ix[to_keep]


I am working on the file "FBA_tutorials/minibook-2nd-code-master/chapter3/34-computing.ipynb".
The first task is:
Finally, we'll need the dropoff coordinates in the following:
Can you write Python code for it?


I am working on the file "FBA_tutorials/minibook-2nd-code-master/chapter3/34-computing.ipynb".
The first task is:
Here, we've selected the elements at [1, 1], [3, 1], and [5, 1]. The slicing syntax in Python is start:end:step where start is included and end is excluded. If start or end are omitted, they default to 0 or the length of the dimension, respectively, whereas step defaults to 1. For example, 1: is equivalent to 1:n:1 where n is the size of the axis.
Let's select the longitudes of all pickup locations, in other words, the first column:
Can you write Python code for it?


I am working on the file "FBA_tutorials/minibook-2nd-code-master/chapter3/34-computing.ipynb".
The first task is:
INFO (Indexing in NumPy): Indexing in Python and NumPy starts at 0, and excludes the last element. The first half of weights is made of weights[0], weights[1], up to weights[n-1]. There are n of such elements. The slice weights[:n] is equivalent to weights[0:n]: it starts at weights[0], and ends at weights[n] excluded, so the last element is effectively weights[n-1].

We could also have used array manipulation routines provided by NumPy, such as np.tile() to concatenate copies of an array along several dimensions, or np.repeat() to make copies of every element along several dimensions. You will find the list of manipulation functions at http://docs.scipy.org/doc/numpy/reference/routines.array-manipulation.html.
Next, we create a (2n, 2) array defined by the vertical concatenation of the pickup and dropoff locations for the evening trips:
Can you write Python code for it?

points = np.r_[pickup[evening],
               dropoff[evening]]

points.shape


I am working on the file "FBA_tutorials/minibook-2nd-code-master/chapter3/34-computing.ipynb".
The first task is:
In NumPy, symbols like arithmetic, inequality, and boolean operators work on ndarrays on an element-wise basis. Here is how to select all trips where the longitude is between lon_min and lon_max:
Can you write Python code for it?

in_lon = (lon_min <= lon) & (lon <= lon_max)
in_lon


I am working on the file "FBA_tutorials/minibook-2nd-code-master/chapter3/34-computing.ipynb".
The first task is:
Selection and indexing
Let's illustrate selection and indexing with NumPy. These operations are similar to those offered by pandas on DataFrame and Series objects.
In NumPy, a given element can be retrieved with pickup[i, j], where i is the 0-indexed row number, and j is the 0-indexed column number:
Can you write Python code for it?


I am working on the file "FBA_tutorials/minibook-2nd-code-master/chapter3/34-computing.ipynb".
The first task is:
The concise np.r_[] syntax allows us to concatenate arrays along the first (vertical) dimension. We could also have used more explicit manipulation functions such as np.vstack() or np.concatenate().
Now, we convert these points from geographical coordinates to pixel coordinates, using the same function as in the previous chapter.
Can you write Python code for it?

def lat_lon_to_pixels(lat, lon):
    lat_rad = lat * np.pi / 180.0
    lat_rad = np.log(np.tan((lat_rad + np.pi / 2.0) / 2.0))
    x = 100 * (lon + 180.0) / 360.0
    y = 100 * (lat_rad - np.pi) / (2.0 * np.pi)
    return (x, y)

lon, lat = points.T
x, y = lat_lon_to_pixels(lat, lon)


I am working on the file "FBA_tutorials/minibook-2nd-code-master/chapter3/34-computing.ipynb".
The first task is:
The np.nonzero() function returns the indices corresponding to True in a boolean array, as shown here:
Can you write Python code for it?


I am working on the file "FBA_tutorials/minibook-2nd-code-master/chapter3/34-computing.ipynb".
The first task is:
The result is a 1D ndarray.
We also get the second column of pickup:
Can you write Python code for it?


I am working on the file "FBA_tutorials/minibook-2nd-code-master/chapter3/34-computing.ipynb".
The first task is:
Then, we get all trips where both the longitude and latitude belong to our rectangle:
Can you write Python code for it?

in_lonlat = in_lon & in_lat
in_lonlat.sum()


I am working on the file "FBA_tutorials/minibook-2nd-code-master/chapter3/34-computing.ipynb".
The first task is:
These two arrays contain the horizontal and vertical bins.
Finally, we compute the histogram with the np.histogram2d() function. We pass as arguments the y, x coordinates of the points (reversed because we want the grid's first axis to represent the y coordinate), the weights, and the bins. This function computes a weighted sum of the points, in every bin. It returns several objects, the first of which is the density map we are interested in:
Can you write Python code for it?

grid, _, _ = np.histogram2d(y, x, weights=weights,
                            bins=(bins_y, bins_x))


I am working on the file "FBA_tutorials/minibook-2nd-code-master/chapter3/34-computing.ipynb".
The first task is:
This logistic function is called the expit function. It can also be found in the SciPy package at scipy.special.expit(). scipy.special provides many other special functions such as Bessel functions, Gamma functions, hypergeometric functions, and so on.
Finally, we display the density map with plt.imshow():
Can you write Python code for it?

plt.figure(figsize=(8, 8))
plt.imshow(density,
           origin='lower',
           interpolation='bicubic'
           )
plt.axis('off')
plt.tight_layout()


I am working on the file "FBA_tutorials/minibook-2nd-code-master/chapter3/34-computing.ipynb".
The first task is:
We have made extensive use of trigonometric functions provided by NumPy: np.radians() (converting numbers from degrees into radians), np.cos(), np.sin(), np.arctan2(x, y) (returning the arctangent of x/y), and so on. These mathematical functions are defined on real numbers, but NumPy provides vectorized versions of them. These vectorized functions not only work on numbers but also on arbitrary numerical ndarrays. As we have explained earlier, these functions are orders of magnitude faster than Python loops. You will find the list of mathematical functions in NumPy at http://docs.scipy.org/doc/numpy/reference/routines.math.html.
All in all, NumPy makes it quite natural to implement mathematical formulas on arrays of numbers. The syntax is exactly the same as with scalar operations.
Now, let's compute the straight line distances of all taxi trips:
Can you write Python code for it?

distances = geo_distance(lon, lat, lon1, lat1)


I am working on the file "FBA_tutorials/minibook-2nd-code-master/chapter3/34-computing.ipynb".
The first task is:
We load the NYC taxi dataset with pandas:
Can you write Python code for it?

data = pd.read_csv('../chapter2/data/nyc_data.csv',
                   parse_dates=['pickup_datetime', 'dropoff_datetime'])


I am working on the file "FBA_tutorials/minibook-2nd-code-master/chapter3/34-computing.ipynb".
The first task is:
We now define the bins for the 2D histogram in our density map. This defines a 2D grid over which to compute the histogram.
Can you write Python code for it?

lon_min, lat_min = -74.0214, 40.6978
lon_max, lat_max = -73.9524, 40.7982

x_min, y_min = lat_lon_to_pixels(lat_min, lon_min)
x_max, y_max = lat_lon_to_pixels(lat_max, lon_max)

bin = .00003
bins_x = np.arange(x_min, x_max, bin)
bins_y = np.arange(y_min, y_max, bin)


I am working on the file "FBA_tutorials/minibook-2nd-code-master/chapter3/34-computing.ipynb".
The first task is:
matplotlib's plt.hist() function computes a histogram and plots it. It is a convenient wrapper around NumPy's np.histogram() function that just computes a histogram. You will find more statistical functions in NumPy at http://docs.scipy.org/doc/numpy/reference/routines.statistics.html.
A density map with NumPy
We have reviewed the most common array operations in this section. We will now see a more advanced example combining several techniques. We will compute and display a 2D density map of the most common pickup and dropoff locations, at specific times in the day.
First, let's select the evening taxi trips. This time, we use pandas, which offers particularly rich date and time features. We eventually get a NumPy array with the .values attribute:
Can you write Python code for it?

evening = (data.pickup_datetime.dt.hour >= 19).values

n = np.sum(evening)

n


I am working on the file "FC_and_CNN.ipynb".
The first task is:
CNN
Can you write Python code for it?

from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation, Flatten
from keras.layers import Conv2D, MaxPooling2D
from keras.optimizers import Adam
from keras.layers.normalization import BatchNormalization
from keras.callbacks import EarlyStopping

train_cnn_model = True
def catdog_cnn(input_image_shape):
    model = Sequential()
    model.add(Conv2D(32, (3, 3),input_shape=input_image_shape))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(Conv2D(32, (3, 3)))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))

    model.add(Conv2D(64, (3, 3)))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(Conv2D(64, 3, 3))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Dropout(0.25))

    model.add(Conv2D(128, (3, 3)))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(Conv2D(128, (3, 3)))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Dropout(0.25))

    model.add(Flatten())
    model.add(Dense(256, activation='relu'))
    model.add(Dropout(0.5))
    model.add(Dense(1))

    model.add(Activation('sigmoid'))
    optimizer = Adam()
    objective = 'binary_crossentropy'
    model.compile(loss=objective, optimizer=optimizer, metrics=['accuracy'])
    return model

if train_cnn_model:
    cnn_model = catdog_cnn(input_image_shape)
else:
    cnn_model = keras.models.load_model("cnn_model_trained.h5")
cnn_model.summary()

if train_cnn_model:
    nb_epoch=80
    print("Model compiled, start training")
    early_stopping_callback = EarlyStopping(monitor='val_loss', min_delta=0, patience=8,
                                            verbose=1, mode='auto')
    history = cnn_model.fit(train_features, train_labels,validation_data=(val_features,val_labels),
   ...


I am working on the file "FC_and_CNN.ipynb".
The first task is:
Preprocessing
Can you write Python code for it?

#media = np.mean(train_features,axis=(0,2,3),keepdims=True)
#print(media.shape,media.ravel())

#media = media[:,np.newaxis,np.newaxis]
#train_features-=media
#val_features-=media
#test_features-=media

from keras.applications.imagenet_utils import preprocess_input

preprocesado = False

if not preprocesado:
    train_features = preprocess_input(train_features)
    val_features = preprocess_input(val_features)
    test_features = preprocess_input(test_features)
    preprocesado = True


I am working on the file "FDHighOrder.ipynb".
The first task is:
A measure of internal stability known as $h$-ellipticity is defined by
$$ E^h(L) = \frac{\min_{\pi/2 \le |\theta| \le \pi} L(\theta)}{\max_{|\theta| \le \pi} L(\theta)} . $$

What is $E^h(L)$ for the second order "version 2" stencil?
How about for uncentered formulas and higher order?

Spectral collocation
Suppose that instead of using only a fixed number of neighbors in our differencing stencil, we use all points in the domain?
Can you write Python code for it?

n = 10
x = numpy.linspace(-1, 1, n)
L = numpy.zeros((n,n))
for i in range(n):
    L[i] = -fdstencil(x[i], x)[2]

u = numpy.cos(3*x)
pyplot.plot(x, L.dot(u), 'o')
pyplot.plot(x, 9*u);


I am working on the file "FDHighOrder.ipynb".
The first task is:
Differentiation
We can differentiate Chebyshev polynomials using the recurrence
$$ \frac{T_n'(x)}{n} = 2 T_{n-1}(x) + \frac{T_{n-2}'(x)}{n-2} $$
which we can differentiate to evaluate higher derivatives.
Can you write Python code for it?

def chebeval(z, n=None):
    """Build matrices to evaluate the n-term Chebyshev expansion and its derivatives at point(s) z"""
    z = numpy.array(z, ndmin=1)
    if n is None:
        n = len(z)
    Tz = vander_chebyshev(z, n)
    dTz = numpy.zeros_like(Tz)
    dTz[:,1] = 1
    dTz[:,2] = 4*z
    ddTz = numpy.zeros_like(Tz)
    ddTz[:,2] = 4
    for n in range(3,n):
        dTz[:,n]  = n * (2*Tz[:,n-1] + dTz[:,n-2]/(n-2))
        ddTz[:,n] = n * (2*dTz[:,n-1] + ddTz[:,n-2]/(n-2))
    return [Tz, dTz, ddTz]

n = 44
x = numpy.linspace(-1, 1, n)
T = vander_chebyshev(x)
print('cond = {:e}'.format(numpy.linalg.cond(T)))
Tinv = numpy.linalg.inv(T)
L = numpy.zeros((n,n))
for i in range(n):
    L[i] = chebeval(x[i], n)[2].dot(Tinv)

u = numpy.cos(3*x)
pyplot.plot(x, L.dot(u), 'o')
xx = numpy.linspace(-1, 1, 100)
pyplot.plot(xx, -9*numpy.cos(3*xx));


I am working on the file "FDHighOrder.ipynb".
The first task is:
Given the coefficients $c = V^{-1} u$, we find
$$ \begin{align} p(0) &= c_0 \ p'(0) &= c_1 \ p''(0) &= c_2 \cdot 2! \ p^{(k)}(0) &= c_k \cdot k! . \end{align} $$
To compute the stencil coefficients $s_i^0$ for interpolation to $x=0$,
$$ p(0) = s_0^0 u_0 + s_1^0 u_1 + \dotsb = \sum_i s_i^0 u_i $$
we can write
$$ p(0) = e_0^T \underbrace{V^{-1} u}c = \underbrace{e_0^T V^{-1}}{(s^0)^T} u $$
where $e_0$ is the first column of the identity.  Evidently $s^0$ can also be expressed as
$$ s^0 = V^{-T} e_0 . $$
We can compute stencil coefficients for any order derivative $p^{(k)}(0) = (s^k)^T u$ by solving the linear system
$$ s^k = V^{-T} e_k \cdot k! . $$
Alternatively, invert the Vandermonde matrix $V$ and scale row $k$ of $V^{-1}$ by $k!$.
Can you write Python code for it?

def fdstencil(z, x):
    x = numpy.array(x)
    V = numpy.vander(x - z, increasing=True)
    scaling = numpy.array([numpy.math.factorial(i) for i in range(len(x))])
    return (numpy.linalg.inv(V).T * scaling).T

x = numpy.linspace(0,3,4)
S = fdstencil(0, x)
print(S)

hs = 2.**(-numpy.arange(6))
errors = numpy.zeros((3,len(hs)))
for i,h in enumerate(hs):
    z = 1 + .3*h
    S = fdstencil(z, 1+x*h)
    u = numpy.sin(1+x*h)
    errors[:,i] = S[:3].dot(u) - numpy.array([numpy.sin(z), numpy.cos(z), -numpy.sin(z)])

pyplot.loglog(hs, numpy.abs(errors[0]), 'o', label="$p(0)$")
pyplot.loglog(hs, numpy.abs(errors[1]), '<', label="$p'(0)$")
pyplot.loglog(hs, numpy.abs(errors[2]), 's', label="$p''(0)$")
for k in (1,2,3):
    pyplot.loglog(hs, hs**k, label='$h^{%d}$' % k)
pyplot.legend(loc='upper left');


I am working on the file "FDHighOrder.ipynb".
The first task is:
Notes on accuracy

When using three points, we fit a polynomial of degree 2.  The leading error term for interpolation $p(0)$ is thus $O(h^3)$.
Each derivative gives up one order of accuracy, therefore differencing to a general (non-centered or non-uniform grid) point is $O(h^2)$ for the first derivative and $O(h)$ for the second derivative.
Centered differences on uniform grids can provide cancelation, raising the order of accuracy by one.  So our standard 3-point centered second derivative is $O(h^2)$ as we have seen in the Taylor analysis and numerically.
The Vandermonde matrix is notoriously ill-conditioned when using many points.  For such cases, we recommend using a more numerically stable method from Fornberg.
Can you write Python code for it?


I am working on the file "FDHighOrder.ipynb".
The first task is:
We are suffering from two problems here.  The first is that the monomial basis is very ill-conditioned when using many terms.  This is true as continuous functions, not just when sampled onto a particular grid.
Can you write Python code for it?

x = numpy.linspace(-1, 1, 50)
V = numpy.vander(x, 15)
pyplot.plot(x, V)
numpy.linalg.cond(V)


I am working on the file "FDMS/TME2/TME2FDMS_Florian_Toque.ipynb".
The first task is:
Data generation
Can you write Python code for it?

ds = sklearn.datasets.make_classification(n_samples=2500,
                                          n_features=30,    # 30 features
                                          n_informative=8,  # only 8 informatives ones
                                          n_redundant=0,
                                          n_repeated=2,     # and 2 duplicate
                                          n_classes=2,
                                          n_clusters_per_class=1,
                                          weights=None,
                                          flip_y=0.03,
                                          class_sep=0.8,
                                          hypercube=True,
                                          shift=0.0,
                                          scale=1.0,
                                          shuffle=True,
                                          random_state=None)
X= ds[0]
y= ds[1]

# labels: [0,1] -> [-1,1]
for idx,i in enumerate(y):
    if (i==0):
        y[idx]=-1

print(X[0])
print(y[0])


I am working on the file "FDMS/TME2/TME2FDMS_Florian_Toque.ipynb".
The first task is:
L2 chooses less features than L1, and keeps a good score.

Evaluation using sklearn Lasso
Sklearn's Lasso works the same, although way faster, and a lambda 0 < λ < 1 is more practical
Can you write Python code for it?

#used to cross-val on lasso and elastic-net
def scorer(estimator, X, y):
    pred = estimator.predict(X)
    cpt=0.0
    for idx,i in enumerate(pred):
        if i<0:
            cpt += 1 if y[idx]==-1 else 0
        else:
            cpt += 1 if y[idx]==1 else 0
    return cpt/len(y)

lass = Lasso(alpha = 0.2)
lass.fit(X,y)

scores = cross_validation.cross_val_score(lass, X, y, cv=5,scoring=scorer)
print("Cross validation scores: %s, mean: %.02f"%(scores,np.mean(scores)))
print(lass.coef_)
print("Feature used: %d"%np.count_nonzero(lass.coef_))

eps=0.00001
la = []
cross_sc = []
used_features = []

for lamb in np.arange(0.05,1.05,0.05):
    theta = copy.deepcopy(X[0])
    gd = Lasso(alpha = lamb)
    nbIterations = 4000
    gd.fit(X,y)
    scoresSvm = cross_validation.cross_val_score(gd, X, y, cv=5,scoring=scorer)
    print("Lamda: %s | Cross val mean: %.03f | Features: %d"%(str(lamb).ljust(5),np.mean(scoresSvm),np.count_nonzero(gd.coef_)))
    #print("Lamda: %.02f | Cross val mean: %.02f | Features: %d"%(lamb,gd.score(X,y),gd.used_features))
    cross_sc.append(np.mean(scoresSvm))
    la.append(lamb)
    used_features.append(np.count_nonzero(gd.coef_))

fig, ax1 = plt.subplots()

ax2 = ax1.twinx()
ax1.plot(la, cross_sc, '#6DC433')
ax2.plot(la, used_features, '#5AC8ED')

ax1.set_xlabel('lambda')
ax1.set_ylabel('Cross val score', color='#6DC433')
ax2.set_ylabel('Nb features used', color='#5AC8ED')

ax1.yaxis.grid(False)
ax2.grid(False)
plt.show()


I am working on the file "FDMS/TME2/TME2FDMS_Florian_Toque.ipynb".
The first task is:
We can see that L2 is more sensitive to outliers because it chooses 7 usefull features against 8 for L1
Selecting lambda
Similar to L1
Can you write Python code for it?

eps=0.00001
la = []
cross_sc = []
used_features = []

for lamb in np.arange(0,4000,200):
    theta = copy.deepcopy(X[0])
    gd = GradientDescentL2(theta,lamb,eps)
    nbIterations = 5000
    gd.fit(X,y,nbIterations)
    scoresSvm = cross_validation.cross_val_score(gd, X, y, cv=5,scoring="accuracy")
    print("Lamda: %s | Cross val mean: %.03f | Features: %d"%(str(lamb).ljust(5),np.mean(scoresSvm),gd.used_features))
    cross_sc.append(np.mean(scoresSvm))
    la.append(lamb)
    used_features.append(gd.used_features)

fig, ax1 = plt.subplots()

ax2 = ax1.twinx()
ax1.plot(la, cross_sc, '#6DC433')
ax2.plot(la, used_features, '#5AC8ED')

ax1.set_xlabel('lambda')
ax1.set_ylabel('Cross val score', color='#6DC433')
ax2.set_ylabel('Nb features used', color='#5AC8ED')

ax1.yaxis.grid(False)
ax2.grid(False)
plt.show()


I am working on the file "FDMS/TME2/TME2FDMS_Florian_Toque.ipynb".
The first task is:
We can see that only 8 features are usefull (score 87%), with more features we can make overfitting and scores are only 2% better.

L2
The difference between the L1 and L2 regularization is that L1 work on the sum of the weights, and L2 work on the sum of the square of the weights, and is therefor more sensitive to outliers.
Advantage: good predictions with significants constraints
Can you write Python code for it?

class GradientDescentL2(base.BaseEstimator):
    def __init__(self,theta,lamb,eps):
        self.theta=theta
        self.eps=eps
        self.lamb=lamb
        self.used_features=len(theta)

    def fit(self,X,y,nbIt=1000,printevery=-1):
        l=len(X)
        xTrans = X.transpose()

        for i in xrange(0,nbIt):
            index = np.random.randint(l)
            loss = np.dot(X, self.theta) - y
            cost = np.sum(loss ** 2) * (1 / l) + (self.lamb*np.linalg.norm(self.theta))**2
            gradient = np.dot(xTrans,(np.dot(self.theta,xTrans)-y))

            if i%(nbIt/100)==0:
                thetaprime = self.theta - self.eps * (np.sign(theta)*self.lamb)
            else:
                thetaprime = self.theta - self.eps * gradient

            for k in xrange(0,len(theta)):
                self.theta[k] = 0 if thetaprime[k]*theta[k]<0 else thetaprime[k]

            if printevery!=-1 and i%printevery==0:
                    print("Iteration %s | Cost: %f | Score: %.03f" % (str(i).ljust(6), cost,self.score(X,y)))
                    ttt = self.nb_used_features()
                    print("%d features used"%(ttt))
                    self.used_features=ttt
            elif i%1000==0:
                ttt = self.nb_used_features()
                self.used_features=ttt

    def predict(self,x):
        ret=[]
        for i in x:
            ret.append(1 if np.dot(i,self.theta)>0 else -1)
        return ret

    def score(self,X,y):
        cpt=0.0
        allpred = self.predict(X)
        for idx,i in enumerate(allpred):
            cpt += 1 if i==y[idx] else 0
        return cpt/len(X)

    def nb_used_features(self):
        cpt=0
        for ii in self.theta:
            if ii==0:
                cpt+=1
        return len(self.theta)-cpt


I am working on the file "FDMS/TME2/TME2FDMS_Florian_Toque.ipynb".
The first task is:
We observe that, as expected, the more we take L1 into account the less features are used.
Can you write Python code for it?

eps=0.00001
la = []
cross_sc = []
used_features = []

for lamb in np.arange(0.05,1.05,0.05):
    theta = copy.deepcopy(X[0])
    gd = ElasticNet(alpha = 0.2, l1_ratio=lamb)
    nbIterations = 4000
    gd.fit(X,y)
    scoresSvm = cross_validation.cross_val_score(gd, X, y, cv=5,scoring=scorer)
    print("Lamda: %s | Cross val mean: %.03f | Features: %d"%(str(lamb).ljust(5),np.mean(scoresSvm),np.count_nonzero(gd.coef_)))
    #print("Lamda: %.02f | Cross val mean: %.02f | Features: %d"%(lamb,gd.score(X,y),gd.used_features))
    cross_sc.append(np.mean(scoresSvm))
    la.append(lamb)
    used_features.append(np.count_nonzero(gd.coef_))

fig, ax1 = plt.subplots()

ax2 = ax1.twinx()
ax1.plot(la, cross_sc, '#FF9900')
ax2.plot(la, used_features, '#9933FF')

ax1.set_xlabel('L1 L2 ratio')
ax1.set_ylabel('Cross val score', color='#FF9900')
ax2.set_ylabel('Nb features used', color='#9933FF')

ax1.yaxis.grid(False)
ax2.grid(False)
plt.show()


I am working on the file "FDMS/TME2/TME2_Paul_Willot.ipynb".
The first task is:
Comparaison of L1 and L2 using sklearn ElasticNet
Can you write Python code for it?

lass = ElasticNet(alpha = 0.2, l1_ratio=0)
lass.fit(X,y)
scores = cross_validation.cross_val_score(lass, X, y, cv=5,scoring=scorer)
print("Cross validation scores: %s, mean: %.02f"%(scores,np.mean(scores)))
print("Feature used: %d"%np.count_nonzero(lass.coef_))

lass = ElasticNet(alpha = 0.2, l1_ratio=0.5)
lass.fit(X,y)
scores = cross_validation.cross_val_score(lass, X, y, cv=5,scoring=scorer)
print("Cross validation scores: %s, mean: %.02f"%(scores,np.mean(scores)))
print("Feature used: %d"%np.count_nonzero(lass.coef_))

lass = ElasticNet(alpha = 0.2, l1_ratio=1)
lass.fit(X,y)
scores = cross_validation.cross_val_score(lass, X, y, cv=5,scoring=scorer)
print("Cross validation scores: %s, mean: %.02f"%(scores,np.mean(scores)))
print("Feature used: %d"%np.count_nonzero(lass.coef_))


I am working on the file "FDMS/TME2/TME2_Paul_Willot.ipynb".
The first task is:
Evaluation using sklearn Lasso
Sklearn's Lasso works the same, although way faster, and a lambda 0 < λ < 1 is more practical
Can you write Python code for it?


I am working on the file "FDMS/TME2/TME2_Paul_Willot.ipynb".
The first task is:
Selecting lambda
Similar to L1
Can you write Python code for it?


I am working on the file "FDMS/TME2/TME2_Paul_Willot.ipynb".
The first task is:
We observe that, as expected, the more we take L1 into account the less features are used.
Can you write Python code for it?


I am working on the file "FDMS/TME3/DataViz.ipynb".
The first task is:
13.765.202 lines in train.csv
8.022.757 lines in test.csv

Load the dataset
Can you write Python code for it?

%%time
filename = "data/reduced_train_100000.csv"
#filename = "data/reduced_test_100000.csv"
raw = pd.read_csv(filename)
raw = raw.set_index('Id')

raw['Expected'].describe()


I am working on the file "FDMS/TME3/DataViz.ipynb".
The first task is:
How much observations is there for each ID ?
Can you write Python code for it?

X,y=getXy(raw)

tmp = []
for i in X:
    tmp.append(len(i))
tmp = np.array(tmp)
sns.countplot(tmp,order=range(tmp.min(),tmp.max()+1))
plt.title("Number of ID per number of observations\n(On complete dataset)")
plt.plot()

print("Average gauge observation in mm: %0.2f"%y.mean())


I am working on the file "FDMS/TME3/DataViz.ipynb".
The first task is:
Now let's do the analysis on different subsets:
On fully filled dataset
Can you write Python code for it?

noAnyNan = raw.loc[raw[features_columns].dropna(how='any').index.unique()]

X,y=getXy(noAnyNan)

tmp = []
for i in X:
    tmp.append(len(i))
tmp = np.array(tmp)
sns.countplot(tmp,order=range(tmp.min(),tmp.max()+1))
plt.title("Number of ID per number of observations\n(On fully filled dataset)")
plt.plot()

print("Average gauge observation in mm: %0.2f"%y.mean())

pd.DataFrame(y).describe()

noFullNan = raw.loc[raw[features_columns].dropna(how='all').index.unique()]

index[:10]

X,y=getXy(raw)

tmp = []
for i in X:
    tmp.append(len(i))
tmp = np.array(tmp)
sns.countplot(tmp,order=range(tmp.min(),tmp.max()+1))
plt.title("Number of ID per number of observations\n(On partly filled dataset)")
plt.plot()

print("Average gauge observation in mm: %0.2f"%y.mean())

pd.DataFrame(y).describe()

fullNan = raw.drop(raw[features_columns].dropna(how='all').index)

X,y=getXy(fullNan)

tmp = []
for i in X:
    tmp.append(len(i))
tmp = np.array(tmp)
sns.countplot(tmp,order=range(tmp.min(),tmp.max()+1))
plt.title("Number of ID per number of observations\n(On fully empty dataset)")
plt.plot()

print("Average gauge observation in mm: %0.2f"%y.mean())

pd.DataFrame(y).describe()


I am working on the file "FDMS/TME3/DataViz.ipynb".
The first task is:
Quick analysis for the sparsity by column
Can you write Python code for it?

l = float(len(raw["minutes_past"]))
comp = [[1-raw[i].isnull().sum()/l , i] for i in raw.columns]
comp.sort(key=lambda x: x[0], reverse=True)

sns.barplot(zip(*comp)[0],zip(*comp)[1],palette=sns.cubehelix_palette(len(comp), start=.5, rot=-.75))
plt.title("Percentage of non NaN data")
plt.show()


I am working on the file "FDMS/TME3/DataViz.ipynb".
The first task is:
Strangely we notice that the less observations there is, the more it rains on average
However more of the expected rainfall fall below 0.5
What prediction should we make if there is no data?
Can you write Python code for it?

print("%d observations" %(len(raw)))
#print("%d fully filled, %d partly filled, %d fully empty"
#      %(len(noAnyNan),len(noFullNan),len(raw)-len(noFullNan)))
print("%0.1f%% fully filled, %0.1f%% partly filled, %0.1f%% fully empty"
      %(len(noAnyNan)/float(len(raw))*100,
        len(noFullNan)/float(len(raw))*100,
        (len(raw)-len(noFullNan))/float(len(raw))*100))

import numpy as np
from scipy.stats import kendalltau
import seaborn as sns
#sns.set(style="ticks")

rs = np.random.RandomState(11)
x = rs.gamma(2, size=1000)
y = -.5 * x + rs.normal(size=1000)

sns.jointplot(x, y, kind="hex", stat_func=kendalltau, color="#4CB391")


I am working on the file "FDMS/TME3/DataViz.ipynb".
The first task is:
We see that except for the fixed features minutes_past, radardist_km and Expected the dataset is mainly sparse.
Let's transform the dataset to conduct more analysis
We regroup the data by ID
Can you write Python code for it?

# We select all features except for the minutes past,
# because we ignore the time repartition of the sequence for now

features_columns = list([u'Ref', u'Ref_5x5_10th',
       u'Ref_5x5_50th', u'Ref_5x5_90th', u'RefComposite',
       u'RefComposite_5x5_10th', u'RefComposite_5x5_50th',
       u'RefComposite_5x5_90th', u'RhoHV', u'RhoHV_5x5_10th',
       u'RhoHV_5x5_50th', u'RhoHV_5x5_90th', u'Zdr', u'Zdr_5x5_10th',
       u'Zdr_5x5_50th', u'Zdr_5x5_90th', u'Kdp', u'Kdp_5x5_10th',
       u'Kdp_5x5_50th', u'Kdp_5x5_90th'])

def getXy(raw):
    selected_columns = list([ u'radardist_km', u'Ref', u'Ref_5x5_10th',
       u'Ref_5x5_50th', u'Ref_5x5_90th', u'RefComposite',
       u'RefComposite_5x5_10th', u'RefComposite_5x5_50th',
       u'RefComposite_5x5_90th', u'RhoHV', u'RhoHV_5x5_10th',
       u'RhoHV_5x5_50th', u'RhoHV_5x5_90th', u'Zdr', u'Zdr_5x5_10th',
       u'Zdr_5x5_50th', u'Zdr_5x5_90th', u'Kdp', u'Kdp_5x5_10th',
       u'Kdp_5x5_50th', u'Kdp_5x5_90th'])

    data = raw[selected_columns]

    docX, docY = [], []
    for i in data.index.unique():
        if isinstance(data.loc[i],pd.core.series.Series):
            m = [data.loc[i].as_matrix()]
            docX.append(m)
            docY.append(float(raw.loc[i]["Expected"]))
        else:
            m = data.loc[i].as_matrix()
            docX.append(m)
            docY.append(float(raw.loc[i][:1]["Expected"]))
    X , y = np.array(docX) , np.array(docY)
    return X,y

raw.index.unique()


I am working on the file "FDMS/TME3/DataViz.ipynb".
The first task is:
We see there is a lot of ID with 6 or 12 observations, that mean one every 5 or 10 minutes on average.
Can you write Python code for it?


I am working on the file "FDMS/TME3/Kaggle_rain-Copy1.ipynb".
The first task is:
Get rid of Nan value for now
Can you write Python code for it?

#train_clean = train[[not i for i in np.isnan(train["Ref_5x5_10th"])]]


I am working on the file "FDMS/TME3/Kaggle_rain-Copy1.ipynb".
The first task is:
Memento  (mauri)
Can you write Python code for it?

etreg = ExtraTreesRegressor(n_estimators=100, max_depth=None, min_samples_split=1, random_state=0)

"""
columns = train_clean.columns
columns = ["minutes_past","radardist_km","Ref","Ref_5x5_10th", "Ref_5x5_50th"]
columns = [u'Id', u'minutes_past', u'radardist_km', u'Ref', u'Ref_5x5_10th',
       u'Ref_5x5_50th', u'Ref_5x5_90th', u'RefComposite',
       u'RefComposite_5x5_10th', u'RefComposite_5x5_50th',
       u'RefComposite_5x5_90th', u'RhoHV', u'RhoHV_5x5_10th',
       u'RhoHV_5x5_50th', u'RhoHV_5x5_90th', u'Zdr', u'Zdr_5x5_10th',
       u'Zdr_5x5_50th', u'Zdr_5x5_90th', u'Kdp', u'Kdp_5x5_10th',
       u'Kdp_5x5_50th', u'Kdp_5x5_90th', u'Expected']
"""
columns = [u'minutes_past', u'radardist_km', u'Ref', u'Ref_5x5_10th',
       u'Ref_5x5_50th', u'Ref_5x5_90th', u'RefComposite',
       u'RefComposite_5x5_10th', u'RefComposite_5x5_50th',
       u'RefComposite_5x5_90th', u'RhoHV', u'RhoHV_5x5_10th',
       u'RhoHV_5x5_50th', u'RhoHV_5x5_90th', u'Zdr', u'Zdr_5x5_10th',
       u'Zdr_5x5_50th', u'Zdr_5x5_90th', u'Kdp', u'Kdp_5x5_10th',
       u'Kdp_5x5_50th', u'Kdp_5x5_90th']

labels = train_clean["Expected"].values
features = train_clean[list(columns)].values

imp = Imputer(missing_values='NaN', strategy='mean', axis=0)
imp.fit(features)
features_trans = imp.transform(features)

ftrain = features_trans[:3000]
ltrain = labels[:3000]
ftest = features_trans[3000:]
ltest = labels[3000:]

%%time
etreg.fit(ftrain,ltrain)

def scorer(estimator, X, y):
    return (estimator.predict(X[0])-y)**2

%%time
et_score = cross_val_score(etreg, features_trans, labels, cv=5)

print("Features: %s\nScore: %s\tMean: %.03f"%(columns, et_score,et_score.mean()))

r = random.randrange(len(ltrain))
print(r)
print(etreg.predict(ftrain[r]))
print(ltrain[r])

r = random.randrange(len(ltest))
print(r)
print(etreg.predict(ftest[r]))
print(ltest[r])

err = (etreg.predict(ftest)-ltest)**2

err.sum()/len(err)


I am working on the file "FDMS/TME3/Kaggle_rain-Copy1.ipynb".
The first task is:
Submit
Can you write Python code for it?

filename = "data/reduced_test_5000.csv"
test = pd.read_csv(filename)

columns = [u'minutes_past', u'radardist_km', u'Ref', u'Ref_5x5_10th',
       u'Ref_5x5_50th', u'Ref_5x5_90th', u'RefComposite',
       u'RefComposite_5x5_10th', u'RefComposite_5x5_50th',
       u'RefComposite_5x5_90th', u'RhoHV', u'RhoHV_5x5_10th',
       u'RhoHV_5x5_50th', u'RhoHV_5x5_90th', u'Zdr', u'Zdr_5x5_10th',
       u'Zdr_5x5_50th', u'Zdr_5x5_90th', u'Kdp', u'Kdp_5x5_10th',
       u'Kdp_5x5_50th', u'Kdp_5x5_90th']
features = test[list(columns)].values

imp = Imputer(missing_values='NaN', strategy='mean', axis=0)
imp.fit(features)
features_trans = imp.transform(features)

fall = test[test.columns].values

fall[20]

features_trans[0]

i = 1
pred = 0
while fall[i][0] == 1:
    #print(fall[i])
    pred+=etreg.predict(features_trans[i])[0]
    #print(etreg.predict(features_trans[i])[0])
    i+=1
print(i)

fall[-1][0]

%%time
res=[]
i=0
while i<len(fall) and i < 10000:
    pred = 0
    lenn = 0
    curr=fall[i][0]
    while i<len(fall) and fall[i][0] == curr:
        #print(fall[i])
        pred+=etreg.predict(features_trans[i])[0]
        #print(etreg.predict(features_trans[i])[0])
        i+=1
        lenn += 1
    res.append((curr,pred/lenn))
    #i+=1
    #print(i)

len(res)

res[:10]

def myfunc(hour):
    #rowid = hour['Id'].iloc[0]
    # sort hour by minutes_past
    hour = hour.sort('minutes_past', ascending=True)
    #est = (hour['Id'],random.random())
    est = random.random()
    return est

def marshall_palmer(ref, minutes_past):
    #print("Estimating rainfall from {0} observations".format(len(minutes_past)))
    # how long is each observation valid?
    valid_time = np.zeros_like(minutes_past)
    valid_time[0] = minutes_past.iloc[0]
    for n in xrange(1, len(minutes_past)):
        valid_time[n] = minutes_past.iloc[n] - minutes_past.iloc[n-1]
    valid_time[-1] = valid_time[-1] + 60 - np.sum(valid_time)
    valid_time = valid_time / 60.0

    # sum up rainrate * validtime
    sum = 0
    for dbz, hours in zip(ref,...


I am working on the file "FDMS/TME3/Kaggle_rain-Copy2.ipynb".
The first task is:
13.765.202 lines in train.csv
8.022.757 lines in test.csv

Reduced to

10.000
5.000
Can you write Python code for it?

%%time
filename = "data/reduced_train_100000.csv"
#filename = "data/reduced_test_5000.csv"
raw = pd.read_csv(filename)
raw = raw.set_index('Id')
#train = train.dropna()

l = float(len(raw["minutes_past"]))
comp = []
for i in raw.columns:
    #print(raw"%.03f, %s"%(1-train[i].isnull().sum()/l , i) )
    comp.append([1-raw[i].isnull().sum()/l , i])
comp.sort(key=lambda x: x[0], reverse=True)
comp

raw = raw.dropna()


I am working on the file "FDMS/TME3/Kaggle_rain-Copy2.ipynb".
The first task is:
Get rid of Nan value for now
Can you write Python code for it?


I am working on the file "FDMS/TME3/Kaggle_rain-Copy2.ipynb".
The first task is:
Memento  (mauri)
Can you write Python code for it?

etreg = ExtraTreesRegressor(n_estimators=100, max_depth=None, min_samples_split=1, random_state=0)

"""
columns = train_clean.columns
columns = ["minutes_past","radardist_km","Ref","Ref_5x5_10th", "Ref_5x5_50th"]
columns = [u'Id', u'minutes_past', u'radardist_km', u'Ref', u'Ref_5x5_10th',
       u'Ref_5x5_50th', u'Ref_5x5_90th', u'RefComposite',
       u'RefComposite_5x5_10th', u'RefComposite_5x5_50th',
       u'RefComposite_5x5_90th', u'RhoHV', u'RhoHV_5x5_10th',
       u'RhoHV_5x5_50th', u'RhoHV_5x5_90th', u'Zdr', u'Zdr_5x5_10th',
       u'Zdr_5x5_50th', u'Zdr_5x5_90th', u'Kdp', u'Kdp_5x5_10th',
       u'Kdp_5x5_50th', u'Kdp_5x5_90th', u'Expected']
"""
columns = [u'minutes_past', u'radardist_km', u'Ref', u'Ref_5x5_10th',
       u'Ref_5x5_50th', u'Ref_5x5_90th', u'RefComposite',
       u'RefComposite_5x5_10th', u'RefComposite_5x5_50th',
       u'RefComposite_5x5_90th', u'RhoHV', u'RhoHV_5x5_10th',
       u'RhoHV_5x5_50th', u'RhoHV_5x5_90th', u'Zdr', u'Zdr_5x5_10th',
       u'Zdr_5x5_50th', u'Zdr_5x5_90th', u'Kdp', u'Kdp_5x5_10th',
       u'Kdp_5x5_50th', u'Kdp_5x5_90th']

labels = train["Expected"].values
features = train[list(columns)].values

imp = Imputer(missing_values='NaN', strategy='mean', axis=0)
imp.fit(features)
features_trans = imp.transform(features)

len(features_trans)

split = 0.2
ps = int(len(features_trans) * split)
ftrain = features_trans[:ps]
ltrain = labels[:ps]
ftest = features_trans[ps:]
ltest = labels[ps:]

%%time
etreg.fit(ftrain,ltrain)

def scorer(estimator, X, y):
    return (estimator.predict(X[0])-y)**2

%%time
et_score = cross_val_score(etreg, features_trans, labels, cv=3)

print("Features: %s\nScore: %s\tMean: %.03f"%(columns, et_score,et_score.mean()))

r = random.randrange(len(ltrain))
print(r)
print(etreg.predict(ftrain[r]))
print(ltrain[r])

r = random.randrange(len(ltest))
print(r)
print(etreg.predict(ftest[r]))
print(ltest[r])

err = (etreg.predict(ftest)-ltest)**2

err.sum()/len(err)


I am working on the file "FDMS/TME3/Kaggle_rain-Copy2.ipynb".
The first task is:
Submit
Can you write Python code for it?


I am working on the file "FDMS/TME3/Kaggle_rain-Copy2.ipynb".
The first task is:
verifier val aberantes sur labels
Can you write Python code for it?


I am working on the file "FDMS/TME3/Kaggle_rain-Copy3.ipynb".
The first task is:

Can you write Python code for it?

#columns = [u'minutes_past', u'radardist_km', u'Ref', u'Ref_5x5_10th',
#       u'Ref_5x5_50th', u'Ref_5x5_90th', u'RefComposite',
#       u'RefComposite_5x5_10th', u'RefComposite_5x5_50th',
#       u'RefComposite_5x5_90th', u'RhoHV', u'RhoHV_5x5_10th',
#       u'RhoHV_5x5_50th', u'RhoHV_5x5_90th', u'Zdr', u'Zdr_5x5_10th',
#       u'Zdr_5x5_50th', u'Zdr_5x5_90th', u'Kdp', u'Kdp_5x5_10th',
#       u'Kdp_5x5_50th', u'Kdp_5x5_90th']
#columns = [u'radardist_km', u'Ref', u'Ref_5x5_10th']
columns = [ u'radardist_km', u'Ref', u'Ref_5x5_10th',
       u'Ref_5x5_50th', u'Ref_5x5_90th', u'RefComposite',
       u'RefComposite_5x5_10th', u'RefComposite_5x5_50th',
       u'RefComposite_5x5_90th', u'RhoHV', u'RhoHV_5x5_10th',
       u'RhoHV_5x5_50th', u'RhoHV_5x5_90th', u'Zdr', u'Zdr_5x5_10th',
       u'Zdr_5x5_50th', u'Zdr_5x5_90th', u'Kdp', u'Kdp_5x5_10th',
       u'Kdp_5x5_50th', u'Kdp_5x5_90th']
nb_features = len(columns)
data = raw[list(columns)]
data.head(5)

data.head(20)

%%time
#max_padding = 20
docX, docY = [], []
for i in raw.index.unique():
    if isinstance(raw.loc[i],pd.core.series.Series):
        m = [raw.loc[i].as_matrix()]
        #pad = np.pad(m, ((max_padding -len(m), 0),(0,0)), 'constant')  # pre-padding
        docX.append(m)
        docY.append(float(raw.loc[i]["Expected"]))
    else:
        m = data.loc[i].as_matrix()
        #pad = np.pad(m, ((max_padding -len(m), 0),(0,0)), 'constant')
        docX.append(m)
        docY.append(float(raw.loc[i][:1]["Expected"]))
    #docY.append(train.loc[i][:1]["Expected"].as_matrix)
X = np.array(docX)
y = np.array(docY)

np.shape(X)

XX = [np.array(t).mean(0) for t in X]

np.shape(XX)

XX[0]

global_means = np.nanmean(data,0)
#global_means = data.mean(0).values

a = [
        [1,2,np.nan],
        [3,4,np.nan],
        [2,np.nan,np.nan]
    ]

n = np.nanmean(a,0)

[np.isnan(i) for i in n]

n

np.count_nonzero(~np.isnan(X[0])) / float(X[0].size)

t = []
for i in X:
    t.append(np.count_nonzero(~np.isnan(i)) /...


def marshall_palmer(ref, minutes_past):
    #print("Estimating rainfall from {0} observations".format(len(minutes_past)))
    # how long is each observation valid?
    valid_time = np.zeros_like(minutes_past)
    valid_time[0] = minutes_past.iloc[0]
    for n in xrange(1, len(minutes_past)):
        valid_time[n] = minutes_past.iloc[n] - minutes_past.iloc[n-1]
    valid_time[-1] = valid_time[-1] + 60 - np.sum(valid_time)
    valid_time = valid_time / 60.0

    # sum up rainrate * validtime
    sum = 0
    for dbz, hours in zip(ref, valid_time):
        # See: https://en.wikipedia.org/wiki/DBZ_(meteorology)
        if np.isfinite(dbz):
            mmperhr = pow(pow(10, dbz/10)/200, 0.625)
            sum = sum + mmperhr * hours
    return sum

def simplesum(ref,hour):
    hour.sum()

# each unique Id is an hour of data at some gauge
def myfunc(hour):
    #rowid = hour['Id'].iloc[0]
    # sort hour by minutes_past
    hour = hour.sort('minutes_past', ascending=True)
    est = marshall_palmer(hour['Ref'], hour['minutes_past'])
    return est

estimates = raw.groupby(raw.index).apply(myfunc)
estimates.head(20)

err = (estimates-(np.hstack((y_train,y_test))))**2
err.sum()/len(err)


I am working on the file "FDMS/TME3/Kaggle_rain-Copy3.ipynb".
The first task is:
13.765.202 lines in train.csv
8.022.757 lines in test.csv

Reduced to

10.000
5.000
Can you write Python code for it?

%%time
#filename = "data/reduced_train_100000.csv"
#filename = "data/reduced_test_5000.csv"
filename = "data/reduced_train_100000.csv"
raw = pd.read_csv(filename)
raw = raw.set_index('Id')
#train = train.dropna()

l = float(len(raw["minutes_past"]))
comp = []
for i in raw.columns:
    #print(raw"%.03f, %s"%(1-train[i].isnull().sum()/l , i) )
    comp.append([1-raw[i].isnull().sum()/l , i])
comp.sort(key=lambda x: x[0], reverse=True)
comp

raw = raw.dropna()


I am working on the file "FDMS/TME3/Kaggle_rain-Copy3.ipynb".
The first task is:
Forums indicate that a higher than 1m rainfall is probably an error. Which is quite understandable. We filter that out
Can you write Python code for it?

raw = raw[raw['Expected'] < 1000]

raw['Expected'].describe()

split = 0.2
train = raw.tail(int(len(raw)*1-split))
test = raw.tail(int(len(raw)*split))


I am working on the file "FDMS/TME3/Kaggle_rain-Copy3.ipynb".
The first task is:
Get rid of Nan value for now
Can you write Python code for it?


I am working on the file "FDMS/TME3/Kaggle_rain-Copy3.ipynb".
The first task is:
Memento  (mauri)
Can you write Python code for it?


I am working on the file "FDMS/TME3/Kaggle_rain-Copy3.ipynb".
The first task is:
Submit
Can you write Python code for it?


I am working on the file "FDMS/TME3/Kaggle_rain-Copy4_from2.ipynb".
The first task is:

Can you write Python code for it?


I am working on the file "FDMS/TME3/Kaggle_rain-Copy4_from2.ipynb".
The first task is:
13.765.202 lines in train.csv
8.022.757 lines in test.csv

Reduced to

10.000
5.000
Can you write Python code for it?

%%time
filename = "data/reduced_train_100000.csv"
#filename = "data/reduced_test_100000.csv"
raw = pd.read_csv(filename)
raw = raw.set_index('Id')
#train = train.dropna()

l = float(len(raw["minutes_past"]))
comp = []
for i in raw.columns:
    #print(raw"%.03f, %s"%(1-train[i].isnull().sum()/l , i) )
    comp.append([1-raw[i].isnull().sum()/l , i])
comp.sort(key=lambda x: x[0], reverse=True)
comp

raw = raw.dropna()

raw.columns

i = raw[list([u'Ref', u'Ref_5x5_10th',
       u'Ref_5x5_50th', u'Ref_5x5_90th', u'RefComposite',
       u'RefComposite_5x5_10th', u'RefComposite_5x5_50th',
       u'RefComposite_5x5_90th', u'RhoHV', u'RhoHV_5x5_10th',
       u'RhoHV_5x5_50th', u'RhoHV_5x5_90th', u'Zdr', u'Zdr_5x5_10th',
       u'Zdr_5x5_50th', u'Zdr_5x5_90th', u'Kdp', u'Kdp_5x5_10th',
       u'Kdp_5x5_50th', u'Kdp_5x5_90th'])].dropna(how='all').index

fullna.difference(i)

idWithNoNan = np.unique(i)

idWithNoNan[:10]

AllId = np.unique(raw.index)

AllId[:10]

tr = [burg not in idWithNoNan for burg in AllId]

tmp = []
for idx,i in enumerate(tr):
    if i:
        tmp.append(idx+1)

tmp[-10:]

tttt = AllId*tr

fullna = raw.drop(raw[list([u'Ref', u'Ref_5x5_10th',
       u'Ref_5x5_50th', u'Ref_5x5_90th', u'RefComposite',
       u'RefComposite_5x5_10th', u'RefComposite_5x5_50th',
       u'RefComposite_5x5_90th', u'RhoHV', u'RhoHV_5x5_10th',
       u'RhoHV_5x5_50th', u'RhoHV_5x5_90th', u'Zdr', u'Zdr_5x5_10th',
       u'Zdr_5x5_50th', u'Zdr_5x5_90th', u'Kdp', u'Kdp_5x5_10th',
       u'Kdp_5x5_50th', u'Kdp_5x5_90th'])].dropna(how='all').index).index

fullna.head(100)


I am working on the file "FDMS/TME3/Kaggle_rain-Copy4_from2.ipynb".
The first task is:
Forums indicate that a higher than 1m rainfall is probably an error. Which is quite understandable. We filter that out
Can you write Python code for it?


I am working on the file "FDMS/TME3/Kaggle_rain-Copy4_from2.ipynb".
The first task is:
Get rid of Nan value for now
Can you write Python code for it?


I am working on the file "FDMS/TME3/Kaggle_rain-Copy4_from2.ipynb".
The first task is:
Memento  (mauri)
Can you write Python code for it?


I am working on the file "FDMS/TME3/Kaggle_rain-Copy4_from2.ipynb".
The first task is:
RNN
Can you write Python code for it?

import pandas as pd
from random import random

flow = (list(range(1,10,1)) + list(range(10,1,-1)))*1000
pdata = pd.DataFrame({"a":flow, "b":flow})
pdata.b = pdata.b.shift(9)
data = pdata.iloc[10:] * random()  # some noise

#columns = [u'minutes_past', u'radardist_km', u'Ref', u'Ref_5x5_10th',
#       u'Ref_5x5_50th', u'Ref_5x5_90th', u'RefComposite',
#       u'RefComposite_5x5_10th', u'RefComposite_5x5_50th',
#       u'RefComposite_5x5_90th', u'RhoHV', u'RhoHV_5x5_10th',
#       u'RhoHV_5x5_50th', u'RhoHV_5x5_90th', u'Zdr', u'Zdr_5x5_10th',
#       u'Zdr_5x5_50th', u'Zdr_5x5_90th', u'Kdp', u'Kdp_5x5_10th',
#       u'Kdp_5x5_50th', u'Kdp_5x5_90th']
columns = [u'radardist_km', u'Ref', u'Ref_5x5_10th']
nb_features = len(columns)
data = train[list(columns)]
data.head(10)

data.iloc[0].as_matrix()

train.head(5)

train.loc[11]

train.loc[11][:1]["Expected"].as_matrix

#train.index.unique()

def _load_data(data, n_prev = 100):
    """
    data should be pd.DataFrame()
    """

    docX, docY = [], []
    for i in range(len(data)-n_prev):
        docX.append(data.iloc[i:i+n_prev].as_matrix())
        docY.append(data.iloc[i+n_prev].as_matrix())
    alsX = np.array(docX)
    alsY = np.array(docY)

    return alsX, alsY

def train_test_split(df, test_size=0.1):
    ntrn = round(len(df) * (1 - test_size))

    X_train, y_train = _load_data(df.iloc[0:ntrn])
    X_test, y_test = _load_data(df.iloc[ntrn:])

    return (X_train, y_train), (X_test, y_test)

(X_train, y_train), (X_test, y_test) = train_test_split(data)

np.shape(X_train)

t = np.array([2,1])
t.shape = (1,2)
t.tolist()[0]

np.shape(t)

X_train[:2,:2]

train.index.unique()

max_padding = 20

%%time

docX, docY = [], []
for i in train.index.unique():
    if isinstance(train.loc[i],pd.core.series.Series):
        m = [data.loc[i].as_matrix()]
        pad = np.pad(m, ((max_padding -len(m), 0),(0,0)), 'constant')  # pre-padding
        docX.append(pad)
        docY.append(float(train.loc[i]["Expected"]))
    else:
        m =...


I am working on the file "FDMS/TME3/Kaggle_rain-Copy4_from2.ipynb".
The first task is:
Submit
Can you write Python code for it?


I am working on the file "FDMS/TME3/Kaggle_rain-Copy4_from2.ipynb".
The first task is:
t = []
for i in XX:
    t.append(np.count_nonzero(~np.isnan(i)) / float(i.size))
pd.DataFrame(np.array(t)).describe()
Can you write Python code for it?

XX = []
for i in X:
    nm = np.nanmean(i,0)
    for idx,j in enumerate(nm):
        if np.isnan(j):
            nm[idx]=global_means[idx]
    XX.append(np.array(nm))

XX = [np.array(t).mean(0) for t in X]

split = 0.2
ps = int(len(XX) * (1-split))
X_train = XX[:ps]
y_train = y[:ps]
X_test = XX[ps:]
y_test = y[ps:]

etreg = ExtraTreesRegressor(n_estimators=100, max_depth=None, min_samples_split=1, random_state=0)

%%time
etreg.fit(X_train,y_train)

%%time
et_score = cross_val_score(etreg, XX, y, cv=5)

print("Features: %s\nScore: %s\tMean: %.03f"%(columns, et_score,et_score.mean()))

err = (etreg.predict(X_test)-y_test)**2
err.sum()/len(err)


I am working on the file "FDMS/TME3/Kaggle_rain-Copy4_from2.ipynb".
The first task is:
verifier val aberantes sur labels
Can you write Python code for it?


I am working on the file "FDMS/TME3/Kaggle_rain.ipynb".
The first task is:
13.765.202 lines in train.csv
8.022.757 lines in test.csv

Reduced to

10.000
5.000
Can you write Python code for it?

filename = "data/reduced_train_1000000.csv"
train = pd.read_csv(filename)
train = train.set_index('Id')
train = train.dropna()

train.head()

train["Expected"].describe()


I am working on the file "FDMS/TME3/Kaggle_rain.ipynb".
The first task is:
Forums indicate that a higher than 1m rainfall is probably an error. Which is quite understandable. We filter that out
Can you write Python code for it?

train = train[train['Expected'] < 1000]

train_clean.head()

train_clean.describe()

train_clean['Expected'].describe()


I am working on the file "FDMS/TME3/Kaggle_rain.ipynb".
The first task is:
Get rid of Nan value for now
Can you write Python code for it?

train_clean = train[[not i for i in np.isnan(train["Ref_5x5_10th"])]]


I am working on the file "FDMS/TME3/Kaggle_rain.ipynb".
The first task is:
Memento  (mauri)
Can you write Python code for it?

RandomForestRegressor()

etreg = ExtraTreesRegressor(n_estimators=100, max_depth=None, min_samples_split=1, random_state=0)

"""
columns = train_clean.columns
columns = ["minutes_past","radardist_km","Ref","Ref_5x5_10th", "Ref_5x5_50th"]
columns = [u'Id', u'minutes_past', u'radardist_km', u'Ref', u'Ref_5x5_10th',
       u'Ref_5x5_50th', u'Ref_5x5_90th', u'RefComposite',
       u'RefComposite_5x5_10th', u'RefComposite_5x5_50th',
       u'RefComposite_5x5_90th', u'RhoHV', u'RhoHV_5x5_10th',
       u'RhoHV_5x5_50th', u'RhoHV_5x5_90th', u'Zdr', u'Zdr_5x5_10th',
       u'Zdr_5x5_50th', u'Zdr_5x5_90th', u'Kdp', u'Kdp_5x5_10th',
       u'Kdp_5x5_50th', u'Kdp_5x5_90th', u'Expected']
"""
columns = [u'minutes_past', u'radardist_km', u'Ref', u'Ref_5x5_10th',
       u'Ref_5x5_50th', u'Ref_5x5_90th', u'RefComposite',
       u'RefComposite_5x5_10th', u'RefComposite_5x5_50th',
       u'RefComposite_5x5_90th', u'RhoHV', u'RhoHV_5x5_10th',
       u'RhoHV_5x5_50th', u'RhoHV_5x5_90th', u'Zdr', u'Zdr_5x5_10th',
       u'Zdr_5x5_50th', u'Zdr_5x5_90th', u'Kdp', u'Kdp_5x5_10th',
       u'Kdp_5x5_50th', u'Kdp_5x5_90th']

labels = train_clean["Expected"].values
features = train_clean[list(columns)].values

imp = Imputer(missing_values='NaN', strategy='mean', axis=0)
imp.fit(features)
features_trans = imp.transform(features)

ftrain = features_trans[:3000]
ltrain = labels[:3000]
ftest = features_trans[3000:]
ltest = labels[3000:]

%%time
etreg.fit(ftrain,ltrain)

def scorer(estimator, X, y):
    return (estimator.predict(X[0])-y)**2

%%time
et_score = cross_val_score(etreg, features_trans, labels, cv=5)

print("Features: %s\nScore: %s\tMean: %.03f"%(columns, et_score,et_score.mean()))

r = random.randrange(len(ltrain))
print(r)
print(etreg.predict(ftrain[r]))
print(ltrain[r])

r = random.randrange(len(ltest))
print(r)
print(etreg.predict(ftest[r]))
print(ltest[r])

err = (etreg.predict(ftest)-ltest)**2

err.sum()/len(err)


I am working on the file "FDMS/TME3/Kaggle_rain.ipynb".
The first task is:
Submit
Can you write Python code for it?


I am working on the file "FDMS/TME3/Model_V2.ipynb".
The first task is:

Can you write Python code for it?

%%time
#filename = "data/reduced_test_5000.csv"
filename = "data/test.csv"
test = pd.read_csv(filename)
test = test.set_index('Id')

features_columns = list([u'Ref', u'Ref_5x5_10th',
       u'Ref_5x5_50th', u'Ref_5x5_90th', u'RefComposite',
       u'RefComposite_5x5_10th', u'RefComposite_5x5_50th',
       u'RefComposite_5x5_90th', u'RhoHV', u'RhoHV_5x5_10th',
       u'RhoHV_5x5_50th', u'RhoHV_5x5_90th', u'Zdr', u'Zdr_5x5_10th',
       u'Zdr_5x5_50th', u'Zdr_5x5_90th', u'Kdp', u'Kdp_5x5_10th',
       u'Kdp_5x5_50th', u'Kdp_5x5_90th'])

def getX(raw):
    selected_columns = list([ u'radardist_km', u'Ref', u'Ref_5x5_10th',
       u'Ref_5x5_50th', u'Ref_5x5_90th', u'RefComposite',
       u'RefComposite_5x5_10th', u'RefComposite_5x5_50th',
       u'RefComposite_5x5_90th', u'RhoHV', u'RhoHV_5x5_10th',
       u'RhoHV_5x5_50th', u'RhoHV_5x5_90th', u'Zdr', u'Zdr_5x5_10th',
       u'Zdr_5x5_50th', u'Zdr_5x5_90th', u'Kdp', u'Kdp_5x5_10th',
       u'Kdp_5x5_50th', u'Kdp_5x5_90th'])

    data = raw[selected_columns]

    docX= []
    for i in data.index.unique():
        if isinstance(data.loc[i],pd.core.series.Series):
            m = [data.loc[i].as_matrix()]
            docX.append(m)
        else:
            m = data.loc[i].as_matrix()
            docX.append(m)
    X = np.array(docX)
    return X

X=getX(test)

tmp = []
for i in X:
    tmp.append(len(i))
tmp = np.array(tmp)
sns.countplot(tmp,order=range(tmp.min(),tmp.max()+1))
plt.title("Number of ID per number of observations\n(On test dataset)")
plt.plot()

testFull = test.dropna()

%%time
X=getX(testFull)  # 1min
XX = [np.array(t).mean(0) for t in X]  # 10s

pd.DataFrame(etreg.predict(XX)).describe()

predFull = zip(testFull.index.unique(),etreg.predict(XX))

testNan = test.drop(test[features_columns].dropna(how='all').index)

tmp = np.empty(len(testNan))
tmp.fill(0.445000)   # 50th percentile of full Nan dataset
predNan = zip(testNan.index.unique(),tmp)

testLeft = test.drop(testNan.index.unique()).drop(testFull.index.unique())

tmp =...


from sklearn.neighbors import KNeighborsRegressor
from sklearn import grid_search

knn = KNeighborsRegressor(n_neighbors=6,weights='distance')

parameters = {'weights':('distance','uniform'),'algorithm':('auto', 'ball_tree', 'kd_tree', 'brute')}
grid_knn = grid_search.GridSearchCV(knn, parameters,n_jobs=2)

%%time
grid_knn.fit(X_train,y_train)

print(grid_knn.grid_scores_)
print("Best: ",grid_knn.best_params_)

knn = grid_knn.best_estimator_

knn.fit(X_train,y_train)

err = (knn.predict(X_test)-y_test)**2
err.sum()/len(err)


from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Activation
from keras.optimizers import SGD

in_dim = len(XX[0])
out_dim = 1

model = Sequential()
# Dense(64) is a fully-connected layer with 64 hidden units.
# in the first layer, you must specify the expected input data shape:
# here, 20-dimensional vectors.
model.add(Dense(128, input_shape=(in_dim,)))
model.add(Activation('tanh'))
model.add(Dropout(0.5))
model.add(Dense(1, init='uniform'))
model.add(Activation('linear'))

#sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
#model.compile(loss='mean_squared_error', optimizer=sgd)

rms = RMSprop()
model.compile(loss='mean_squared_error', optimizer=rms)

#model.fit(X_train, y_train, nb_epoch=20, batch_size=16)
#score = model.evaluate(X_test, y_test, batch_size=16)

prep = []
for i in y_train:
    prep.append(min(i,20))

prep=np.array(prep)
mi,ma = prep.min(),prep.max()
fy = (prep-mi) / (ma-mi)
#my = fy.max()
#fy = fy/fy.max()

model.fit(np.array(X_train), fy, batch_size=10, nb_epoch=10, validation_split=0.1)

pred = model.predict(np.array(X_test))*ma+mi

err = (pred-y_test)**2
err.sum()/len(err)

r = random.randrange(len(X_train))
print("(Train) Prediction %0.4f, True: %0.4f"%(model.predict(np.array([X_train[r]]))[0][0]*ma+mi,y_train[r]))

r = random.randrange(len(X_test))
print("(Test)  Prediction %0.4f, True: %0.4f"%(model.predict(np.array([X_test[r]]))[0][0]*ma+mi,y_test[r]))


from sklearn import svm

svr = svm.SVR(C=100000)

%%time
srv = svr.fit(X_train,y_train)

err = (svr.predict(X_test)-y_test)**2
err.sum()/len(err)

err = (svr.predict(X_test)-y_test)**2
err.sum()/len(err)

%%time
svr_score = cross_val_score(svr, XX, y, cv=5)
print("Score: %s\tMean: %.03f"%(svr_score,svr_score.mean()))


etreg = ExtraTreesRegressor(n_estimators=200, max_depth=None, min_samples_split=1, random_state=0)

parameters = {'n_estimators':range(100,200,20)}
grid_rf = grid_search.GridSearchCV(etreg, parameters,n_jobs=2)

%%time
grid_rf.fit(X_train,y_train)

print(grid_rf.grid_scores_)
print("Best: ",grid_rf.best_params_)

grid_rf.best_params_

es = etreg
#es = grid_rf.best_estimator_

%%time
es = es.fit(X_train,y_train)

err = (es.predict(X_test)-y_test)**2
err.sum()/len(err)


I am working on the file "FDMS/TME3/Model_V2.ipynb".
The first task is:
13.765.202 lines in train.csv
8.022.757 lines in test.csv

Few words about the dataset
Predictions is made in the USA corn growing states (mainly Iowa, Illinois, Indiana) during the season with the highest rainfall (as illustrated by Iowa for the april to august months)
The Kaggle page indicate that the dataset have been shuffled, so working on a subset seems acceptable
The test set is not a extracted from the same data as the training set however, which make the evaluation trickier
Load the dataset
Can you write Python code for it?

%%time
filename = "data/reduced_train_100000.csv"
#filename = "data/reduced_train_100000.csv"
raw = pd.read_csv(filename)
raw = raw.set_index('Id')

raw['Expected'].describe()


I am working on the file "FDMS/TME3/Model_V2.ipynb".
The first task is:
Predicitons
As a first try, we make predictions on the complete data, and return the 50th percentile and uncomplete and fully empty data
Can you write Python code for it?

%%time
X,y=getXy(noAnyNan)

%%time
#XX = [np.array(t).mean(0) for t in X]
XX = [np.append(np.nanmean(np.array(t),0),(np.array(t)[1:] - np.array(t)[:-1]).sum(0) ) for t in X]

t = X[0]

t

np.percentile(t,50,axis=0)

XX=[]
for t in X:
    #print(idx)

    tmp = np.append(np.nanmean(np.array(t),0),(np.array(t)[1:] - np.array(t)[:-1]).sum(0) )
    tmp = np.append(tmp,np.percentile(t,10,axis=0))
    tmp = np.append(tmp,np.percentile(t,50,axis=0))
    tmp = np.append(tmp,np.percentile(t,90,axis=0))
    test = t

    try:
        taa=test[:,0]
    except TypeError:
        taa=[test[0][0]]
    valid_time = np.zeros_like(taa)
    valid_time[0] = taa[0]
    for n in xrange(1,len(taa)):
        valid_time[n] = taa[n] - taa[n-1]
    valid_time[-1] = valid_time[-1] + 60 - np.sum(valid_time)
    valid_time = valid_time / 60.0

    sum=0
    try:
        column_ref=test[:,2]
    except TypeError:
        column_ref=[test[0][2]]
    for dbz, hours in zip(column_ref, valid_time):
        # See: https://en.wikipedia.org/wiki/DBZ_(meteorology)
        if np.isfinite(dbz):
            mmperhr = pow(pow(10, dbz/10)/200, 0.625)
            sum = sum + mmperhr * hours

    #XX.append(np.append(np.array(sum),tmp))
    #XX.append(np.array([sum]))
    XX.append(tmp)

XX[0]

XX[2]

def splitTrainTest(X, y, split=0.2):
    tmp1, tmp2 = [], []
    ps = int(len(X) * (1-split))
    index_shuf = range(len(X))
    random.shuffle(index_shuf)
    for i in index_shuf:
        tmp1.append(X[i])
        tmp2.append(y[i])
    return tmp1[:ps], tmp2[:ps], tmp1[ps:], tmp2[ps:]

X_train,y_train, X_test, y_test = splitTrainTest(XX,y)


I am working on the file "FDMS/TME3/Model_V2.ipynb".
The first task is:
We regroup the data by ID
Can you write Python code for it?

# We select all features except for the minutes past,
# because we ignore the time repartition of the sequence for now

features_columns = list([u'Ref', u'Ref_5x5_10th',
       u'Ref_5x5_50th', u'Ref_5x5_90th', u'RefComposite',
       u'RefComposite_5x5_10th', u'RefComposite_5x5_50th',
       u'RefComposite_5x5_90th', u'RhoHV', u'RhoHV_5x5_10th',
       u'RhoHV_5x5_50th', u'RhoHV_5x5_90th', u'Zdr', u'Zdr_5x5_10th',
       u'Zdr_5x5_50th', u'Zdr_5x5_90th', u'Kdp', u'Kdp_5x5_10th',
       u'Kdp_5x5_50th', u'Kdp_5x5_90th'])

def getXy(raw):
    selected_columns = list([ u'minutes_past',u'radardist_km', u'Ref', u'Ref_5x5_10th',
       u'Ref_5x5_50th', u'Ref_5x5_90th', u'RefComposite',
       u'RefComposite_5x5_10th', u'RefComposite_5x5_50th',
       u'RefComposite_5x5_90th', u'RhoHV', u'RhoHV_5x5_10th',
       u'RhoHV_5x5_50th', u'RhoHV_5x5_90th', u'Zdr', u'Zdr_5x5_10th',
       u'Zdr_5x5_50th', u'Zdr_5x5_90th', u'Kdp', u'Kdp_5x5_10th',
       u'Kdp_5x5_50th', u'Kdp_5x5_90th'])

    data = raw[selected_columns]

    docX, docY = [], []
    for i in data.index.unique():
        if isinstance(data.loc[i],pd.core.series.Series):
            m = [data.loc[i].as_matrix()]
            docX.append(m)
            docY.append(float(raw.loc[i]["Expected"]))
        else:
            m = data.loc[i].as_matrix()
            docX.append(m)
            docY.append(float(raw.loc[i][:1]["Expected"]))
    X , y = np.array(docX) , np.array(docY)
    return X,y


I am working on the file "FDMS/TME3/Model_V3.ipynb".
The first task is:

Can you write Python code for it?

etreg = ExtraTreesRegressor(n_estimators=200, max_depth=None, min_samples_split=1, random_state=0)

parameters = {'n_estimators':range(100,200,20)}
grid_rf = grid_search.GridSearchCV(etreg, parameters,n_jobs=2,scoring=manualScorer)

%%time
grid_rf.fit(X_train,y_train)

print(grid_rf.grid_scores_)
print("Best: ",grid_rf.best_params_)

grid_rf.best_params_

es = etreg
#es = grid_rf.best_estimator_

%%time
es = es.fit(X_train,y_train)

err = (es.predict(X_train)-y_train)**2
err.sum()/len(err)

err = (es.predict(X_test)-y_test)**2
err.sum()/len(err)


def manualScorer(estimator, X, y):
    err = (estimator.predict(X_test)-y_test)**2
    return -err.sum()/len(err)


def marshall_palmer(ref, minutes_past):
    #print("Estimating rainfall from {0} observations".format(len(minutes_past)))
    # how long is each observation valid?
    valid_time = np.zeros_like(minutes_past)
    valid_time[0] = minutes_past.iloc[0]
    for n in xrange(1, len(minutes_past)):
        valid_time[n] = minutes_past.iloc[n] - minutes_past.iloc[n-1]
    valid_time[-1] = valid_time[-1] + 60 - np.sum(valid_time)
    valid_time = valid_time / 60.0

    # sum up rainrate * validtime
    sum = 0
    for dbz, hours in zip(ref, valid_time):
        # See: https://en.wikipedia.org/wiki/DBZ_(meteorology)
        if np.isfinite(dbz):
            mmperhr = pow(pow(10, dbz/10)/200, 0.625)
            sum = sum + mmperhr * hours
    return sum

def simplesum(ref,hour):
    hour.sum()

# each unique Id is an hour of data at some gauge
def myfunc(hour):
    #rowid = hour['Id'].iloc[0]
    # sort hour by minutes_past
    hour = hour.sort('minutes_past', ascending=True)
    est = marshall_palmer(hour['Ref'], hour['minutes_past'])
    return est

info = raw.groupby(raw.index)

estimates = raw.groupby(raw.index).apply(myfunc)
estimates.head(20)

%%time
etreg.fit(X_train,y_train)

%%time
et_score = cross_val_score(etreg, XX, y, cv=5)
print("Score: %s\tMean: %.03f"%(et_score,et_score.mean()))

%%time
et_score = cross_val_score(etreg, XX, y, cv=5)
print("Score: %s\tMean: %.03f"%(et_score,et_score.mean()))

err = (etreg.predict(X_test)-y_test)**2
err.sum()/len(err)

err = (etreg.predict(X_test)-y_test)**2
err.sum()/len(err)

r = random.randrange(len(X_train))
print(r)
print(etreg.predict(X_train[r]))
print(y_train[r])

r = random.randrange(len(X_test))
print(r)
print(etreg.predict(X_test[r]))
print(y_test[r])


gbr = GradientBoostingRegressor(loss='ls', learning_rate=0.1, n_estimators=900,
                                subsample=1.0, min_samples_split=2, min_samples_leaf=1,
                                min_weight_fraction_leaf=0.0, max_depth=4, init=None,
                                random_state=None, max_features=None, alpha=0.5,
                                verbose=0, max_leaf_nodes=None, warm_start=False)

%%time
gbr = gbr.fit(X_train,y_train)
os.system('say "終わりだ"') #its over!

#parameters = {'max_depth':range(2,5,1),'alpha':[0.5,0.6,0.7,0.8,0.9]}
#parameters = {'subsample':[0.2,0.4,0.5,0.6,0.8,1]}
#parameters = {'subsample':[0.2,0.5,0.6,0.8,1],'n_estimators':[800,1000,1200]}
#parameters = {'max_depth':range(2,4,1)}
#parameters = {'n_estimators':[600,800,900]}
#parameters = {'loss':['ls', 'lad', 'huber', 'quantile'],'alpha':[0.3,0.5,0.8,0.9]}

grid_gbr = grid_search.GridSearchCV(gbr, parameters,n_jobs=2,scoring=manualScorer)

%%time
grid_gbr = grid_gbr.fit(X_train,y_train)

print(grid_gbr.grid_scores_)
print("Best: ",grid_gbr.best_params_)

err = (gbr.predict(X_train)-y_train)**2
print(err.sum()/len(err))
err = (gbr.predict(X_test)-y_test)**2
print(err.sum()/len(err))

err = (gbr.predict(X_train)-y_train)**2
print(err.sum()/len(err))
err = (gbr.predict(X_test)-y_test)**2
print(err.sum()/len(err))


I am working on the file "FDMS/TME3/Model_V3.ipynb".
The first task is:
On fully filled dataset
Can you write Python code for it?

#noAnyNan = raw.loc[raw[features_columns].dropna(how='any').index.unique()]
noAnyNan = raw.dropna()

noFullNan = raw.loc[raw[features_columns].dropna(how='all').index.unique()]

fullNan = raw.drop(raw[features_columns].dropna(how='all').index)


I am working on the file "FDMS/TME3/Model_V3.ipynb".
The first task is:
Per wikipedia, a value of more than 421 mm/h is considered "Extreme/large hail"
If we encounter the value 327.40 meter per hour, we should probably start building Noah's ark
Therefor, it seems reasonable to drop values too large, considered as outliers
Can you write Python code for it?

# Considering that the gauge may concentrate the rainfall, we set the cap to 1000
# Comment this line to analyse the complete dataset
l = len(raw)
raw = raw[raw['Expected'] < 300]  #1000
print("Dropped %d (%0.2f%%)"%(l-len(raw),(l-len(raw))/float(l)*100))

raw.head(5)

raw.describe()


I am working on the file "FDMS/TME3/Model_V3.ipynb".
The first task is:
Predicitons
As a first try, we make predictions on the complete data, and return the 50th percentile and uncomplete and fully empty data
Can you write Python code for it?

%%time
X,y=getXy(noAnyNan)

%%time
#XX = [np.array(t).mean(0) for t in X]
XX = [np.append(np.nanmean(np.array(t),0),(np.array(t)[1:] - np.array(t)[:-1]).sum(0) ) for t in X]

%%time
XX=[]
for t in X:
    #print(idx)

    tmp = np.append(np.nanmean(np.array(t),0),(np.array(t)[1:] - np.array(t)[:-1]).sum(0) )
    tmp = np.append(tmp,np.percentile(t,10,axis=0))
    tmp = np.append(tmp,np.percentile(t,50,axis=0))
    tmp = np.append(tmp,np.percentile(t,90,axis=0))

    test = t

    try:
        taa=test[:,0]
    except TypeError:
        taa=[test[0][0]]
    valid_time = np.zeros_like(taa)
    valid_time[0] = taa[0]
    for n in xrange(1,len(taa)):
        valid_time[n] = taa[n] - taa[n-1]
    valid_time[-1] = valid_time[-1] + 60 - np.sum(valid_time)
    valid_time = valid_time / 60.0

    sum=0
    try:
        column_ref=test[:,2]
    except TypeError:
        column_ref=[test[0][2]]
    for dbz, hours in zip(column_ref, valid_time):
        # See: https://en.wikipedia.org/wiki/DBZ_(meteorology)
        if np.isfinite(dbz):
            mmperhr = pow(pow(10, dbz/10)/200, 0.625)
            sum = sum + mmperhr * hours

    XX.append(np.append(np.array(sum),tmp))
    #XX.append(np.array([sum]))
    #XX.append(tmp)

XX[2]

def splitTrainTest(X, y, split=0.2):
    tmp1, tmp2 = [], []
    ps = int(len(X) * (1-split))
    index_shuf = range(len(X))
    random.shuffle(index_shuf)
    for i in index_shuf:
        tmp1.append(X[i])
        tmp2.append(y[i])
    return tmp1[:ps], tmp2[:ps], tmp1[ps:], tmp2[ps:]

X_train,y_train, X_test, y_test = splitTrainTest(XX,y)


I am working on the file "FDMS/TME3/Model_V4.ipynb".
The first task is:

Can you write Python code for it?

t = []
for i in XX:
    t.append(np.count_nonzero(~np.isnan(i)) / float(i.size))
pd.DataFrame(np.array(t)).describe()


etreg = ExtraTreesRegressor(n_estimators=200, max_depth=None, min_samples_split=1, random_state=0)

parameters = {'n_estimators':range(100,200,20)}
grid_rf = grid_search.GridSearchCV(etreg, parameters,n_jobs=2,scoring=manualScorer)

%%time
grid_rf.fit(X_train,y_train)

print(grid_rf.grid_scores_)
print("Best: ",grid_rf.best_params_)

grid_rf.best_params_

es = etreg
#es = grid_rf.best_estimator_

%%time
es = es.fit(X_train,y_train)

print(es.score(X_train,y_train))
print(es.score(X_test,y_test))

err = (es.predict(X_train)-y_train)**2
err.sum()/len(err)

err = (es.predict(X_test)-y_test)**2
err.sum()/len(err)


%%time
#filename = "data/reduced_test_5000.csv"
filename = "data/test.csv"
test = pd.read_csv(filename)
test = test.set_index('Id')

features_columns = list([u'Ref', u'Ref_5x5_10th',
       u'Ref_5x5_50th', u'Ref_5x5_90th', u'RefComposite',
       u'RefComposite_5x5_10th', u'RefComposite_5x5_50th',
       u'RefComposite_5x5_90th', u'RhoHV', u'RhoHV_5x5_10th',
       u'RhoHV_5x5_50th', u'RhoHV_5x5_90th', u'Zdr', u'Zdr_5x5_10th',
       u'Zdr_5x5_50th', u'Zdr_5x5_90th', u'Kdp', u'Kdp_5x5_10th',
       u'Kdp_5x5_50th', u'Kdp_5x5_90th'])

def getX(raw):
    selected_columns = list([ u'minutes_past',u'radardist_km', u'Ref', u'Ref_5x5_10th',
       u'Ref_5x5_50th', u'Ref_5x5_90th', u'RefComposite',
       u'RefComposite_5x5_10th', u'RefComposite_5x5_50th',
       u'RefComposite_5x5_90th', u'RhoHV', u'RhoHV_5x5_10th',
       u'RhoHV_5x5_50th', u'RhoHV_5x5_90th', u'Zdr', u'Zdr_5x5_10th',
       u'Zdr_5x5_50th', u'Zdr_5x5_90th', u'Kdp', u'Kdp_5x5_10th',
       u'Kdp_5x5_50th', u'Kdp_5x5_90th'])

    data = raw[selected_columns]

    docX= []
    for i in data.index.unique():
        if isinstance(data.loc[i],pd.core.series.Series):
            m = [data.loc[i].as_matrix()]
            docX.append(m)
        else:
            m = data.loc[i].as_matrix()
            docX.append(m)
    X = np.array(docX)
    return X

#%%time
#X=getX(test)

#tmp = []
#for i in X:
#    tmp.append(len(i))
#tmp = np.array(tmp)
#sns.countplot(tmp,order=range(tmp.min(),tmp.max()+1))
#plt.title("Number of ID per number of observations\n(On test dataset)")
#plt.plot()

testFull = test.dropna()

%%time
X=getX(testFull)  # 1min
#XX = [np.array(t).mean(0) for t in X]  # 10s

XX=addFeatures(X)

pd.DataFrame(gbr.predict(XX)).describe()

predFull = zip(testFull.index.unique(),gbr.predict(XX))

testNan = test.drop(test[features_columns].dropna(how='all').index)

tmp = np.empty(len(testNan))
tmp.fill(0.445000)   # 50th percentile of full Nan dataset
predNan = zip(testNan.index.unique(),tmp)

testLeft =...


knn = KNeighborsRegressor(n_neighbors=6,weights='distance',algorithm='ball_tree')

#parameters = {'weights':('distance','uniform'),'algorithm':('auto', 'ball_tree', 'kd_tree', 'brute')}
parameters = {'n_neighbors':range(1,10,1)}
grid_knn = grid_search.GridSearchCV(knn, parameters,scoring=manualScorer)

%%time
grid_knn.fit(X_train,y_train)

print(grid_knn.grid_scores_)
print("Best: ",grid_knn.best_params_)

knn = grid_knn.best_estimator_

knn= knn.fit(X_train,y_train)

print(knn.score(X_train,y_train))
print(knn.score(X_test,y_test))

err = (knn.predict(X_train)-y_train)**2
err.sum()/len(err)

err = (knn.predict(X_test)-y_test)**2
err.sum()/len(err)


gbr = GradientBoostingRegressor(loss='ls', learning_rate=0.1, n_estimators=900,
                                subsample=1.0, min_samples_split=2, min_samples_leaf=1, max_depth=4, init=None,
                                random_state=None, max_features=None, alpha=0.5,
                                verbose=0, max_leaf_nodes=None, warm_start=False)

%%time
gbr = gbr.fit(X_train,y_train)
#os.system('say "終わりだ"') #its over!

#parameters = {'max_depth':range(2,5,1),'alpha':[0.5,0.6,0.7,0.8,0.9]}
#parameters = {'subsample':[0.2,0.4,0.5,0.6,0.8,1]}
#parameters = {'subsample':[0.2,0.5,0.6,0.8,1],'n_estimators':[800,1000,1200]}
#parameters = {'max_depth':range(2,4,1)}
parameters = {'n_estimators':[400,800,1100]}
#parameters = {'loss':['ls', 'lad', 'huber', 'quantile'],'alpha':[0.3,0.5,0.8,0.9]}
#parameters = {'learning_rate':[0.1,0.5,0.9]}

grid_gbr = grid_search.GridSearchCV(gbr, parameters,n_jobs=2,scoring=manualScorer)

%%time
grid_gbr = grid_gbr.fit(X_train,y_train)

print(grid_gbr.grid_scores_)
print("Best: ",grid_gbr.best_params_)

err = (gbr.predict(X_train)-y_train)**2
print(err.sum()/len(err))
err = (gbr.predict(X_test)-y_test)**2
print(err.sum()/len(err))

err = (gbr.predict(X_train)-y_train)**2
print(err.sum()/len(err))
err = (gbr.predict(X_test)-y_test)**2
print(err.sum()/len(err))


I am working on the file "FDMS/TME3/Model_V4.ipynb".
The first task is:
Here for legacy
Can you write Python code for it?

from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Activation
from keras.optimizers import SGD,RMSprop

in_dim = len(XX[0])
out_dim = 1

model = Sequential()
# Dense(64) is a fully-connected layer with 64 hidden units.
# in the first layer, you must specify the expected input data shape:
# here, 20-dimensional vectors.
model.add(Dense(128, input_shape=(in_dim,)))
model.add(Activation('tanh'))
model.add(Dropout(0.5))
model.add(Dense(1, init='uniform'))
model.add(Activation('linear'))

#sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
#model.compile(loss='mean_squared_error', optimizer=sgd)

rms = RMSprop()
model.compile(loss='mean_squared_error', optimizer=rms)

#model.fit(X_train, y_train, nb_epoch=20, batch_size=16)
#score = model.evaluate(X_test, y_test, batch_size=16)

prep = []
for i in y_train:
    prep.append(min(i,20))

prep=np.array(prep)
mi,ma = prep.min(),prep.max()
fy = (prep-mi) / (ma-mi)
#my = fy.max()
#fy = fy/fy.max()

model.fit(np.array(X_train), fy, batch_size=10, nb_epoch=10, validation_split=0.1)

pred = model.predict(np.array(X_test))*ma+mi

err = (pred-y_test)**2
err.sum()/len(err)

r = random.randrange(len(X_train))
print("(Train) Prediction %0.4f, True: %0.4f"%(model.predict(np.array([X_train[r]]))[0][0]*ma+mi,y_train[r]))

r = random.randrange(len(X_test))
print("(Test)  Prediction %0.4f, True: %0.4f"%(model.predict(np.array([X_test[r]]))[0][0]*ma+mi,y_test[r]))


I am working on the file "FDMS/TME3/Model_V4.ipynb".
The first task is:
We regroup the data by ID
Can you write Python code for it?


I am working on the file "FDMS/TME3/Model_V4.ipynb".
The first task is:
max prof 24
nb trees 84
min sample per leaf 17
min sample to split 51
Can you write Python code for it?

from sklearn import svm

svr = svm.SVR(C=100000)

%%time
srv = svr.fit(X_train,y_train)

err = (svr.predict(X_train)-y_train)**2
err.sum()/len(err)

err = (svr.predict(X_test)-y_test)**2
err.sum()/len(err)

%%time
svr_score = cross_val_score(svr, XX, y, cv=5)
print("Score: %s\nMean: %.03f"%(svr_score,svr_score.mean()))


I am working on the file "FDMS/TME3/Model_V5-Flo.ipynb".
The first task is:

Can you write Python code for it?

etreg = ExtraTreesRegressor(n_estimators=200, max_depth=None, min_samples_split=1, random_state=0,n_jobs=4)

parameters = {'n_estimators':range(100,200,10)}
grid_rf = grid_search.GridSearchCV(etreg, parameters,n_jobs=4,scoring=manualScorer)

%%time
grid_rf.fit(X_train,y_train)

print(grid_rf.grid_scores_)
print("Best: ",grid_rf.best_params_)

grid_rf.best_params_

#etreg = grid_rf.best_estimator_

%%time
etreg = etreg.fit(X_train,y_train)

print(etreg.score(X_train,y_train))
print(etreg.score(X_test,y_test))

err = (etreg.predict(X_train)-y_train)**2
err.sum()/len(err)

err = (etreg.predict(X_test)-y_test)**2
err.sum()/len(err)


gbr = GradientBoostingRegressor(loss='ls', learning_rate=0.1, n_estimators=900,
                                subsample=1.0, min_samples_split=2, min_samples_leaf=1, max_depth=4, init=None,
                                random_state=None, max_features=None, alpha=0.5,
                                verbose=0, max_leaf_nodes=None, warm_start=False)

%%time
gbr = gbr.fit(X_train,y_train)
#os.system('say "終わりだ"') #its over!

#parameters = {'max_depth':range(2,5,1),'alpha':[0.5,0.6,0.7,0.8,0.9]}
#parameters = {'subsample':[0.2,0.4,0.5,0.6,0.8,1]}
#parameters = {'subsample':[0.2,0.5,0.6,0.8,1],'n_estimators':[800,1000,1200]}
#parameters = {'max_depth':range(2,4,1)}
parameters = {'n_estimators':[400,800,1100]}
#parameters = {'loss':['ls', 'lad', 'huber', 'quantile'],'alpha':[0.3,0.5,0.8,0.9]}
#parameters = {'learning_rate':[0.1,0.5,0.9]}

grid_gbr = grid_search.GridSearchCV(gbr, parameters,n_jobs=2,scoring=manualScorer)

%%time
grid_gbr = grid_gbr.fit(X_train,y_train)

print(grid_gbr.grid_scores_)
print("Best: ",grid_gbr.best_params_)

print(gbr.score(X_train,y_train))
print(gbr.score(X_test,y_test))

err = (gbr.predict(X_train)-y_train)**2
print(err.sum()/len(err))
err = (gbr.predict(X_test)-y_test)**2
print(err.sum()/len(err))

err = (gbr.predict(X_train)-y_train)**2
print(err.sum()/len(err))
err = (gbr.predict(X_test)-y_test)**2
print(err.sum()/len(err))


# the dbz feature does not influence xgbr so much
xgbr = xgb.XGBRegressor(max_depth=6, learning_rate=0.1, n_estimators=700, silent=True,
                        objective='reg:linear', nthread=-1, gamma=0, min_child_weight=1,
                        max_delta_step=0, subsample=1, colsample_bytree=1, colsample_bylevel=1,
                        reg_alpha=0, reg_lambda=1, scale_pos_weight=1, base_score=0.5,
                        seed=0, missing=None)

%%time
xgbr = xgbr.fit(X_train,y_train)

# without the nmf features
# print(xgbr.score(X_train,y_train))
## 0.993948231144
# print(xgbr.score(X_test,y_test))
## 0.613931733332

# with nmf features
print(xgbr.score(X_train,y_train))
print(xgbr.score(X_test,y_test))


rfr = RandomForestRegressor(n_estimators=200, criterion='mse', max_depth=None, min_samples_split=2,
                            min_samples_leaf=1, min_weight_fraction_leaf=0.0, max_features='auto',
                            max_leaf_nodes=None, bootstrap=True, oob_score=False, n_jobs=-1,
                            random_state=None, verbose=0, warm_start=False)

%%time
rfr = rfr.fit(X_train,y_train)

print(rfr.score(X_train,y_train))
print(rfr.score(X_test,y_test))


def manualScorer(estimator, X, y):
    err = (estimator.predict(X_test)-y_test)**2
    return -err.sum()/len(err)


I am working on the file "FDMS/TME3/Model_V5-Flo.ipynb".
The first task is:
Here for legacy
Can you write Python code for it?

in_dim = len(XX[0])
out_dim = 1

model = Sequential()
# Dense(64) is a fully-connected layer with 64 hidden units.
# in the first layer, you must specify the expected input data shape:
# here, 20-dimensional vectors.
model.add(Dense(128, input_shape=(in_dim,)))
model.add(Activation('tanh'))
model.add(Dropout(0.5))
model.add(Dense(1, init='uniform'))
model.add(Activation('linear'))

#sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
#model.compile(loss='mean_squared_error', optimizer=sgd)

rms = RMSprop()
model.compile(loss='mean_squared_error', optimizer=rms)

#model.fit(X_train, y_train, nb_epoch=20, batch_size=16)
#score = model.evaluate(X_test, y_test, batch_size=16)

prep = []
for i in y_train:
    prep.append(min(i,20))

prep=np.array(prep)
mi,ma = prep.min(),prep.max()
fy = (prep-mi) / (ma-mi)
#my = fy.max()
#fy = fy/fy.max()

model.fit(np.array(X_train), fy, batch_size=10, nb_epoch=10, validation_split=0.1)

pred = model.predict(np.array(X_test))*ma+mi

err = (pred-y_test)**2
err.sum()/len(err)

r = random.randrange(len(X_train))
print("(Train) Prediction %0.4f, True: %0.4f"%(model.predict(np.array([X_train[r]]))[0][0]*ma+mi,y_train[r]))

r = random.randrange(len(X_test))
print("(Test)  Prediction %0.4f, True: %0.4f"%(model.predict(np.array([X_test[r]]))[0][0]*ma+mi,y_test[r]))


I am working on the file "FDMS/TME3/Model_V5-Flo.ipynb".
The first task is:
On fully filled dataset
Can you write Python code for it?

#noAnyNan = raw.loc[raw[features_columns].dropna(how='any').index.unique()]
noAnyNan = raw.dropna()

noFullNan = raw.loc[raw[features_columns].dropna(how='all').index.unique()]

fullNan = raw.drop(raw[features_columns].dropna(how='all').index)

print(len(raw))
print(len(noAnyNan))
print(len(noFullNan))
print(len(fullNan))


I am working on the file "FDMS/TME3/Model_V5-Flo.ipynb".
The first task is:
Predicitons
As a first try, we make predictions on the complete data, and return the 50th percentile and uncomplete and fully empty data
Can you write Python code for it?

%%time
#X,y=getXy(noAnyNan)
X,y=getXy(noFullNan)

#%%time
#XX = [np.array(t).mean(0) for t in X]
#XX = np.array([np.append(np.nanmean(np.array(t),0),(np.array(t)[1:] - np.array(t)[:-1]).sum(0) ) for t in X])

XX=[]
for t in X:
    nm = np.nanmean(t,0)
    for idx,j in enumerate(nm):
        if np.isnan(j):
            nm[idx]=global_means[idx]
    XX.append(nm)
XX=np.array(XX)

# rescale to clip min at 0 (for non negative matrix factorization)
XX_rescaled=XX[:,:]-np.min(XX,0)

%%time
nn = ProjectedGradientNMF()
W = nn.fit_transform(XX_rescaled)
#H = nn.components_

global_means = np.nanmean(noFullNan,0)

XX=[]
for t in X:
    nm = np.nanmean(t,0)
    for idx,j in enumerate(nm):
        if np.isnan(j):
            nm[idx]=global_means[idx]
    XX.append(nm)
XX=np.array(XX)

# rescale to clip min at 0 (for non negative matrix factorization)
XX_rescaled=XX[:,:]-np.min(XX,0)

nmf = NMF(max_iter=1000)
W = nmf.fit_transform(XX_rescaled)
#H = nn.components_

# used to fill fully empty datas
global_means = np.nanmean(noFullNan,0)

# reduce the sequence structure of the data and produce
# new hopefully informatives features
def addFeatures(X,mf=0):
    # used to fill fully empty datas
    #global_means = np.nanmean(X,0)

    XX=[]
    nbFeatures=float(len(X[0][0]))
    for idxt,t in enumerate(X):

        # compute means, ignoring nan when possible, marking it when fully filled with nan
        nm = np.nanmean(t,0)
        tt=[]
        for idx,j in enumerate(nm):
            if np.isnan(j):
                nm[idx]=global_means[idx]
                tt.append(1)
            else:
                tt.append(0)
        tmp = np.append(nm,np.append(tt,tt.count(0)/nbFeatures))

        # faster if working on fully filled data:
        #tmp = np.append(np.nanmean(np.array(t),0),(np.array(t)[1:] - np.array(t)[:-1]).sum(0) )

        # add the percentiles
        tmp = np.append(tmp,np.nanpercentile(t,10,axis=0))
        tmp = np.append(tmp,np.nanpercentile(t,50,axis=0))
        tmp =...


I am working on the file "FDMS/TME3/Model_V5-Flo.ipynb".
The first task is:
Predict on testset
Can you write Python code for it?


I am working on the file "FDMS/TME3/Model_V5-Flo.ipynb".
The first task is:
max prof 24
nb trees 84
min sample per leaf 17
min sample to split 51
Can you write Python code for it?

svr = SVR(kernel='rbf', C=800.0)

%%time
srv = svr.fit(X_train,y_train)

print(svr.score(X_train,y_train))
print(svr.score(X_test,y_test))

err = (svr.predict(X_train)-y_train)**2
err.sum()/len(err)

err = (svr.predict(X_test)-y_test)**2
err.sum()/len(err)

%%time
svr_score = cross_val_score(svr, XX, y, cv=5)
print("Score: %s\nMean: %.03f"%(svr_score,svr_score.mean()))


I am working on the file "FDMS/TME3/Model_V5.ipynb".
The first task is:

Can you write Python code for it?

svr.predict(X_test)

s = modelList[0]

t.mean(1)

modelList = [svr,knn,etreg,rfr,xgbr,gbr]
reducedModelList = [knn,etreg,xgbr,gbr]

score_train = [[str(f).split("(")[0],f.score(X_train,y_train)] for f in modelList]
score_test = [[str(f).split("(")[0],f.score(X_test,y_test)] for f in modelList]
for idx,i in enumerate(score_train):
    print(i[0])
    print("   train: %.03f"%i[1])
    print("   test:  %.03f"%score_test[idx][1])

#reducedModelList = [knn,etreg,xgbr,gbr]
globalPred = np.array([f.predict(XX) for f in reducedModelList]).T
#globalPred.mean(1)

globalPred[0]

y[0]

err = (globalPred.mean(1)-y)**2
print(err.sum()/len(err))

err = (globalPred.mean(1)-y)**2
print(err.sum()/len(err))

for f in modelList:
    print(str(f).split("(")[0])
    err = (f.predict(XX)-y)**2
    print(err.sum()/len(err))

err = (XX[:,0]-y)**2
print(err.sum()/len(err))

for f in modelList:
    print(str(f).split("(")[0])
    print(f.score(XX,y))

XX[:10,0] # feature 0 is marshall-palmer

svrMeta = SVR()

%%time
svrMeta = svrMeta.fit(globalPred,y)

err = (svrMeta.predict(globalPred)-y)**2
print(err.sum()/len(err))


t = []
for i in XX:
    t.append(np.count_nonzero(~np.isnan(i)) / float(i.size))
pd.DataFrame(np.array(t)).describe()


I am working on the file "FDMS/TME3/Model_V5.ipynb".
The first task is:
Predicitons
As a first try, we make predictions on the complete data, and return the 50th percentile and uncomplete and fully empty data
Can you write Python code for it?

%%time
#X,y=getXy(noAnyNan)
X,y=getXy(noFullNan)

len(X[0][0])

#%%time
#XX = [np.array(t).mean(0) for t in X]
#XX = np.array([np.append(np.nanmean(np.array(t),0),(np.array(t)[1:] - np.array(t)[:-1]).sum(0) ) for t in X])

global_means = np.nanmean(noFullNan,0)

XX=[]
for t in X:
    nm = np.nanmean(t,0)
    for idx,j in enumerate(nm):
        if np.isnan(j):
            nm[idx]=global_means[idx]
    XX.append(nm)
XX=np.array(XX)

# rescale to clip min at 0 (for non negative matrix factorization)
XX_rescaled=XX[:,:]-np.min(XX,0)

%%time
nmf = ProjectedGradientNMF()
W = nmf.fit_transform(XX_rescaled)
#H = nn.components_

# used to fill fully empty datas
global_means = np.nanmean(noFullNan,0)

# reduce the sequence structure of the data and produce
# new hopefully informatives features
def addFeatures(X,mf=0):
    # used to fill fully empty datas
    #global_means = np.nanmean(X,0)

    XX=[]
    nbFeatures=float(len(X[0][0]))
    for idxt,t in enumerate(X):

        # compute means, ignoring nan when possible, marking it when fully filled with nan
        nm = np.nanmean(t,0)
        tt=[]
        for idx,j in enumerate(nm):
            if np.isnan(j):
                nm[idx]=global_means[idx]
                tt.append(1)
            else:
                tt.append(0)
        tmp = np.append(nm,np.append(tt,tt.count(0)/nbFeatures))

        # faster if working on fully filled data:
        #tmp = np.append(np.nanmean(np.array(t),0),(np.array(t)[1:] - np.array(t)[:-1]).sum(0) )

        # add the percentiles
        tmp = np.append(tmp,np.nanpercentile(t,10,axis=0))
        tmp = np.append(tmp,np.nanpercentile(t,50,axis=0))
        tmp = np.append(tmp,np.nanpercentile(t,90,axis=0))

        for idx,i in enumerate(tmp):
            if np.isnan(i):
                tmp[idx]=0

        # adding the dbz as a feature
        test = t
        try:
            taa=test[:,0]
        except TypeError:
            taa=[test[0][0]]
        valid_time = np.zeros_like(taa)
       ...