Muennighoff/xP3x-sample huggingface dataset:
inputs: "3.2. KSEA in detail
In the following, we show in detail the computations that are carried out inside the provided functions. Let us concentrate on a single condition (60 minutes after stimulation with prostaglandin E2) and a single kinase (CDK1)."
Please write code following the instructions in jupyter notebook style.

targets: 
data_condition = data_fc['60min'].copy()

p_values = data_p_value['60min']

kinase = 'CDK1'

substrates = adj_matrix[kinase].replace(0, np.nan).dropna().index

detected_p_sites = data_fc.index

intersect = list(set(substrates).intersection(detected_p_sites))
inputs: "<div class="alert alert-info">
There are many other vectorized computations that you can do on NumPy arrays, including multiplication (<code>np.prod</code>), mean (<code>np.mean</code>), and variance (<code>np.var</code>). They all act essentially the same way as <code>np.sum</code> -- give the function an array, and it computes the relevant function across all the elements in the array.
</div>

Exercise: Euclidean distance (2 points)
Recall that the Euclidean distance $d$ is given by the following equation:
$$
d(a, b) = \sqrt{\sum_{i=1}^N (a_i - b_i) ^ 2}
$$
In NumPy, this is a fairly simple computation because we can rely on array computations and the np.sum function to do all the heavy lifting for us.
<div class="alert alert-success">
Complete the function <code>euclidean_distance</code> below to compute $d(a,b)$, as given by the equation above. Note that you can compute the square root using <code>np.sqrt</code>.
</div>"
Please write code following the instructions in jupyter notebook style.

targets: 
def euclidean_distance(a, b):
    """Computes the Euclidean distance between a and b.
    
    Hint: your solution can be done in a single line of code!
    
    Parameters
    ----------
    a, b : numpy arrays or scalars with the same size
    
    Returns
    -------
    the Euclidean distance between a and b
    
    """
    ### BEGIN SOLUTION
    return np.sqrt(np.sum((a - b) ** 2))
    ### END SOLUTION
inputs: # load model without top layer
n_images_train=500
n_images_test=500
input_image_shape = (3,224,224)
train_features, train_labels,train_files, \
test_features, test_labels, test_files = dvc.training_test_datasets(all_files,
                                                                    n_images_train,n_images_test,
                                                                    input_image_shape)

# load_img from keras.preprocessing loads the images in [0,255] scale
train_features = preprocess_input(train_features)
test_features = preprocess_input(test_features)

from keras.models import Model
base_model = VGG16(weights='imagenet')
model = Model(input=base_model.input, output=base_model.get_layer('fc2').output)

print("Predicting train images")
train_features_cnn = model.predict(train_features,verbose=1)
print("Predicting test images")
test_features_cnn = model.predict(test_features,verbose=1)

train_features_cnn.shape

from sklearn import svm
from sklearn.model_selection import GridSearchCV

tuned_parameters = {'kernel': ['linear'],
                     'C': [1, 10, 100, 1000]}

clf = GridSearchCV(svm.SVC(C=1), tuned_parameters, cv=5,n_jobs=7)
clf.fit(train_features_cnn, train_labels)

clf.best_estimator_

print("Train score: {}".format(clf.score(train_features_cnn,train_labels)))
print("Test score: {}".format(clf.score(test_features_cnn,test_labels)))
Given the above code, generate some markdown instructions for it.

targets: 
Using pretrained CNN as feature extractors
inputs: %%sql
select year, count(*) from SeattleCrimeIncidents
group by year;
Given the above code, generate some markdown instructions for it.

targets: 
How many crimes per year?
inputs: $||\mathbf{w}||2 = \sqrt{\sum{j=1}^D w_j^2}$

targets: def norm_2(x):
    c=0
    for i in range(len(x)):
        c += x[i]**2
    return math.sqrt(c)
inputs: %matplotlib inline
import numpy as np
from scipy import linalg
import matplotlib.pyplot as plt
import matplotlib as mpl

from sklearn import mixture


def plot_gmm(X, models, n_components, covariance_type='diag',
            figsize=(10, 20), suptitle=None, xlabel=None, ylabel=None):
    color_iter = ['r', 'g', 'b', 'c', 'm', 'y', 'k', 'gray', 'pink', 'lime']
    plt.figure(figsize=figsize)
    plt.suptitle(suptitle, fontsize=20)

    for i, model in enumerate(models):
        mm = getattr(mixture, model)(n_components=n_components,
                                     covariance_type=covariance_type)
        mm.fit(X_pos_qid)
        Y = mm.predict(X_pos_qid)

        plt.subplot(len(models), 1, 1 + i)
        for i, color in enumerate(color_iter):
            plt.scatter(X_pos_qid[Y == i, 0], X_pos_qid[Y == i, 1], .7, color=color)
        plt.title(model, fontsize=15)
        plt.xlabel(xlabel, fontsize=12)
        plt.ylabel(ylabel, fontsize=12)
        plt.grid()

    plt.show()

from collections import UserDict
import numpy as np


class DictDict(UserDict):
    def __init__(self, bd):
        UserDict.__init__(self)
        self._set_bd(bd)
        
    def sub_keys(self):
        return self[list(self.keys())[0]].keys()
            
    def select(self, sub_keys):
        vals = []
        for key in self:
            vals.append([self[key][sub_key] for sub_key in sub_keys])
        return np.array(vals)
    
    def sub_append(self, sub_key, values):
        for index, key in enumerate(self):
            self[key][sub_key] = values[index]

    
class Users(DictDict):
    def _set_bd(self, bd):
        pos_uid, _, _ = _get_pos(bd['train'], sign_val=None)
        for key in pos_uid:
            u = np.array(pos_uid[key])
            ave_pos_uid = sum(abs(u)) / float(len(u))
            acc_ratio_uid = len(u[u > 0]) / float(len(u))
            self[key] = {'ave_pos_uid': ave_pos_uid,
                         'acc_ratio_uid': acc_ratio_uid}

            
class Questions(DictDict):
    def _set_bd(self,...
Given the above code, generate some markdown instructions for it.

targets: 
GMM
Classifying questions
features: avg_pos, accuracy rate
inputs: %set_cell_height 300

# Specify the model.
model_1 = tfdf.keras.RandomForestModel(num_trees=30)

# Train the model.
model_1.fit(x=train_ds)
Given the above code, generate some markdown instructions for it.

targets: 
Finaly, train and evaluate the model as usual. TF-DF  automatically detects multi-valued categorical features as categorical-set.
inputs: I am working on the file "P3:_creating_customer_segments/.ipynb_checkpoints/customer_segments-checkpoint.ipynb".
The first task is:
Implementation: Dimensionality Reduction
When using principal component analysis, one of the main goals is to reduce the dimensionality of the data — in effect, reducing the complexity of the problem. Dimensionality reduction comes at a cost: Fewer dimensions used implies less of the total variance in the data is being explained. Because of this, the cumulative explained variance ratio is extremely important for knowing how many dimensions are necessary for the problem. Additionally, if a signifiant amount of variance is explained by only two or three dimensions, the reduced data can be visualized afterwards.
In the code block below, you will need to implement the following:
 - Assign the results of fitting PCA in two dimensions with good_data to pca.
 - Apply a PCA transformation of good_data using pca.transform, and assign the reuslts to reduced_data.
 - Apply a PCA transformation of the sample log-data log_samples using pca.transform, and assign the results to pca_samples.
Can you write Python code for it?

targets: 
# TODO: Fit PCA to the good data using only two dimensions
pca = PCA(n_components=2)
pca.fit(good_data)

# TODO: Apply a PCA transformation the good data
reduced_data = pca.transform(good_data)

# TODO: Apply a PCA transformation to the sample log-data
pca_samples = pca.transform(log_samples)

# Create a DataFrame for the reduced data
reduced_data = pd.DataFrame(reduced_data, columns = ['Dimension 1', 'Dimension 2'])
inputs: I am working on the file "Cal3-PythonGraphes.ipynb".
The first task is:
2.16 Graphes de contour
Colormaps and contour figures are useful for plotting functions of two variables. In most of these functions we will use a colormap to encode one dimension of the data. There are a number of predefined colormaps. It is relatively straightforward to define custom colormaps. For a list of pre-defined colormaps, see: http://www.scipy.org/Cookbook/Matplotlib/Show_colormaps
Can you write Python code for it?

targets: 
alpha = 0.7
phi_ext = 2 * np.pi * 0.5

def flux_qubit_potential(phi_m, phi_p):
    return 2 + alpha - 2 * np.cos(phi_p) * np.cos(phi_m) - alpha * np.cos(phi_ext - 2*phi_p)

phi_m = np.linspace(0, 2*np.pi, 100)
phi_p = np.linspace(0, 2*np.pi, 100)
X,Y = np.meshgrid(phi_p, phi_m)
Z = flux_qubit_potential(X, Y).T
inputs: # old name: process_data
def transform_data(selected_columns, data):
    '''
    Apply log-transform and MinMaxScaler() to the selected data columns which are not fractions (frac_*)
    
    Parameters
    ==========
    selected_columns : list
        list of columns to leave in processed data
    data : pandas.DataFrame
        data to process (note that data should contain all selected_columns)
        
    Returns
    =======
    log_scaled_data : pandas.DataFrame
        log-transformed and scaled data selected by selected_columns
    '''
    
    data.reset_index(drop=True, inplace=True)
    log_data = data[selected_columns].copy()
    
    skewed = log_data.columns.tolist()
    skewed = [item for item in skewed if not item.startswith('frac_')]
    log_data[skewed] = log_data[skewed].apply(lambda x: np.log10(x + 1))

    scaler = MinMaxScaler().fit(log_data)
    log_scaled_data = scaler.transform(log_data)
    log_scaled_data = pd.DataFrame(log_scaled_data, columns=log_data.columns)
    
    return log_scaled_data

def replace_group_numbers(best_preds):
    '''
    Replace group numbers in best_preds with sorting by group size 
    (so that the largest group is 0, the second largest is 1 etc.)
    
    Parameters
    ==========
    best_preds : numpy array
        unsorted array of predictions
    
    Returns
    =======
    best_preds_sorted : numpy array
        sorted array of predictions
    '''
    
    pp = pd.DataFrame(best_preds, columns = ["old_group"])
    dict_pp = {item[0]: i for i, item in enumerate(Counter(best_preds).most_common())}
    pp['new_group'] = pp['old_group'].replace(dict_pp)
    best_preds_sorted = np.array(pp['new_group'])
    return best_preds_sorted

def kmeans(log_scaled_data):
    '''
    Apply KMeans clustering algorithm with 2 <= cluster_number <= 6 to log_scaled_data 
    (transformed and scaled by transform_data() function)
    
    Parameters
    ==========
    log_scaled_data : pandas.DataFrame
        data log-transormed and MinMaxScaler()-ed for KMeans...
Given the above code, generate some markdown instructions for it.

targets: 
Clustering
Write a new clustering algorithm that:
- starts from stud_data or its subset (with monotonic index)
- finds a 2-column set with the largest score (using KMeans) and renames it that 0 is the largest group, 1 is the second largest etc.
- returns index file (with indices 0, 1, ) that could be used for further analysis
inputs: "Alternatively, we can get a random value between 0 and 1 using random.random() and simply multiply it by 10 to achieve the same thing. There are cases where one method might be preferable over the other, so it is useful to know them both."
Please write code following the instructions in jupyter notebook style.

targets: 
random.random()*10
inputs: "By combining Boolean operations, masking operations, and aggregates, we can very quickly answer these sorts of questions for our dataset.
Aside: Using the Keywords and/or Versus the Operators &/|
One common point of confusion is the difference between the keywords and and or on one hand, and the operators &amp; and | on the other hand.
When would you use one versus the other?
The difference is this: and and or gauge the truth or falsehood of entire object, while &amp; and | refer to bits within each object.
When you use and or or, it's equivalent to asking Python to treat the object as a single Boolean entity.
In Python, all nonzero integers will evaluate as True. Thus:"
Please write code following the instructions in jupyter notebook style.

targets: 
bool(42), bool(0)

bool(42 and 0)

bool(42 or 0)
inputs: Create a median composite

targets: median_composite = landsat_ds.median('time').persist()
inputs: I am working on the file "1504_Seismic_petrophysics_1/Seismic_petrophysics_1.ipynb".
The first task is:
Notice that there is no gas_sand log (occurrence of gas sands) yet, as the insitu log did not record any gas sand. We will deal with that later.
I use the above flag logs to create the LFC log and store it into the logs DataFrame:
Can you write Python code for it?

targets: 
temp_lfc = np.zeros(np.shape(logs.VSH))
temp_lfc[brine_sand.values] = 1    # LFC will be 1 when ssb (brine sand flag) is True
temp_lfc[oil_sand.values] = 2      # LFC will be 2 when sso (oil sand flag) is True
temp_lfc[shale.values] = 4         # LFC will be 4 when sh (shale flag) is True
logs['LFC'] = temp_lfc             # Copy the temporary log temp_lfc into the DataFrame with name `LFC`

logs.to_csv('qsiwell2_lfc.csv',index=False) # save the data for use in Part 2
inputs: #$HIDE_INPUT$
y = tunnel["NumVehicles"]

model = LinearRegression(fit_intercept=False)
_ = model.fit(X, y)

y_pred = pd.Series(model.predict(X), index=y.index)
X_fore = dp.out_of_sample(steps=90)
y_fore = pd.Series(model.predict(X_fore), index=X_fore.index)

ax = y.plot(color='0.25', style='.', title="Tunnel Traffic - Seasonal Forecast")
ax = y_pred.plot(ax=ax, label="Seasonal")
ax = y_fore.plot(ax=ax, label="Seasonal Forecast", color='C3')
_ = ax.legend()
Given the above code, generate some markdown instructions for it.

targets: 
With our feature set created, we're ready to fit the model and make predictions. We'll add a 90-day forecast to see how our model extrapolates beyond the training data. The code here is the same as that in earlier lessons.
inputs: Add EPA emission factors for fossil fuels not included in the EIA file
https://www.epa.gov/sites/production/files/2015-07/documents/emission-factors_2014.pdf
- BFG (blast furnace gas)
- MSW (non-biomass portion)
- OG (other gas, assume fuel gas)
Add zero values for all bio fuels. Still need to figure out what to do for synthetic gas from coal and petroleum...

targets: ef.loc['BFG',:] = ['Blast Furnace Gas', 274.32, None , 'https://www.epa.gov/sites/production/files/2015-07/documents/emission-factors_2014.pdf']
ef.loc['MSN',:] = ['Non-biomass Municipal Solid Waste', 90.7, 'Assume the same as MSW', 'https://www.epa.gov/sites/production/files/2015-07/documents/emission-factors_2014.pdf']

# Use the bituminous coal factor for synthetic coal
ef.loc['SC',:] = ['Synthetic Coal', 93.3, 'Use same factor as BIT and RC', None]

# Use fuel gas for other gases
ef.loc['OG', :] = ['Other gases', 59, 'Assume fuel gas', 'https://www.epa.gov/sites/production/files/2015-07/documents/emission-factors_2014.pdf']

# Use BIT factor for synthetic gas from coal (SGC) and DFO for SGP
# The actual factors won't matter as much because we're going to use the EPA 
# emission values. 
ef.loc['SGC', ['Fossil Factor', 'Notes']] = [93.3, 'Using BIT value. Likely not correct']
ef.loc['SGP', ['Fossil Factor', 'Notes']] = [73.16, 'Using DFO value. Likely not correct']

non_fossil_fuels = ['AB', 'BLQ', 'LFG', 'MSB', 'NUC', 'OBG', 'OBL', 'OBS',
                    'OTH', 'PUR', 'SLW', 'SUN', 'WAT', 'WDL', 'WDS', 'WH', 'WND']

for fuel in non_fossil_fuels:
    ef.loc[fuel, ['Fossil Factor', 'Notes']] = [0, 'non-fossil fuel']
inputs: Choose the vertical line on the right panel to create a vertical projection. Position it at $Q_y=0$. Click to the floppy disk to save projection to a text file. The result should look as shown below.

targets: projfile = "data/proj1.txt"
data = np.loadtxt(projfile)
plt.semilogy(data[:,0], data[:,1], color='r')
plt.xlabel(r'$Q_z$ (nm$^{-1}$)')
plt.ylabel('Intensity (a. u.)')
plt.title(r"Vertical slice at $Q_y=0$ nm$^{-1}$")
plt.xlim(0.0, 2.3);
inputs: # loss A
#    ||h(s) - a|^p - R|^q
# This is danabo's mode regression loss

p = 0.1
q = 1/P
# p = q = 2.0
def loss(model, s, a, y):
  reg = tf.linalg.global_norm(model.trainable_variables)
  return tf.reduce_mean(tf.abs(-tf.abs(model(s)-a)**p - y)**q) + 0.003*reg

# loss B
#    |h(s) - a|^p * exp(R/tau)
# This is one of Dale's surrogate loss, specifically dot-product loss.

p = 1.0
tau = 1/10.
def loss(model, s, a, y):
  reg = tf.linalg.global_norm(model.trainable_variables)
  target = tf.cast(tf.exp(y/tau), tf.float32)
  return tf.reduce_mean(tf.abs(model(s)-a)**p * target) + 0.0005*reg

np.var(s)

# Initialize model

device_string = '/device:GPU:0'
# device_string = '/device:TPU:0'
# device_string = ''  # CPU

with tf.device(device_string):
  model(data[0])
  print(loss(model, *data).numpy())  # Initialize model

optimizer = tf.keras.optimizers.Adam(learning_rate=0.0001)

def sample_batch(batch_size, *args):
  assert args
  idx = np.random.choice(args[0].shape[0], batch_size)
  return tuple([arg[idx] for arg in args])

for i in range(10000):
  # batch = sample_batch(100, *data)
  batch = data
  optimizer.minimize(lambda: loss(model, *batch), model.trainable_variables)
  if i % 100 == 0:
    print(i, '\t', loss(model, *data).numpy())
Given the above code, generate some markdown instructions for it.

targets: 
What loss functions best recover the curve $f$ from our dataset?
inputs: Combine all the walkers into a single array of generated samples.  You could trim off some initial "burn-in" here, but I don't recommend it (see lecture notes):

targets: samples = sampler.chain[:, :, :].reshape((-1, ndim))

samples.shape
inputs: Converting Categorical Features
We'll need to convert categorical features to dummy variables using pandas! Otherwise our machine learning algorithm won't be able to directly take in those features as inputs.

targets: train.info()

sex = pd.get_dummies(train['Sex'],drop_first=True)
embark = pd.get_dummies(train['Embarked'],drop_first=True)

train.drop(['Sex','Embarked','Name','Ticket'],axis=1,inplace=True)

train = pd.concat([train,sex,embark],axis=1)

train.head()
inputs: #  df[['Company Name', 'TOV Category', 'Amount']].head()
df.groupby(('Company Name', 'TOV Category')).sum().sort_values('Amount', ascending=False).head(10)
Given the above code, generate some markdown instructions for it.

targets: 
Payments by pharma companies by category
Break down the above table by category, too. Again, this is only declared payments.
inputs: %%R
iris %>%
group_by(Species) %>%
summarise(Sepal.Width.Avg = mean(Sepal.Width)) %>%
arrange(Sepal.Width.Avg)

%%R
library(ggplot2)
ggplot(data=iris, aes(x=Sepal.Length, y=Sepal.Width, color=Species)) + geom_point(size=3)
Given the above code, generate some markdown instructions for it.

targets: 
Le formatage de l'output n'est pas parfaitement identique à un notebook en R nativement
* bloc d'output est plus long
* headers pas en gras
* ...
inputs: "(NILMTK imports the 'common metadata' from the NILM Metadata project, which includes a wide range of different category taxonomies)"
Please write code following the instructions in jupyter notebook style.

targets: 
nilmtk.global_meter_group.select_using_appliances(building=2, category='laundry appliances')
inputs: I am working on the file "01-Regression/LRaudio.ipynb".
The first task is:
Embedding the audio file.
Note that this is not working directly in GitHub (I think all JavaScript is stripped out), fork it or download it to play the audio
Can you write Python code for it?

targets: 
from IPython.display import Audio
Audio("../outputs/OriginalTestClip.wav")
inputs: I am working on the file "Db2 Jupyter Extensions Tutorial.ipynb".
The first task is:
Notice how the string constant was updated to contain two quotes when inserted into the SQL statement. This is done automatically by the %sql magic command, so there is no need to use the two single quotes when assigning a string to a variable. However, you must use the two single quotes when using constants in a SQL statement.
Builtin Variables
There are 5 predefined variables defined in the program:

database - The name of the database you are connected to
uid - The userid that you connected with
hostname = The IP address of the host system
port - The port number of the host system
max - The maximum number of rows to return in an answer set

Theses variables are all part of a structure called _settings. To retrieve a value, use the syntax:
python
db = _settings['database']
There are also 3 variables that contain information from the last SQL statement that was executed.

sqlcode - SQLCODE from the last statement executed
sqlstate - SQLSTATE from the last statement executed
sqlerror - Full error message returned on last statement executed

You can access these variables directly in your code. The following code segment illustrates the use of the SQLCODE variable.
Can you write Python code for it?

targets: 
empnos = ['000010','999999']
for empno in empnos:
    ans1 = %sql -r SELECT SALARY FROM EMPLOYEE WHERE EMPNO = :empno
    if (sqlcode != 0):
        print("Employee "+ empno + " left the company!")
    else:
        print("Employee "+ empno + " salary is " + str(ans1[1][0]))
inputs: # Build a Distributed Random Forest (DRF) model with default settings

# Import the function for DRF
from h2o.estimators.random_forest import H2ORandomForestEstimator

# Set up DRF for regression
# Add a seed for reproducibility
drf_default = H2ORandomForestEstimator(model_id = 'drf_default', seed = 1234)

# Use .train() to build the model
drf_default.train(x = features, 
                  y = 'Survived', 
                  training_frame = titanic_train)

# Check the DRF model summary
drf_default

# Check the model performance on test dataset
drf_default.model_performance(titanic_test)
Given the above code, generate some markdown instructions for it.

targets: 
<br>
Distributed Random Forest
inputs: <img src='./tables/Dorogokupets2007_Au.png'>

targets: print('for T = ', temp)
for eta_i, p_i in zip(eta, p):
    print("{0: .3f} {1: .2f} ".format(eta_i, p_i))

v = dorogokupets2007_au.cal_v(p, temp * np.ones_like(p), min_strain=0.6)
print(1.-(v/v0))
inputs: "Alternate Representations
modf() takes a single floating point number and returns a tuple containing the fractional and whole number parts of the input value."
Please write code following the instructions in jupyter notebook style.

targets: 
import math

for i in range(6):
    print '{}/2 = {}'.format(i, math.modf(i/2.0))
inputs: "4.4  For testuserid, once again, print out the ratings using the bizs list as before. How have they changed with respect to Question 2? Why might this be?"
Please write code following the instructions in jupyter notebook style.

targets: 
#your code here
print "for user",usernamefromid(fulldf,testuserid), 'avg', fulldf[fulldf.user_id==testuserid].stars.mean() 
for i in bizs:
    print "========="
    print biznamefromid(fulldf, i), i
    print rating(fulldf, dbfull, i, testuserid, k=7, reg=3.) 
    u,a=get_other_ratings(i, testuserid, fulldf)
    print "User Score:",u,"Avg score",a
inputs: # Fit anamorphosis by changing, zmax, zmin, and extrapolation function
PCI, H, raw, zana, gauss, z, P, \
raw_var, PCI_var, fig1, \
zamin, zamax, yamin, yamax, \
zpmin, zpmax, ypmin, ypmax = pygslib.nonlinear.anamor(
                 z = data['Z'], 
                 w = data['Declustering Weight'], 
                 zmin = data['Z'].min()-0.1, 
                 zmax = data['Z'].max()+1,
                 zpmin = None,
                 zpmax = data['Z'].max()+1.5,
                 ymin=-2.9, ymax=2.9,
                 ndisc = 5000,
                 ltail=1, utail=4, ltpar=1, utpar=1.8, K=30)

PCI
Given the above code, generate some markdown instructions for it.

targets: 
Interactive anamorphosis modeling
inputs: 1.4.10

targets: from image import file2image
from image import image2display
from plotting import plot
img = file2image("img01.png")
#image2display(img)
width = len(img[0])
height = len(img)
lt = []
[[lt.append(x+(187*1j-y*1j)) for x in range(width) if sum(img[y][x])<360]for y in range(height)]
plot(lt,188)
inputs: "Creating a DatetimeIndex on a Dataframe
Much of the original motivation for writing pandas was for financial time series modeling. As a result, there are many efficiently implemented methods and conveniences built in the data structures. 
Many of these depend on dataframe rows being addressed (indexed) with a time-like data type, which is exactly what the DatetimeIndex is for. 
If our data has some sort of timestamp in it, how do we convert it to an associated datetimeindex?"
Please write code following the instructions in jupyter notebook style.

targets: 
d = pd.read_csv('misc/small.csv', names=['date','value'])

d.tail()

# is 'date' a data-like data type?
d.dtypes
inputs: "<div id='ex' />

Examples of QR: Classic vs Modified Gram-Schmidt
Back to TOC"
Please write code following the instructions in jupyter notebook style.

targets: 
# Fore: BLACK, RED, GREEN, YELLOW, BLUE, MAGENTA, CYAN, WHITE, RESET.
# Back: BLACK, RED, GREEN, YELLOW, BLUE, MAGENTA, CYAN, WHITE, RESET.
# Style: DIM, NORMAL, BRIGHT, RESET_ALL
textQRClassic = lambda x: Style.BRIGHT+Fore.RED+x+Style.RESET_ALL
textQRModified = lambda x: Style.BRIGHT+Fore.BLUE+x+Style.RESET_ALL
textAnswer = lambda x: Back.BLACK+Fore.CYAN+x+Style.RESET_ALL

# Example 4.16 from textbook.
d = 1e-10
A = np.array([[1,1,1],[d,0,0],[0,d,0],[0,0,d]])
print(textBold('A: \n'), A)

Qc,Rc = QR(A, type_gram_schmidt = 'classic')
Qm,Rm = QR(A, type_gram_schmidt = 'modified')

################################
print(textBoldH('\n What are the Q\'s?'))
print(textQRClassic('Qc: \n'), Qc)
print(textQRModified('Qm: \n'), Qm)

################################
print(textBoldH('\n Are truly orthogonal the q_i\'s?'))
print(textQRClassic('Qc.T*Qc: \n'),np.dot(Qc.T,Qc))
print(textQRClassic('||Qc.T*Qc-I_3||: '),np.linalg.norm(np.dot(Qc.T,Qc)-np.eye(3))) # Warning: We are just using the transpose since the matrices are real!
print(textAnswer('Not really, since q2^T*q3 \\neq 0.'))

print(textQRModified('\n Qm.T*Qm: \n'),np.dot(Qm.T,Qm))
print(np.linalg.norm(np.dot(Qm.T,Qm)-np.eye(3)))
print(textQRModified('||Qm.T*Qm-I_3||: '),np.linalg.norm(np.dot(Qm.T,Qm)-np.eye(3)))
print(textAnswer('This looks much better!'))

################################
print(textBoldH('\n Do we recover A with each algorithm?'))
print(textQRClassic('A-Qc*Rc: \n'),A-np.dot(Qc,Rc))
print(textQRClassic('||A-Qc*Rc||: '),np.linalg.norm(A-np.dot(Qc,Rc)))
print(textAnswer('Yes!'))

print(textQRModified('\n A-Qm*Rm: \n'),A-np.dot(Qm,Rm))
print(textQRModified('||A-Qm*Rm||: '),np.linalg.norm(A-np.dot(Qm,Rm)))
print(textAnswer('Yes!'))
inputs: I am working on the file "Deep Learning.ai/Numpy - Understanding Backprop.ipynb".
The first task is:
Compute the loss
Can you write Python code for it?

targets: 
# Using cross - entropy loss
# Softmax loss = data loss + regularization loss

# Study about the Softmax loss and how it works before seeing how it works 
#in this case it could be intuitive
num_examples = X.shape[0]
# get unnormalized probabilities
exp_scores = np.exp(scores)
# normalize them for each example
'''We now have an array probs of size [300 x 3], where each row now contains the class
probabilities. In particular, since we’ve normalized them every row now sums to one. We 
can now query for the log probabilities assigned to the correct classes in each example:'''
probs = exp_scores / np.sum(exp_scores, axis=1 , keepdims=True)
# Calculating the correct_log probability 
'''The array correct_logprobs is a 1D array of just the probabilities assigned to the
correct classes for each example. The full loss is then the average of these log 
probabilities and the regularization loss:'''
correct_logprobs = -np.log(probs[range(num_examples),y])



# compute ths loss : average cross-entropy loss and regularization 
data_loss = np.sum(correct_logprobs)/ num_examples
reg_loss = 0.5*reg*np.sum(W*W)

'''∂Li∂fk=pk−1(yi=k)

Notice how elegant and simple this expression is. Suppose the probabilities we 
computed were p = [0.2, 0.3, 0.5], and that the correct class was the middle
one (with probability 0.3). According to this derivation the gradient on the 
scores would be df = [0.2, -0.7, 0.5]. Recalling what the interpretation of the 
gradient, we see that this result is highly intuitive: increasing the first or 
last element of the score vector f (the scores of the incorrect classes) leads to 
an increased loss (due to the positive signs +0.2 and +0.5) - and increasing the
loss is bad, as expected. However, increasing the score of the correct class has
negative influence on the loss. The gradient of -0.7 is telling us that increasing
the correct class score would lead to a decrease of the loss Li, which makes sense.'''
dscores = probs 
dscores[range(num_examples),y]-=1
dscores/=num_examples

dW =...
inputs: I am working on the file "Adicionais/derivada.ipynb".
The first task is:
Para avaliar a derivada em um ponto, por exemplo, para calcular $f'(1)$, digitamos:
Can you write Python code for it?

targets: 
diff(f(x),x).subs(x,1)
inputs: "Batch normalization and initialization
We will now run a small experiment to study the interaction of batch normalization and weight initialization.
The first cell will train 8-layer networks both with and without batch normalization using different scales for weight initialization. The second layer will plot training accuracy, validation set accuracy, and training loss as a function of the weight initialization scale."
Please write code following the instructions in jupyter notebook style.

targets: 
np.random.seed(231)
# Try training a very deep net with batchnorm
hidden_dims = [50, 50, 50, 50, 50, 50, 50]

num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

bn_solvers = {}
solvers = {}
weight_scales = np.logspace(-4, 0, num=20)
for i, weight_scale in enumerate(weight_scales):
  print('Running weight scale %d / %d' % (i + 1, len(weight_scales)))
  bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, use_batchnorm=True)
  model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, use_batchnorm=False)

  bn_solver = Solver(bn_model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
  bn_solver.train()
  bn_solvers[weight_scale] = bn_solver

  solver = Solver(model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
  solver.train()
  solvers[weight_scale] = solver

# Plot results of weight scale experiment
best_train_accs, bn_best_train_accs = [], []
best_val_accs, bn_best_val_accs = [], []
final_train_loss, bn_final_train_loss = [], []

for ws in weight_scales:
  best_train_accs.append(max(solvers[ws].train_acc_history))
  bn_best_train_accs.append(max(bn_solvers[ws].train_acc_history))
  
  best_val_accs.append(max(solvers[ws].val_acc_history))
  bn_best_val_accs.append(max(bn_solvers[ws].val_acc_history))
  
  final_train_loss.append(np.mean(solvers[ws].loss_history[-100:]))
  bn_final_train_loss.append(np.mean(bn_solvers[ws].loss_history[-100:]))
  
plt.subplot(3, 1, 1)
plt.title('Best val accuracy vs weight initialization scale')
plt.xlabel('Weight initialization...
inputs: I am working on the file "ModelingExamples2.ipynb".
The first task is:
Question
Are two given histograms drawn from the same distribution?
$[3,5,12,4]$
$[8, 14, 31, 14]$
Visualizing the Dirichlet Distribution
[http://blog.bogatron.net/blog/2014/02/02/visualizing-dirichlet-distributions/]
Can you write Python code for it?

targets: 
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.tri as tri
from functools import reduce
from scipy.special import gammaln

corners = np.array([[0, 0], [1, 0], [0.5, 0.75**0.5]])
triangle = tri.Triangulation(corners[:, 0], corners[:, 1])

refiner = tri.UniformTriRefiner(triangle)
trimesh = refiner.refine_triangulation(subdiv=4)


# Mid-points of triangle sides opposite of each corner
midpoints = [(corners[(i + 1) % 3] + corners[(i + 2) % 3]) / 2.0 \
             for i in range(3)]

def xy2bc(xy, tol=1.e-3):
    '''Converts 2D Cartesian coordinates to barycentric.'''
    s = [(corners[i] - midpoints[i]).dot(xy - midpoints[i]) / 0.75 \
         for i in range(3)]
    return np.clip(s, tol, 1.0 - tol)

class Dirichlet(object):
    def __init__(self, alpha):
        self._alpha = np.array(alpha)
        self._coef = gammaln(np.sum(self._alpha)) - np.sum(gammaln(self._alpha))
    def log_pdf(self, x):
        return self._coef + np.sum(np.log(x)*(self._alpha - 1)) 
    def pdf(self, x):
        '''Returns pdf value for `x`.'''
        return np.exp(self.log_pdf(x))
        
def draw_pdf_contours(dist, nlevels=200, subdiv=8, **kwargs):

    refiner = tri.UniformTriRefiner(triangle)
    trimesh = refiner.refine_triangulation(subdiv=subdiv)
    pvals = [dist.pdf(xy2bc(xy)) for xy in zip(trimesh.x, trimesh.y)]

    plt.tricontourf(trimesh, pvals, nlevels, **kwargs)
    plt.axis('equal')
    plt.xlim(0, 1)
    plt.ylim(0, 0.75**0.5)
    plt.axis('off')

draw_pdf_contours(Dirichlet([1, 3, 1]))

draw_pdf_contours(Dirichlet([1.99, 3.99, 10.99]))
inputs: "And we can extract one part of a list using slicing:"
Please write code following the instructions in jupyter notebook style.

targets: 
friend_list = ['John', 'Bob', 'Marry']
list_with_fewer_friends = friend_list[:2]
print(list_with_fewer_friends)
inputs: %%bigquery
CREATE OR REPLACE MODEL
    babyweight.model_3

TRANSFORM(
    weight_pounds,
    is_male,
    mother_age,
    plurality,
    gestation_weeks,
    ML.FEATURE_CROSS(
        STRUCT(
            is_male,
            ML.BUCKETIZE(
                mother_age,
                GENERATE_ARRAY(15, 45, 1)
            ) AS bucketed_mothers_age,
            plurality,
            ML.BUCKETIZE(
                gestation_weeks,
                GENERATE_ARRAY(17, 47, 1)
            ) AS bucketed_gestation_weeks
        )
    ) AS crossed
)

OPTIONS (
    MODEL_TYPE="LINEAR_REG",
    INPUT_LABEL_COLS=["weight_pounds"],
    L2_REG=0.1,
    DATA_SPLIT_METHOD="NO_SPLIT") AS

SELECT
    *
FROM
    babyweight.babyweight_data_train
Given the above code, generate some markdown instructions for it.

targets: 
Model 3:  Apply the TRANSFORM clause
Before we perform our prediction, we should encapsulate the entire feature set in a TRANSFORM clause. This way we can have the same transformations applied for training and prediction without modifying the queries.
Let's apply the TRANSFORM clause to the model_3 and run the query.
inputs: # %load _solutions/case4_air_quality_analysis34.py

# %load _solutions/case4_air_quality_analysis35.py
Given the above code, generate some markdown instructions for it.

targets: 
<div class="alert alert-success">

__EXERCISE 15__

The maximum daily, 8 hour mean, should be below 100 µg/m³. What is the number of exceedances of this limit for each year/station?

<details><summary>Hints</summary>

- Have a look at the `rolling` method to perform moving window operations.

</details>

<br>_Note:_
This is not an actual limit for NO$_2$, but a nice exercise to introduce the `rolling` method. Other pollutans, such as 0$_3$ have actually such kind of limit values based on 8-hour means.

</div>
inputs: "4. Instantiate and Estimator with the Required Parameters"
Please write code following the instructions in jupyter notebook style.

targets: 
CHECKPOINT_STEPS=1000

hparams  = tf.contrib.training.HParams(
    training_steps = 10000,
    periodicities = [200],
    input_window_size = 40,
    output_window_size=10,
    batch_size = 15,
    loss = tf.contrib.timeseries.ARModel.NORMAL_LIKELIHOOD_LOSS, # NORMAL_LIKELIHOOD_LOSS | SQUARED_LOSS
    hidden_units = None,
    learning_rate = 0.1
    
)


model_dir = 'trained_models/{}'.format(MODEL_NAME)

run_config = tf.estimator.RunConfig().replace(
    save_checkpoints_steps=CHECKPOINT_STEPS,
    tf_random_seed=19830610,
    model_dir=model_dir
)
                                             
print("Model directory: {}".format(run_config.model_dir))
print("Hyper-parameters: {}".format(hparams))
print("")

train_input_fn = generate_input_fn(
    file_name=TRAIN_DATA_FILE,
    mode = tf.estimator.ModeKeys.TRAIN,
    batch_size=hparams.batch_size,
    windows_size = hparams.input_window_size + hparams.output_window_size
)

estimator = create_estimator(run_config, hparams)
inputs: # Matching structure (example): 
# bnetza_id uba_id
# 1         1
# 2         1
# 3         1
# 4         2
# 5         2

# Get relevant entries from the matchinglist and merge the corresponding 
# UBA Data to the list.
matchnt1= matchinglist[
    (matchinglist.duplicated(subset=['uba_id_string'], keep=False) == True)
    & (matchinglist.duplicated(subset=['ID BNetzA'], keep=False)== False)]
matchnt1 = pd.merge(matchnt1, plantlist_uba,
                    left_on='uba_id_string', right_on='uba_id_string', how='left')
matchnt1 = matchnt1.set_index('ID BNetzA')

# Import BNetzA Capacities and CHP criterion into matchnt1 dataframe
plantlist_capacities = pd.DataFrame(plantlist[['capacity', 'chp']]).rename(
    columns={'capacity': 'capacity_bnetza', 'chp': 'chp_bnetza'})
matchnt1 = pd.merge(matchnt1, plantlist_capacities,
                    left_index=True, right_index=True, how='left')

# Get sum of BNetzA Capacitites for each UBA Index and merge into matchnt1 dataframe
plantlist_uba_capacitysum = pd.DataFrame(
    matchnt1.groupby('uba_id_string').sum()['capacity_bnetza']).rename(
        columns={'capacity_bnetza': 'capacity_bnetza_aggregate'})
matchnt1 = pd.merge(matchnt1, plantlist_uba_capacitysum,
                    left_on='uba_id_string', right_index=True, how='left')

# Scale UBA Capacities based BNetzA Data
matchnt1['uba_capacity_scaled'] = (matchnt1['uba_capacity']
                                   * matchnt1['capacity_bnetza']
                                   / matchnt1['capacity_bnetza_aggregate'])

# determine sum of capacities with chp capability and add to matchnt1
plantlist_uba_chp_capacities = matchnt1[(matchnt1['chp_bnetza'] == 'yes')]
plantlist_uba_chp_capacitysum = pd.DataFrame(
    plantlist_uba_chp_capacities.groupby('uba_id_string')
    .sum()['capacity_bnetza']) 
plantlist_uba_chp_capacitysum = plantlist_uba_chp_capacitysum.rename(
    columns={'capacity_bnetza': 'capacity_bnetza_with_chp'})

matchnt1 = pd.merge(matchnt1, plantlist_uba_chp_capacitysum,
                   ...
Given the above code, generate some markdown instructions for it.

targets: 
7.2.2 case n-1
Match multiple BNetza IDs to one UBA ID
inputs: I am working on the file "Daily/20150902_phoenix_bol_corrs.ipynb".
The first task is:
For reference, at [Fe/H] = $-0.5$ dex, we have
Teff      logg   [Fe/H]  [a/Fe]         B           V           R           I
3000.00    5.00   -0.50    0.20        -6.533      -5.496      -4.424      -3.154
The interpolation routine has seemingly handled the non-monotonic nature of the metallicity column, as all interpolate values lie between values at the two respective nodes.

Now let's import an isochrone and calcuate colors for stellar models for comparison against MARCS bolometric corrections.
Can you write Python code for it?

targets: 
iso = np.genfromtxt('/Users/grefe950/evolve/dmestar/iso/dmestar_00120.0myr_z+0.00_a+0.00_marcs.iso')
inputs: *5.3 Write a function that takes three parameters. The third is a callable (function) that takes two parameters. The function should call its third parameter with the first two as parameters. If the third parameter is not specified the function should add the two arguments.
This is very similar to the previous function but the third parameter is a function instead of a string.

targets: # TODO def call_func(...)

def product(x, y):
    return x * y

assert(call_func(3, 4, product) == 12)
assert(call_func("foo", "bar") == "foobar")
inputs: # Create a random forest classifier
clf = RandomForestClassifier(n_estimators=10000, random_state=0, n_jobs=-1)

# Train the classifier
clf.fit(X_train, y_train)

# Print the name and gini importance of each feature
for feature in zip(feat_labels, clf.feature_importances_):
    print(feature)
Given the above code, generate some markdown instructions for it.

targets: 
Train A Random Forest Classifier
inputs: Check how the annotators did on internal gold questions

targets: # Starting from batch three we have included internal gold questions for toxicity and constructiveness (20 each) with 
# internal_gold_constructiveness flag True. In the code below, we examine to what extent the annotators agreed with 
# these internal gold questions. 

# Get a subset dataframe with internal gold questions for constructiveness
internal_gold_con_df = df[df['constructive_internal_gold'].notnull()].copy()

# Call secret_gold_evaluation_constructiveness function from data_quality_analysis_functions
print('Disagreement on constructiveness secret gold questions (%): ', secret_gold_evaluation_constructiveness(internal_gold_con_df))

# Get a subset dataframe with internal gold questions for toxicity
internal_gold_tox_df = df[df['crowd_toxicity_level_internal_gold'].notnull()].copy()

# Call secret_gold_evaluation_toxicity function from data_quality_analysis_functions
print('Disagreement on toxicity secret gold questions (%): ', secret_gold_evaluation_toxicity(internal_gold_tox_df))
inputs: Create Data Frame

targets: # Create feature matrix
X = np.array([[1, 2], 
              [6, 3], 
              [8, 4], 
              [9, 5], 
              [np.nan, 4]])
inputs: I am working on the file "BAMM.101x/Web_scraping.ipynb".
The first task is:
<h3>get the value of the login token</h3>
Can you write Python code for it?

targets: 
def get_login_token(response):
    soup = BeautifulSoup(response.text, 'lxml')
    token = soup.find('input',{'name':"wpLoginToken"}).get('value')
    return token
inputs: As this plot (created entirely without MCMC!) shows, the MCMC method did well, but our estimate of the true parameters has become biased by the stochastic noise! You can test this by increasing the number of sample points, which increases the size of the noise sample, and reduces the bias.
Finally, we can look at the bit that really matters: The model predictions made from models with the parameters we found (a posterior predictive check). Thes can be plotted using the series() method.

targets: fig, axes = pints.plot.series(chain, problem)

# Customise the plot, and add the original, noise-free data
fig.set_size_inches(12,4.5)
plt.plot(times, org_values, c='orange', label='Noise-free data')
plt.legend()
plt.show()
inputs: I am working on the file "5-functions.ipynb".
The first task is:
this is handy: 
* if we want a function to work one way, but occasionally need it to do something else, we can allow people to pass a diff parameter when they need to but provide a default

below example shows how python matches values to parameters:
Can you write Python code for it?

targets: 
def display(a=1, b=2, c=3):
    print('a:', a, 'b:', b, 'c:', c)

print('no parameters:')
display()
print('one parameter:')
display(55)
print('two parameters:')
display(55, 66)
inputs: ## all Nans to white space
x = x.replace(np.nan, ' ', regex = True)
x = x.replace(np.nan, 'NaN', regex = True)
## convert to all floats
with warnings.catch_warnings():
    warnings.simplefilter("ignore")
    x = x.convert_objects(convert_numeric = True)
x = x.interpolate()
x.isnull().any().any() ## to check if any missing data

Given the above code, generate some markdown instructions for it.

targets: 
Or imputating missing values with the interpolate function from pandas.
inputs: ""
Please write code following the instructions in jupyter notebook style.

targets: 
gbr = GradientBoostingRegressor(loss='ls', learning_rate=0.1, n_estimators=900,
                                subsample=1.0, min_samples_split=2, min_samples_leaf=1, max_depth=4, init=None,
                                random_state=None, max_features=None, alpha=0.5,
                                verbose=0, max_leaf_nodes=None, warm_start=False)

%%time
gbr = gbr.fit(X_train,y_train)
#os.system('say "終わりだ"') #its over!

#parameters = {'max_depth':range(2,5,1),'alpha':[0.5,0.6,0.7,0.8,0.9]}
#parameters = {'subsample':[0.2,0.4,0.5,0.6,0.8,1]}
#parameters = {'subsample':[0.2,0.5,0.6,0.8,1],'n_estimators':[800,1000,1200]}
#parameters = {'max_depth':range(2,4,1)}
parameters = {'n_estimators':[400,800,1100]}
#parameters = {'loss':['ls', 'lad', 'huber', 'quantile'],'alpha':[0.3,0.5,0.8,0.9]}
#parameters = {'learning_rate':[0.1,0.5,0.9]}



grid_gbr = grid_search.GridSearchCV(gbr, parameters,n_jobs=2,scoring=manualScorer)

%%time
grid_gbr = grid_gbr.fit(X_train,y_train)

print(grid_gbr.grid_scores_)
print("Best: ",grid_gbr.best_params_)

print(gbr.score(X_train,y_train))
print(gbr.score(X_test,y_test))
inputs: Below you can see the fields loaded and a count of the values per field;

targets: df.count()
inputs: 2.3- Line Profiling: line_profiler
Python's native cProfile module and the corresponding %prun magic break down the execution time of code function by function. Sometimes, we may need an even more fine- grained analysis of code performance with a line-by-line report.
To profile code line-by-line, we need an external Python module named line_profiler. To install it run one of these:
* conda install line_profiler
* pip install line_profiler
Once installed import the line_profiler IPython extension module that comes with the package:

targets: %load_ext line_profiler
inputs: "2. Import the independent time series
Two explanatory series are used: the precipitation and the potential evaporation. These need to be pandas Series objects, as for the observed heads.
Important characteristics of these time series are:
- All series are stored as pandas Series objects.
- The series may have irregular time intervals, but then it will be converted to regular time intervals when creating the time series model later on.
- It is preferred to use the same length units as for the observed heads."
Please write code following the instructions in jupyter notebook style.

targets: 
# Import observed precipitation series
precip = pd.read_csv('../data/rain_nb1.csv', parse_dates=['date'],
                     index_col='date', squeeze=True)
print('The data type of the precip series is: %s' % type(precip))

# Import observed evaporation series
evap = pd.read_csv('../data/evap_nb1.csv', parse_dates=['date'],
                   index_col='date', squeeze=True)
print('The data type of the evap series is: %s' % type(evap))

# Calculate the recharge to the groundwater
recharge = precip - evap
print('The data type of the recharge series is: %s' % type(recharge))

# Plot the time series of the precipitation and evaporation
plt.figure()
recharge.plot(label='Recharge', figsize=(10, 4))
plt.xlabel('Time [years]')
plt.ylabel('Recharge (m/year)');