Muennighoff/xP3x-sample huggingface dataset:
inputs: I am working on the file "2015-10_Lecture/Lecture2/code/1_Intro_Theano.ipynb".
The first task is:
The graph fo z:
<img src="files/pics/z_graph.png">
The graph for f:
<img src="files/pics/f_graph.png">
Simple matrix multiplications
The following types for input variables are typically used:
byte: bscalar, bvector, bmatrix, btensor3, btensor4
16-bit integers: wscalar, wvector, wmatrix, wtensor3, wtensor4
32-bit integers: iscalar, ivector, imatrix, itensor3, itensor4
64-bit integers: lscalar, lvector, lmatrix, ltensor3, ltensor4
float: fscalar, fvector, fmatrix, ftensor3, ftensor4
double: dscalar, dvector, dmatrix, dtensor3, dtensor4
complex: cscalar, cvector, cmatrix, ctensor3, ctensor4

scalar: One element (one number)
vector: 1-dimension
matrix: 2-dimensions
tensor3: 3-dimensions
tensor4: 4-dimensions
As we do not need perfect precision we use mainly float instead of double. Most GPUs are also not able to handle doubles.
So in practice you need: iscalar, ivector, imatrix and fscalar, fvector, vmatrix.
Task: Implement the function $$f(x,W,b) = \tanh(xW+b)$$ with $x \in \mathbb{R}^n, b \in \mathbb{R}^k, W \in \mathbb{R}^{n \times k}$.
$n$ input dimension and $k$ output dimension
Can you write Python code for it?

targets: 
import theano
import theano.tensor as T
import numpy as np

# Put your code here
inputs: Bag of Words (BoW) language model
Today we'll see our first, admittedly primitive, computational model of language called "Bag of Words". This model was very popular in early text analysis, and continues to be used today. In fact, the models that have replaced it are still very difficult to actually interpret, giving the BoW approach a slight advantage if we want to understand why the model makes certain decisions.
Getting into the model we'll have to revisit Term Frequency (think Counter). We'll then see the Document-Term Matrix (DTM), which we've discusssed briefly before. We'll have to normalize these counts if we want to compare. Then we'll look at the available Python libraries to streamline this process.
Once we have our BoW model we can analyze it in a high-dimensional vector space, which gives us more insights into the similarities and clustering of different texts. We'll then get into Piper's analysis.
We'll build our model  from scratch with numpy and datascience libraries before we get into the higher level libraries:

targets: %matplotlib inline
import numpy as np
from datascience import *
inputs: Creación de nuevas columnas

targets: # Agruparemos por año y día: creemos dos columnas nuevas
data['year'] = data.index.year
data['month'] = data.index.month
inputs: After cleaning the data we finished with four dataframes: Rent, Rent_Private, Crime and Crime_Total. We will use these three to produce visual aids to better understand how they relate.
3. Visualizing the Data
3.1 Housing Data
In here we will just look into the housing data, and see how different are the prices in the different boroughs and different neighborhoods of Manhattan

targets: # Import seaborn package

import seaborn as sns
inputs: "Changes:

explicit check for input shape
no views and simple sequential structure, output is just nn.Sequential, so can always be saved/passed/etc
no need in AvgPool and additional views, this place is much clearer now
make_layer doesn't use internal state (that's quite faulty place)"
Please write code following the instructions in jupyter notebook style.

targets: 
from torchvision.models.resnet import BasicBlock, Bottleneck, ResNet

x = torch.randn(2, 3, 224, 224)
with torch.no_grad():
    model_old = ResNetOld(BasicBlock, layers=[2, 2, 2, 3])
    model_new = ResNetNew(BasicBlock, layers=[2, 2, 2, 3])
    initialize(model_old)
    initialize(model_new)
    assert torch.allclose(model_old(x), model_new(x), atol=1e-3)

# with torch.no_grad():
#     x = torch.randn([2, 512, 7, 7])
#     torch.allclose(nn.AvgPool2d(7)(x), reduce(x, 'b c h w -> b c', 'mean'), atol=1e-8)
inputs: I am working on the file "20151120-loggins-data/20151120-loggins-data.ipynb".
The first task is:
And now a fact table:
Can you write Python code for it?

targets: 
%%sql
DROP TABLE IF EXISTS f_entrance;
CREATE TABLE f_entrance (
  entrance_id SERIAL PRIMARY KEY,
  day_key SMALLINT NOT NULL,
  hour SMALLINT NOT NULL,
  affiliation VARCHAR(255),
  subaffiliation VARCHAR(255),
  count INT NOT NULL
)
inputs: I am working on the file "Cómo acelerar tu código Python con numba.ipynb".
The first task is:
Creamos nuestro array de datos:
Can you write Python code for it?

targets: 
data = np.random.randn(2000, 2000)
inputs: I am working on the file "Introductory statistical data analysis with pandas PyData Berlin 2017.ipynb".
The first task is:
the relationship between chest pain and the diagnosis holds for both levels of the sex variables, hence it is not a 
moderator.
we would test for moderator variables in the case of a quantitative response variable the same way:
1. divide the population into the sublevels of the third variables
2. conduct an smf.ols test for each to see if the relationship is statistically significant for each level
identifying a confounding variable does not allow to establish causation, just to get closer to a causal connection.
due to infinite number of possible lurking variables, observational studies cannot rly establish causation
a lurking of confounding variable is a third variable that is associated with both the explanatory and response 
variables.
i.e. x=firefighters; y=damage caused by a fire. plot would suggest more firefighters causes more fire damage.
in reality there is a third confounding variable that influences both, seriousness of the fire.
In a study we want to demonstrate that our statistical relationship is valid even after controlling for confounders.
now we will test whether there is a relationship between two quantitative variables, age and cholesterol
for this we use the pearson correlation test
r, going from -1 to 1 only tells us whether the two variables are linearly related. they may be related in nonlinear ways
therefore it's always important to look at r in parallel with a scatterplot of the two variables
r squared is a measure of how much variability in one variable can be explained by the other variable
to calculate the pearson coefficient we need to remove all missing values
Please remember that when two variables are correlated it is possible that:
X causes Y or Y causes X
Z causes both X and Y
X and Y are correlated by chance - a spurious correlation
Can you write Python code for it?

targets: 
df_clean_copy.columns

scat1 = sns.regplot(x='age', y = 'cholesterol', fit_reg=True, data = df_clean_copy)
plt.xlabel('Age')
plt.ylabel('Cholesterol')
plt.suptitle('Relationship between age and cholesterol')
scat1

#the r coefficient is a measure of association, of how closely points are clustered around a line
#correlations are always between -1 and 1
#r measures solely linear association
#association, not causation
#it is a number without units
#it can mislead in presence of outliers or non linear association -> always draw a scatter plot as well and check visually
#when you look at a scatter plot you look at direction form and strength of relationship
#if you identify a nonlinear association, with one or more curves, then you know you ought to add nonlinear explanatory
#variables to your regression model, to aim to capture this nonlinear association as well.
#ecological correlations based on averages can be misleading and overstate strength of associations for individual 
#units
print('association between age and cholesterol')
print(scipy.stats.pearsonr(df_clean_copy['age'], df_clean_copy['cholesterol']))
inputs: I am working on the file "CatsVSDogs/Atelier-keras-CatsVSDogs.ipynb".
The first task is:
Gèle des 4 premiers blocs de convolution
En pratique, et pour pouvoir effectuer ces calculs dans un temps raisonable, nous allons "fine-tuner" seulement le dernier bloc de convolution du modèle, le bloc 5 (couches 16 à 19 dans le summary du modèle précédent) ainsi que le bloc de réseau de neurones que nous avons ajoutés. 
Pour cela on va "geler" (Freeze) les 15 premières couches du modèle pour que leur paramètre ne soit pas optimiser pendant la phase d'apprentissage.
Can you write Python code for it?

targets: 
for layer in model_VGG_LastConv_fcm.layers[:15]:
    layer.trainable = False
inputs: Create a PasqalDevice
Multi-qubit gates can be applied between qubits in the device, provided that the distance between them is smaller than the so-called Rydberg blocade radius (or control radius), that can be passed as a parameter of the device (in units of the lattice size). Here, we instantiate a PasqalVirtualDevice with 36 qubits and a control radius of $2.1$. 
This PasqalVirtualDevice can be used to validate the operations within a Circuit.

targets: from cirq_pasqal.pasqal_device import PasqalConverter

# Initialize and create a circuit
initial_circuit = cirq.Circuit()
initial_circuit.append(cirq.CZ(p_qubits[0], p_qubits[1]))
initial_circuit.append(cirq.Z(p_qubits[0]))
initial_circuit.append(cirq.CX(p_qubits[0], p_qubits[2]))

# Create a Pasqal device with a control radius of 2.1 (in units of the lattice spacing)
p_device=PasqalVirtualDevice(control_radius=2.1, qubits=p_qubits)

# Validate the circuit using the device
try:
    p_device.validate_circuit(initial_circuit)
except ValueError as e:
    # Uh oh!  This circuit does not pass validation.
    print(e)
inputs: # Explore relative difference of different bands
wav_, flux_ = load_aces_spectrum([3900, 4.5, 0, 0])
wav, flux = wav_selector(wav_, flux_, 0.7, 2.5)

table = []
table.append("Band, Cond#1, Split,   Masked,  ratio,   Cond#1,   Split,  Masked, ratio")
table.append("Grad,              False             ,                True              ")

# Get J band SNR normalization value
wav_j, flux_j = convolve_and_resample(
    wav, flux, vsini=1, R=100000, band="J", sampling=3
)
snr_norm = snr_constant_band(wav_j, flux_j, snr=100, band="J")

for band in ["Z", "Y", "J", "H", "K"]:
    atm = Atmosphere.from_band(band, bary=True)

    w, f = convolve_and_resample(wav, flux, vsini=1, R=100000, band=band, sampling=3)
    f /= snr_norm

    atm = atm.at(w)

    a = RVprec_calc_masked(w, f, atm.mask, grad=True)
    b = RVprec_calc_masked(w, f, atm.mask, grad=False)
    c = rv_precision(w, f, atm.mask, grad=True)
    d = rv_precision(w, f, atm.mask, grad=False)
    e = rv_precision(w, f, grad=True)
    f = rv_precision(w, f, grad=False)

    false_ratio = (d - b) / b
    true_ratio = (c - a) / a
    table.append(
        "{0:5}, {1:4.02f}, {2:6.02f}, {3:6.02f}, {4:5.04f}, {5:6.02f}, {6:6.02f}, {7:6.02f}, {8:5.04f}".format(
            band,
            f.value,
            b.value,
            d.value,
            false_ratio,
            e.value,
            a.value,
            c.value,
            true_ratio,
        )
    )

for line in table:
    print(line)
Given the above code, generate some markdown instructions for it.

targets: 
Differences between versions with same gradient is at the 4th sf. These are not the correct scale, this will be addressed in the next section.
Calculating difference ratios.
Assessing the changes to actual Figueira et al. values by not splitting on telluric lines first then applying mask to the weights, or slitting then calculating weights.
Convolving the spectra to vsini=1 an R=100k of a M0 spectra in the Z,Y,J,H,K bands to be consistent with paper. 
Using Old gradient and new gradient methods to access difference.
The old gradient drops the last pixel, which drops many pixels when spectra is split between telluric lines.
inputs: I am working on the file "LongRange/PolynomialFit.ipynb".
The first task is:
Define a quality-of-fit measure, $\chi^2$
Can you write Python code for it?

targets: 
eq1 = Eq(chi2, Integral(w(r)*(f(r)-f_approx(r,ak))**2,r))
eq1
inputs: "A helper function is defined to generate samples:"
Please write code following the instructions in jupyter notebook style.

targets: 
    def gen_samples(n_samples):
        X_all = np.zeros((2, 0))
        Y_all = np.zeros(0)
    
        for i, model in enumerate(models):
            Y = np.zeros(n_samples) + i+1
            X = np.array(model['sigma']).dot(np.random.randn(2, n_samples)) + np.array(model['mu']).reshape((2,1))
    
            X_all = np.hstack((X_all, X))
            Y_all = np.hstack((Y_all, Y))
    
        return (X_all, Y_all)
inputs: "Clip an image by a region of interest
In case you want to display an image over a given region (and not outside), we can clip our dataset using the region as an argument of the clip() method. Let's say that we want to display the ground elevation in France. We can get the geometry of the administrative boundary of France with the FAO feature collection and do the same as before:"
Please write code following the instructions in jupyter notebook style.

targets: 
# Get a feature collection of administrative boundaries.
countries = ee.FeatureCollection('FAO/GAUL/2015/level0').select('ADM0_NAME')

# Filter the feature collection to subset France.
france = countries.filter(ee.Filter.eq('ADM0_NAME', 'France'))

# Clip the image by France.
elv_fr = elv_img.clip(france)

# Create the URL associated with the styled image data.
url = elv_fr.getThumbUrl({
    'min': 0, 'max': 2500, 'region': roi, 'dimensions': 512,
    'palette': ['006633', 'E5FFCC', '662A00', 'D8D8D8', 'F5F5F5']})

# Display a thumbnail of elevation in France.
Image(url=url)
inputs: # should print 0.425558846691
print lasso_coordinate_descent_step(1, np.array([[3./np.sqrt(13),1./np.sqrt(10)],[2./np.sqrt(13),3./np.sqrt(10)]]), 
                                   np.array([1., 1.]), np.array([1., 4.]), 0.1)
Given the above code, generate some markdown instructions for it.

targets: 
To test the function, run the following cell:
inputs: # Set-up E-step.

step_size = tf.get_variable(
    'step_size',
    initializer=np.array(0.2, dtype=dtype),
    trainable=False)

hmc = tfp.mcmc.HamiltonianMonteCarlo(
    target_log_prob_fn=unnormalized_posterior_log_prob,
    num_leapfrog_steps=2,
    step_size=step_size,
    step_size_update_fn=tfp.mcmc.make_simple_step_size_update_policy(
      num_adaptation_steps=None),
    state_gradients_are_stopped=True)

init_random_weights = tf.placeholder(dtype, shape=[len(log_county_uranium_ppm)])

posterior_random_weights, kernel_results = tfp.mcmc.sample_chain(
    num_results=3,
    num_burnin_steps=0,
    num_steps_between_results=0,
    current_state=init_random_weights,
    kernel=hmc)
Given the above code, generate some markdown instructions for it.

targets: 
これで、HMC 遷移カーネルを作成して E ステップのセットアップを完了しました。
注意:


state_stop_gradient=Trueを使用して、M ステップが  MCMC からの抽出を介してバックプロパゲーションするのを防ぎます。(E ステップでは前の最もよく知られている推定量で意図的にパラメータ化されているため、バックプロパゲーションを行う必要はありません。)


tf.placeholder を使用して、最終的に TF グラフを実行するときに、前のイテレーションのランダム MCMC サンプルを次のイテレーションのチェーンの値としてフィードできるようにします。


TFP の適応型 step_size ヒューリスティック、tfp.mcmc.hmc_step_size_update_fn を使用します。
inputs: # Your code goes here
Given the above code, generate some markdown instructions for it.

targets: 
Split the data into train/test
For this exercise we'll split the data frame so that 20% of the data is held out for testing, and the remaining data is used for training. Write code to split the data into two dataframes: one for testing and one for training.
inputs: %%sql macro initialize
var $hello Hello There 
var hello You won't see this

%%sql macro runit
echo The value of hello is *{hello}*
echo {$hello}
Given the above code, generate some markdown instructions for it.

targets: 
Var Command
The var (variable) command sets a macro variable to a value. A variable is referred to in the macro script using curly braces {name}. By default the arguments that are used in the macro call are assigned the variable names {1} to {9}. If you use a named argument (option="value") in the macro call, a variable called {option} will contain the value within the macro.
To set a variable within a macro you would use the var command:
var name value
The variable name can be any name as long as it only includes letters, numbers, underscore _ and $. Variable names are case sensitive so {a} and {A} are different. When the macro finishes executing, the contents of the variables will be lost. If you do want to keep a variable between macros, you should start the name of the variable with a $ sign:
var $name value
This variable will persist between macro calls.
inputs: "As we have discussed that the linear regression model basically finds the best value for the intercept and slope, which results in a line that best fits the data. To see the value of the intercept and slop calculated by the linear regression algorithm for our dataset, execute the following code."
Please write code following the instructions in jupyter notebook style.

targets: 
#To retrieve the intercept:
print(regressor.intercept_)

#For retrieving the slope:
print(regressor.coef_)
inputs: # change these to try this notebook out
# In "production", these will be replaced by the parameters passed to papermill
BUCKET = 'cloud-training-demos-ml'
PROJECT = 'cloud-training-demos'
REGION = 'us-central1'
DEVELOP_MODE = True
NBUCKETS = 5 # for embeddings
NUM_EXAMPLES = 1000*1000 # assume 1 million examples
TRAIN_BATCH_SIZE = 64
DNN_HIDDEN_UNITS = '64,32'

import os
os.environ['BUCKET'] = BUCKET
os.environ['PROJECT'] = PROJECT
os.environ['REGION'] = REGION

%%bash
gcloud config set project $PROJECT
gcloud config set compute/region $REGION
Given the above code, generate some markdown instructions for it.

targets: 
Setup
inputs: Data Preparation

targets: from keras import backend as K
inputs: "<a id="cell4"></a>
4. Adding Macroeconomic Indicators
As a an indicator of economic state at the time a loan is awarded, we will use S&P 1500 composite index.
Source: http://www.econ.yale.edu/~shiller/data.htm"
Please write code following the instructions in jupyter notebook style.

targets: 
# Load monthly consumer price index and composite S&P index data
cpi_sp = pd.read_excel('ie_data.xls', sep = ' ', skiprows=7)
# Use only values starting in 1990
cpi_sp  = cpi_sp[cpi_sp['Date']>1989]

# Keep only the relevant columns: date (year+month), S&P 1500 index and Consumer Price Index (CPI)
cpi_sp = cpi_sp[['Date', 'P', 'CPI']]

# Rename the columns
cpi_sp.columns = ['Date', 'SP', 'CPI']
cpi_sp.head()

# Change the format of the date from year.month to datetime (and deal with issues of .10 getting shortented to .1)
def split_date(x):
    tmp = str(x).split('.')
    if tmp[1] == '1':
        tmp[1] = '10'
    return tmp
cpi_sp['Date_dt'] = cpi_sp['Date'].apply(split_date)
s = '-'
cpi_sp['Date_dt'] = cpi_sp['Date_dt'].apply(lambda x: s.join(x))
cpi_sp['Date_dt'] = cpi_sp['Date_dt'].apply(lambda x: x + '-01')
cpi_sp.reset_index()
cpi_sp.head(10)

cpi_sp.drop('Date', axis = 1, inplace = True)
# Rename the columns
cpi_sp.columns = ['SP', 'CPI', 'Date']

# Adjust S&P based on CPI, relative to January 2016 US Dollar value
cpi2016 = float(cpi_sp[cpi_sp['Date'] == '2016-01-01']['CPI'])
cpi_sp['CPI_to2016'] = cpi2016/cpi_sp['CPI']
cpi_sp['SP_to2016'] = cpi_sp['SP'].multiply(cpi_sp['CPI_to2016'])
cpi_sp.drop(['CPI', 'SP'], axis = 1, inplace = True)
cpi_sp.reset_index(inplace = True, drop = True)
cpi_sp.head()

# Plot monhtly S&P (adjusted to 2016)
fig = plt.figure(figsize = (10,5))
axes = fig.add_subplot(1, 1, 1, facecolor ='gainsboro')
axes.plot(pd.to_datetime(cpi_sp['Date']), cpi_sp['SP_to2016'], c = 'k', linewidth = 3)
plt.xlabel('Year', fontsize = 20)
plt.ylabel('S&P 1500', fontsize = 20)
plt.xticks(fontsize = 20)
plt.yticks(fontsize = 20)
plt.tight_layout()
plt.show()

fig.savefig('1_SP1500.png', dpi = 300)

# Use S&P value relative to loan's approval date
cpi_sp.columns = [['ApprovalDate', 'CPI_to2016', 'SP_to2016']]

# Merge loan data with CPI + S&P data 
# Keep only the date and set it to first of the month in order to use CPI and SP data
loans_7a['ApprovalDate'] = loans_7a['ApprovalDate'].apply(lambda...
inputs: Concept
A network, more technically known as a graph, is comprised of:

a set of nodes
joined by a set of edges

They can be represented as two lists:

A node list: a list of 2-tuples where the first element of each tuple is the representation of the node, and the second element is a dictionary of metadata associated with the node.
An edge list: a list of 3-tuples where the first two elements are the nodes that are connected together, and the third element is a dictionary of metadata associated with the edge.

Since this is a social network of people, there'll be attributes for each individual, such as age, and sex. We can grab that data off from the attributes that are stored with each node by adding the data = True argument. Let's get a list of nodes with their attributes.

targets: # networkx will return a list of tuples in the form (node_id, attribute_dictionary)
print(G.nodes(data = True)[:5])

# excercise: Count how many males and females are represented in the graph
from collections import Counter
sex = [d['sex'] for _, d in G.nodes(data = True)]
Counter(sex)
inputs: I am working on the file "Programming Assignment 1/Predicting sentiment from product reviews with pandas.ipynb".
The first task is:
14.
Now, let us repeat this exercise to find the "most negative reviews." Use the prediction probabilities to find the 20 reviews in the test_data with the lowest probability of being classified as a positive review. Repeat the same steps above but make sure you sort in the opposite order.
Quiz Question:
Which of the following products are represented in the 20 most negative reviews?
Can you write Python code for it?

targets: 
negative_idx = np.argsort(test_scores)[:20]
print negative_idx
print test_scores[negative_idx[0]]
test_data.iloc[negative_idx]
inputs: I am working on the file "11/Homework11-Devulapalli.ipynb".
The first task is:
Looks like, the month of August is when the Police seem to have gone on a fining spree. 
15. Make a graph of the amount of revenue collected per day.
<< Not too sure on how to get this done>> 
16. Manually construct a dataframe out of https://dmv.ny.gov/statistic/2015licinforce-web.pdf (only NYC boroughts - bronx, queens, manhattan, staten island, brooklyn), having columns for borough name, abbreviation, and number of licensed drivers.
Can you write Python code for it?

targets: 
driversdf=pd.read_csv("drivers.csv")

driversdf.head(6)
inputs: 1000 most common words in the training set

targets: #1000 most common words in the training set instead
top_1000 = sorted(train_DF.items(), key=lambda x: x[1],reverse=True)[:1000]
top_1000_index = [dict_index[i[0]] for i in top_1000]
X_1000 = X_transp[top_1000_index] 

Z = linkage(X_1000.toarray(), method='ward' )
plt.figure()
plt.title('1000 most common words dendrograms for ward linkage type')
dn = dendrogram(Z,p=40,truncate_mode='lastp')

Z = linkage(X_1000.toarray(), method='complete' )
plt.figure()
plt.title('1000 most common words dendrograms for complete linkage type')
dn = dendrogram(Z,p=40,truncate_mode='lastp')

Z = linkage(X_1000.toarray(), method='average' )
plt.figure()
plt.title('1000 most common words dendrograms for average linkage type')
dn = dendrogram(Z,p=40,truncate_mode='lastp')
inputs: I am working on the file "FAI_old/wordvectors_CodeAlong.ipynb".
The first task is:
This is how you can look up a word vector.
Can you write Python code for it?

targets: 
def w2v(w): return vecs[wirdidx[w]] # mispelled 'wordidx' up above... lol

w2v('of')
inputs: I am working on the file "Foundations/Python CS/Activity 11.ipynb".
The first task is:
Plot the results:
Can you write Python code for it?

targets: 
# Plot and label the time taken
plt.loglog(N, times, marker='o', label='Recursive factorial')
plt.xlabel('$n$')
plt.ylabel('$t$ (s)')

# Show a reference line of O(n)
plt.loglog(N, 1e-6*np.array(N), label='$O(n)$')

# Add legend
plt.legend(loc=0);
inputs: Constants / Parameters

targets: # PARAMETERS (might be overridden by a calling script)

# if not calling from another script (batch), SUBNOTEBOOK_FLAG might not be defined
try:
    SUBNOTEBOOK_FLAG
except NameError:
    SUBNOTEBOOK_FLAG = False
    
# Not calling as a sub-script? define params here
if not SUBNOTEBOOK_FLAG:
    
    # SET PARAMETER VARIABLES HERE UNLESS CALLING USING %run FROM ANOTHER NOTEBOOK
    
    DATADIR = '../data/temperatures/ISD'
    OUTDIR = '../data/temperatures'

    FTPHOST = 'ftp.ncdc.noaa.gov'
    FETCH_STATIONS_LIST_FILE = True

    TEMP_COL = 'AT' # The label of the hourly temperature column we make/output

    # Resampling and interpolation parameters
    # spline order used for converting to on-the-hour and filling small gaps
    BASE_INTERPOLATION_K = 1 # 1 for linear interpolation
    # give special treatment to data gaps longer than...
    POTENTIALLY_PROBLEMATIC_GAP_SIZE = pd.Timedelta('03:00:00')

    # Time range to use for computing normals (30 year, just like NOAA uses)
    NORM_IN_START_DATE = '1986-07-01'
    NORM_IN_END_DATE = '2016-07-01'
    # Time range or normals to output to use when running 'medfoes on normal temperature' (2 years, avoiding leapyears)
    NORM_OUT_START_DATE = '2014-01-01'
    NORM_OUT_END_DATE = '2015-12-31 23:59:59'
    
print("Cleaning temperature data for ",STATION_CALLSIGN)


# Potentially turn interactive figure display off
if SUPPRESS_FIGURE_DISPLAY:
    plt.ioff()
inputs: %%bash
ls /home/jovyan/DATA/SAM/

%%bash
head -n 10 /home/jovyan/DATA/SAM/K562_B-bulk_R1.chr1.sam.25
Given the above code, generate some markdown instructions for it.

targets: 
Let's take a look at .sam files that were created during iterative mapping:
inputs: 2) Then we rotated the data around the z axis and the x axis, of 60 and 30 degrees respectively (by this order!) and we ended up with the first plot in this notebook (check it above).

targets: from codefiles.datagen import _rot_mat_z, _rot_mat_x

# Rotation angles in degrees
theta_x = 30
theta_z = 60

# The rotation matrix is given by: Rot_x * Rot_z
rotation_matrix = np.dot(_rot_mat_x(theta=np.radians(theta_x)), _rot_mat_z(theta=np.radians(theta_z)))
rotation_matrix
inputs: <p style="text-align: right; direction: rtl; float: right; clear: both;">
    עבור כל תא ברשימת <var>items</var>, שמרנו במקום התואם ברשימת <var>stock</var> את הכמות שנמצאת ממנו בחנות.<br>
    יש 4 גזרים, 3 תפוחים ו־2 בננות על המדף בחנות של אדון קשטן.<br>
    שליפה של כמות המלאי עבור מוצר כלשהו בחנות תתבצע בצורה הבאה:
</p>

targets: def get_stock(item_name, items, stock):
    item_index = items.index(item_name)
    how_many = stock[item_index]
    return how_many
inputs: 201 or 1.7% of the items didn't have both a closed and created time

targets: len(results) - len(timelen)

201 / (len(results) + len(timelen))
inputs: Back
<a id='sol1.8'></a>
Exercise 1.8: Solution

targets: dice = BoxModel([1, 2, 3, 4, 5, 6], size=2)
dice.sim(10000).apply(sum)
inputs: "<font color=green>Better, but still less accurate than 86.6%</font>

Train a support vector machine (SVM) classifier
Among the SVM options available, we'll use C-Support Vector Classification (SVC)"
Please write code following the instructions in jupyter notebook style.

targets: 
from sklearn.svm import SVC
svc_model = SVC(gamma='auto')
svc_model.fit(X_train,y_train)
inputs: I am working on the file "Projects/Challenges/challenge_set_5_prashant.ipynb".
The first task is:
Here, the model is using the 'rating' to predict Domestic gross. Since there's 4 ratings, it's predicting one of 4 domestic gross values.
Challenge 4
Can you write Python code for it?

targets: 
y4, X4 = patsy.dmatrices('DomesticTotalGross ~ Budget + Runtime + Rating', data=df, return_type="dataframe")

X4.head()

model = sm.OLS(y4, X4)
fit = model.fit()
fit.summary()

plt.scatter(records, y4, color='g')
plt.scatter(records, fit.predict(X4))
inputs: 2. Create all products of the two lists

targets: l0 = [0, 1, 2]
l1 = ['a', 'b', 'c']
inputs: "Build VectorAssembler stage"
Please write code following the instructions in jupyter notebook style.

targets: 
feature_columns = ['onehot_' + c for c in categorical_columns]
vectorassembler_stage = VectorAssembler(inputCols=feature_columns, outputCol='features')
inputs: ## Open a random image from the pickled files.

def show_rnd_pkl_image():
    """Function that shows a random pickled image."""
    # First let's create a list of all the files.
    # Create data directory path
    dpath = os.path.abspath(os.path.join(os.getcwd(), os.pardir))
    dpath = os.path.join(dpath, 'data')

    # notMNIST_small directory:
    dsmall = os.path.join(dpath, 'notMNIST_small')
    # notMNIST_large directory:
    dlarge = os.path.join(dpath, 'notMNIST_large')

    # Find all pickle files in each directory.
    # http://stackoverflow.com/a/3215392
    # Create a list of all nonMNIST_small pickles
    lsmall = glob.glob(dsmall + '/*.pickle')
    # Create a list of all nonMNIST_large pickles
    llarge = glob.glob(dlarge + '/*.pickle')

    # Pick a random pickle to load (either large or small !)
    rpklfile = random.choice([lsmall, llarge])
    rpklfile = random.choice(rpklfile)

    # verify randomness
    print(rpklfile)

    with open(rpklfile, 'rb') as rf:
        imgPkl = pickle.load(rf)

    plt.imshow(random.choice(list(imgPkl)))
    
show_rnd_pkl_image()
Given the above code, generate some markdown instructions for it.

targets: 
Problem 2
Let's verify that the data still looks good. Displaying a sample of the labels and images from the ndarray. Hint: you can use matplotlib.pyplot.
inputs: "5. Logistic Regression
5.1 Creating a target function
For simplicity, we choose a target function, $f$, to be a 0/1 probability.
For visualization purposes, we choose the domain of interest to be in 2 dimensions, and choose $\mathbf{x}$ to be picked uniformly from the region $\mathcal{X}=\left[-1,1\right] \times \left[-1,1\right]$,
where $\times$ denotes the Cartesian Product.
A random line is created, and to ensure that it falls within the region of interest, it is created from two random points, $(x_0,y_0)$ and $(x_1,y_1)$ which are generated within $\mathcal{X}$.  The equation for this line in slope-intercept form and in the hypothesis / weights can be shown to be:
Slope-Intercept Form
$$m = - \frac{w_1}{w_2}, c = - \frac{w_0}{w_2}$$
Hypothesis Weights Form
$$\mathbf{w} = \left(-c,-m,1\right)$$"
Please write code following the instructions in jupyter notebook style.

targets: 
def generate_data(n,seed=None):
    if seed is not None:
        np.random.seed(seed)
    x0 = np.ones(n)
    x1 = np.random.uniform(low=-1,high=1,size=(2,n))
    return np.vstack((x0,x1)).T

def get_random_line(seed=None):
    X = generate_data(2,seed=seed)
    x = X[:,1]
    y = X[:,2]
    m = (y[1]-y[0])/(x[1]-x[0])
    c = y[0] - m*x[0]
    return np.array([-c,-m,1])

def draw_line(ax,w,marker='g--',label=None):
    m = -w[1]/w[2]
    c = -w[0]/w[2]
    x = np.linspace(-1,1,20)
    y = m*x + c
    if label is None:
        ax.plot(x,y,marker)
    else:
        ax.plot(x,y,marker,label=label)
    
def get_hypothesis(X,w):
    h=np.dot(X,w)
    return np.sign(h).astype(int)
inputs: "Bloom Filter"
Please write code following the instructions in jupyter notebook style.

targets: 
import math
inputs: !aws lambda update-function-configuration \
   --function-name foo  \
   --region us-east-1 \
   --timeout timeout-in-seconds \
Given the above code, generate some markdown instructions for it.

targets: 
Update a function:
inputs: "Answer 3:  No!
Again, $C$ is not given, so $A$ and $B$ are dependent.
Question 4:  Is $(A \perp B \mid C)$?"
Please write code following the instructions in jupyter notebook style.

targets: 
@pgm_render
def pgm_question4():
    pgm = daft.PGM([4, 4], origin=[0, 0])

    # Nodes
    pgm.add_node(daft.Node("c", r"$C$", 2, 3.5, observed=True))
    pgm.add_node(daft.Node("a", r"$A$", 1.3, 2.5))
    pgm.add_node(daft.Node("b", r"$B$", 2.7, 2.5))

    # Add in the edges.
    pgm.add_edge("a", "c", head_length=0.08)
    pgm.add_edge("c", "b", head_length=0.08)
    
    return pgm

%%capture
pgm_question4("images/pgm/question4.png")
inputs: I am working on the file "Getting Started.ipynb".
The first task is:
I used the as_of parameter here to indicate that this data from January. When as_of is ommited, the dimension will always use the current date.
Updating dimension with new data
Now lets import a new clients table:
Can you write Python code for it?

targets: 
df = pd.read_csv('clients 2015-02.csv')
df
inputs: 34. Microphysics Precipitation --&gt; Large Scale Precipitation
Properties of the large scale precipitation scheme
34.1. Scheme Name
Is Required: FALSE&nbsp;&nbsp;&nbsp;&nbsp;Type: STRING&nbsp;&nbsp;&nbsp;&nbsp;Cardinality: 0.1
Commonly used name of the large scale precipitation parameterisation scheme

targets: # PROPERTY ID - DO NOT EDIT !  
DOC.set_id('cmip6.atmos.microphysics_precipitation.large_scale_precipitation.scheme_name')  

# PROPERTY VALUE: 
# Set as follows: DOC.set_value("value")  
DOC.set_value("Hourdin et al (2006)")
inputs: A side effect of this is that a yield statement in some parts of a list comprehension causes it to evaluate to a generator object.

targets: [i for i in range(3) if (yield i)]
inputs: # Fixed-width
h1(data, "fixed_width", bin_width=6).numpy_bins  

# Numpy-like
print("Expected:", np.histogram(data, 5)[1])

print("We got:", h1(data, "numpy", bin_count=5).numpy_bins)

# Integer - centered around integers; useful for integer data
h1(data, "integer").numpy_bins     

# Exponential - positive numbers required
h1(np.abs(data), "exponential").numpy_bins      # We 'abs' the values

# Quantile - each bin should have a similar statistical importance
h1(data, "quantile", bin_count=5).numpy_bins


# Human - as friendly to your plots as possible, you may set an approximate number of bins
h1(data, "human").numpy_bins             
Given the above code, generate some markdown instructions for it.

targets: 
These names can be used as the second parameter of the h1 function:
inputs: # Feel free to change these parameters according to your system's configuration

BATCH_SIZE = 64
BUFFER_SIZE = 1000
embedding_dim = 256
units = 512
vocab_size = top_k + 1
num_steps = len(img_name_train) // BATCH_SIZE
# Shape of the vector extracted from InceptionV3 is (64, 2048)
# These two variables represent that vector shape
features_shape = 2048
attention_features_shape = 64

# Load the numpy files
def map_func(img_name, cap):
  img_tensor = np.load(img_name.decode('utf-8')+'.npy')
  return img_tensor, cap

dataset = tf.data.Dataset.from_tensor_slices((img_name_train, cap_train))

# Use map to load the numpy files in parallel
dataset = dataset.map(lambda item1, item2: tf.numpy_function(
          map_func, [item1, item2], [tf.float32, tf.int32]),
          num_parallel_calls=tf.data.experimental.AUTOTUNE)

# Shuffle and batch
dataset = dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE)
dataset = dataset.prefetch(buffer_size=tf.data.experimental.AUTOTUNE)
Given the above code, generate some markdown instructions for it.

targets: 
创建用于训练的 tf.data 数据集
我们的图像和描述已准备就绪！接下来，让我们创建一个 tf.data 数据集，用于训练我们的模型。
inputs: I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 2 - Sentiment analysis.ipynb".
The first task is:
Once we have a sentiment score for each presidential sentence, we can compute the average across all sentences within a biography, and sort the resulting average presidential sentiment.
Can you write Python code for it?

targets: 
mean_potus_sentiment = pd.Series({potus:np.mean(scores) for potus,scores in potus_sentiments.items()})
mean_potus_sentiment.sort_values(ascending=False)
inputs: "5. getTestResults; storeTestResults; pull some statistics"
Please write code following the instructions in jupyter notebook style.

targets: 
test_results = get.getTestResults(score, y_test)
test_results.head(10)

store.storeTestResults(test_results)

## as we can see, the data is actually stored
pull.pullTestResults(main.session_id).head()

roc = pull.pullROC(main.session_id)
liftchart = pull.pullLiftChart(main.session_id)

fg = plt.figure(figsize=(10,5))
adj = plt.subplots_adjust(hspace=0.4,wspace=0.2)

sp = plt.subplot(1,2,1)
l1 = plt.plot(roc.false_positive_rate, roc.true_positive_rate)
tl = plt.title("ROC curve")

sp = plt.subplot(1,2,2)
l1 = plt.plot(liftchart.population, liftchart.target_population)
tl = plt.title("Liftchart")

plt.show()

pull.pullAccuracy(main.session_id, 0.5, 'population')

pull.pullAccuracy(main.session_id, 0.5, 'probability')

pull.pullConfMatrix(main.session_id, 0.5, 'probability')

threshold_value = 0.5 
threshold_type = "population"
main.updateThreshold(model_id, threshold_value, threshold_type)
inputs: "Combinational
The combinational syntax allows you to use if/else statements.  These conditional statements are not executed in Python, instead they are lowered to hardware muxes."
Please write code following the instructions in jupyter notebook style.

targets: 
@m.circuit.combinational
def basic_if(I: m.Bits[2], S: m.Bit) -> m.Bit:
    if S:
        x = I[0]
    else:
        x = I[1]
    return x
print(repr(basic_if.circuit_definition))
inputs: "Create a BigQuery dataset
If you already have a dataset ready to save tables to, skip this step.
Set the name of your BigQuery dataset below. Dataset IDs
must be alphanumeric (plus underscores) and must be at most 1024 characters
long."
Please write code following the instructions in jupyter notebook style.

targets: 
DATASET_NAME = "" # Name the dataset you'd like to save the output to
LOCATION = "US"

! bq mk --location=$LOCATION --dataset $PROJECT_ID_BILLING:$DATASET_NAME
inputs: #print df.shape
plt.plot(df['ManuleY']+np.random.randn(df.shape[0])*0.1,df['CatalogY']+np.random.randn(df.shape[0])*0.1,'.')
plt.xlim([-0.4,1.3])
plt.ylim([-0.4,1.3])
plt.xlabel('ManuleY')
plt.ylabel('CatalogY')
Given the above code, generate some markdown instructions for it.

targets: 
Let's first have compare the Manuel selected signals with the catalog selected signals. 
As a reminder: 
- Manuel Y is those signals selected by human eye as looks like transit shape
- Catalog Y is those stars have signals injected in them at the original input catalog that we determined have enough signal to noise to be recovered
inputs: "Acima podemos ver que a região 1 teve o maior número de ocorrências criminais
Podemos agora ver quais são essas ocorrências de forma mais detalhada"
Please write code following the instructions in jupyter notebook style.

targets: 
regiao_1_detalhe = df_abril[df_abril['CLUSTER'] == 1]
regiao_1_detalhe
inputs: Congratulations! You've now successfully completed the tutorial. You started Clipper, created an application and queried it from a frontend client, and deployed two models trained in two different machine learning frameworks (Scikit-Learn and TensorFlow) to the running system.
Head back to the notebook from part 1. When you're done watching the accuracy of your application, stop the cell (hit the little "stop" square in the notebook toolbar).
<img src="img/warning.jpg" style="width: 400px;"/>

This step will stop and remove all Clipper Docker containers running on the host. This command will not affect other Docker containers running on the host machine.

Cleanup
When you're completely done with the tutorial and want to shut down your Clipper instance, you can run the stop_all() command to stop all the Clipper Docker containers.
If you check the accuracy of your frontend application a final time, you should see accuracy around 88%. If the accuracy is below that, you can try sending more feedback to increase the weight on the TensorFlow model even more.

targets: clipper.stop_all()
inputs: "Create the model version
Creating the version will take ~5-10 minutes. Note that your first deploy may take longer."
Please write code following the instructions in jupyter notebook style.

targets: 
# Each time you create a version the name should be unique
VERSION = 'v1'

# Create the version with gcloud
explain_method = 'integrated-gradients'
!gcloud beta ai-platform versions create $VERSION --region=$REGION \
--model $MODEL \
--origin $export_path \
--runtime-version 1.15 \
--framework TENSORFLOW \
--python-version 3.7 \
--machine-type n1-standard-4 \
--explanation-method $explain_method \
--num-integral-steps 25

# Make sure the model deployed correctly. State should be `READY` in the following log
!gcloud ai-platform versions describe $VERSION --model $MODEL
inputs: # data exploration
print(data_raw.shape)
data_raw.head(n = 5)
Given the above code, generate some markdown instructions for it.

targets: 
After loading the data we can have s look at its first several rows.
inputs: Build RNN Cell and Initialize
Stack one or more BasicLSTMCells in a MultiRNNCell.
- The Rnn size should be set using rnn_size
- Initalize Cell State using the MultiRNNCell's zero_state() function
    - Apply the name "initial_state" to the initial state using tf.identity()
Return the cell and initial state in the following tuple (Cell, InitialState)

targets: test_batch_size_ph = tf.placeholder(tf.int32)

test_batch_size_ph.shape

def get_init_cell(batch_size, rnn_size):
    """
    Create an RNN Cell and initialize it.
    :param batch_size: Size of batches
    :param rnn_size: Size of RNNs
    :return: Tuple (cell, initialize state)
    """
    lstm_layers = 2
    cell = tf.contrib.rnn.BasicLSTMCell(num_units=rnn_size)
#     drop = tf.contrib.rnn.DropoutWrapper(cell)
    multi = tf.contrib.rnn.MultiRNNCell([cell] * lstm_layers)
    initial_state = multi.zero_state(batch_size, tf.float32)
    initial_state =  tf.identity(initial_state, "initial_state")
    return multi, initial_state


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_get_init_cell(get_init_cell)
inputs: 12. Soil --&gt; Hydrology --&gt; Freezing
TODO
12.1. Number Of Ground Ice Layers
Is Required: TRUE&nbsp;&nbsp;&nbsp;&nbsp;Type: INTEGER&nbsp;&nbsp;&nbsp;&nbsp;Cardinality: 1.1
How many soil layers may contain ground ice

targets: # PROPERTY ID - DO NOT EDIT !  
DOC.set_id('cmip6.land.soil.hydrology.freezing.number_of_ground_ice_layers')  

# PROPERTY VALUE: 
# Set as follows: DOC.set_value(value)  
DOC.set_value(6)
inputs: # This should tell us that instance p is a Parabola type of class:

p.__class__

p.__class__ == Parabola

p.__class__.__name__

# You can see it's a string from the tick
# marks, but also:

type(p.__class__.__name__)
Given the above code, generate some markdown instructions for it.

targets: 
A couple of attributes of classes:
inputs: I am working on the file "ATIVIDADE2/Lab3_AnaliseExploratoria.ipynb".
The first task is:
Notem que lemos o histograma da seguinte maneira: em cerca de 50 meses foram observadas entre 1750 e 2000 prisões. Porém, não sabemos precisar em quais meses houve um aumento ou redução das prisões. Isso deve ser observado através de um gráfico de linha.
(3e) Box-plot
O Box-plot é um gráfico muito utilizado em estatística para visualizar o resumo estatístico de uma variável.
Para esse exercício vamos plotar duas box-plot sobre a média do número de prisões durante os meses analisados para os crimes do tipo ROBBERY e ASSAULT.
O mapeamento é exatamente o mesmo do exercício anterior, porém filtrando para o tipo de roubo analisado.
Can you write Python code for it?

targets: 
# EXERCICIO

parseDayMonth = lambda x: '{}-{}'.format(x.month,x.year)

bookedRDD = (CrimeHeadlessRDD
             .filter(lambda x: u'"ARREST' in x.Resolution)#<COMPLETAR>
             .map(lambda y:(parseMonthYear(y.Dates),1))#<COMPLETAR>
             .reduceByKey(add)#<COMPLETAR>
             .map(lambda z:z[1])#.<COMPLETAR>
            )


robberyBookedRDD = (CrimeHeadlessRDD
                    .filter(lambda x: u'"ARREST' in x.Resolution and x.Category == "ROBBERY")#<COMPLETAR>
                    .map(lambda y:(parseMonthYear(y.Dates),1))#<COMPLETAR>
                    .reduceByKey(add)#<COMPLETAR>
                    .map(lambda z:z[1])#.<COMPLETAR>
                   )

assaultBookedRDD = (CrimeHeadlessRDD
                    .filter(lambda x: u'"ARREST' in x.Resolution and x.Category == "ASSAULT")#<COMPLETAR>
                    .map(lambda y:(parseMonthYear(y.Dates),1))#<COMPLETAR>
                    .reduceByKey(add)#<COMPLETAR>
                    .map(lambda z:z[1])#.<COMPLETAR>
                   )

robData = robberyBookedRDD.collect()
assData = assaultBookedRDD.collect()

assert robData[0]==27,'valores incorretos'
print 'ok'
assert assData[0]==152,'valores incorretos'
print 'ok'
inputs: # Medindo strings
words = ['Platão', 'Sócrates', 'Eu']
for w in words:
    print(w, len(w))
Given the above code, generate some markdown instructions for it.

targets: 
Já vimos que podem existir um ou mais blocos elif, e o bloco else é opcional. O comando elif é uma abreviação para ``else if'', sendo útil para reduzir a quantidade de indentações. Uma sequência if ... elif ... elif ... é o substituto para os comandos switch ou case disponíveis noutras linguagens de programação.
Python: de volta  ao comando for
Como vimos no Python o comando for permite iterar sobre objectos de qualquer sequência (uma lista ou uma string) ou um conjunto, nas sequências o ciclo for segue a ordem pela qual os objectos aparecem na sequência. Por examplo:
inputs: Altura del edificio

targets: plt.plot(ap.index,ap.h-ap.h.min(),'r-',lw=2)
plt.title('La Altura Entrebosques del S3 al 26 es %g m' %round( (ap.h.max()-ap.h.min()),1 ))
plt.xlabel('Tiempo transcurrido (s)',size=20)
plt.ylabel('Altura del edificio (m)',size=20)

ap.p.min()

ptoh(ap.p.min())

print('floor')
raw_input()
print('head')
raw_input()
print('height')
inputs: "Curiously, the total output of agriculture is higher, if cities whithout agriculure die.
For the other outputs, leaving settlements without agriculture alive increases the output.
Apparently, climate variability has big impact on Income from agriculture, little impact on income from ecosystems and no impact on income from trade.
All of this leaves me wondering 
* 1) if trade links are removed propperly when cities population shrinks.
It looks as if trade links only vanish if settlements die. Check this!
* 2) What impact the decay of the ecosystem and climate variability actually has on the income from ecosystem services. It seems like the impact ist negligible and in the original model, the impact came primarily from loss of settlements. More details on this in experiment 4
* 3) If this behavior depends on the weights of income from agriculture and ecosystem services relative to the income from trade. More on this in Experiment 2"
Please write code following the instructions in jupyter notebook style.

targets: 
key=['total_trade_links']
level=['observables']
unstack = ['kill_cropless']

plotlines(data, level, key, unstack)
inputs: Building a logistic regression classifier with Scikit-Learn

targets: #cat_to_num(data['Sex'])
features = prepare_data(data_train)
features[:5]

from sklearn.linear_model import LogisticRegression
model = LogisticRegression()
model.fit(features, data_train["Survived"])

# Make predictions
model.predict(prepare_data(data_test))

# The accuracy of the model on the test data
# (this will be introduced in more details in chapter 4)
model.score(prepare_data(data_test), data_test["Survived"])
inputs: "<div class="alert alert-info"><h4>Note</h4><p>The MNE sample dataset should be downloaded automatically but be
          patient (approx. 2GB)</p></div>

Read data from file:"
Please write code following the instructions in jupyter notebook style.

targets: 
raw = mne.io.read_raw_fif(raw_fname, add_eeg_ref=False)
print(raw)
print(raw.info)
inputs: I am working on the file "04-Pandas-Data-Tables/Ten-Minute-Tutorial_class.ipynb".
The first task is:
Converting between time span representations
Can you write Python code for it?

targets: 
rng = pd.date_range('1/1/2012', periods=5, freq='M')
ts = pd.Series(np.random.randn(len(rng)), index=rng)
ts

ps = ts.to_period()
ps

ps.to_timestamp()