Muennighoff/xP3x-sample huggingface dataset:
inputs: "Custom Aggregation
An aggregation function collects many pieces of individual data and summarizes it in some way. Examples of built-in aggregation functions are avg() (average), min(), max(), and count().
The API for aggregators used by sqlite3 is defined in terms of a class with two methods. The step() method is called once for each data value as the query is processed. The finalize() method is called one time at the end of the query and should return the aggregate value. This example implements an aggregator for the arithmetic mode. It returns the value that appears most frequently in the input."
Please write code following the instructions in jupyter notebook style.

targets: 
import sqlite3
import collections

db_filename = 'todo.db'


class Mode:

    def __init__(self):
        self.counter = collections.Counter()

    def step(self, value):
        print('step({!r})'.format(value))
        self.counter[value] += 1

    def finalize(self):
        result, count = self.counter.most_common(1)[0]
        print('finalize() -> {!r} ({} times)'.format(
            result, count))
        return result


with sqlite3.connect(db_filename) as conn:
    conn.create_aggregate('mode', 1, Mode)

    cursor = conn.cursor()
    cursor.execute("""
    select mode(deadline) from task where project = 'pymotw'
    """)
    row = cursor.fetchone()
    print('mode(deadline) is:', row[0])
inputs: Create a variable
To create a variable, provide an initial value.  The tf.Variable will have the same dtype as the initialization value.

targets: my_tensor = tf.constant([[1.0, 2.0], [3.0, 4.0]])
my_variable = tf.Variable(my_tensor)

# Variables can be all kinds of types, just like tensors
bool_variable = tf.Variable([False, False, False, True])
complex_variable = tf.Variable([5 + 4j, 6 + 1j])
inputs: I am working on the file ".ipynb_checkpoints/fcFinder Demo-checkpoint.ipynb".
The first task is:
Catching anatomical modifiers and creating annotations out of them
Can you write Python code for it?

targets: 
markup2 = context.getSectionMarkups()[0]
markup2
inputs: "Create and fit Spark ML model"
Please write code following the instructions in jupyter notebook style.

targets: 
from pyspark.ml.classification import LogisticRegression
from pyspark.ml.feature import VectorAssembler
from pyspark.ml import Pipeline

# Create feature vectors. Ignore arr_delay and it's derivate, is_late
feature_assembler = VectorAssembler(
    inputCols=[x for x in training.columns if x not in ["is_late","arrdelay"]],
    outputCol="features")

reg = LogisticRegression().setParams(
    maxIter = 100,
    labelCol="is_late",
    predictionCol="prediction")

model = Pipeline(stages=[feature_assembler, reg]).fit(training)
inputs: # Import everything needed to edit/save/watch video clips
from moviepy.editor import VideoFileClip
from IPython.display import HTML
Given the above code, generate some markdown instructions for it.

targets: 
Test on Videos
You know what's cooler than drawing lanes over images? Drawing lanes over video!
We can test our solution on two provided videos:
solidWhiteRight.mp4
solidYellowLeft.mp4
inputs: # Much larger than 800,000 nodes + edges. Does not need to render in 20 seconds. 
url = 'http://' + SERVER + '/graph/graph.html?dataset=Amazon&scene=default&info=true&play=10000&mapper=splunk&splashAfter=1477695505'
IFrame(url, width=700, height=350)
Given the above code, generate some markdown instructions for it.

targets: 
Amazon 262111 nodes , 1,234,877 Edges

Nodes - Products or Customers
Edges - A customer review
inputs: # Create the figure
fig = figure(figsize=(10,4))
ax = fig.add_subplot(121)
ax.set_title('Decision boundary')

# Chose the x-axis coords of the
# two points to plot the decision 
# boundary line
x1 = array([lower_bound[0],upper_bound[0]])

# Calculate the y-axis coords of the
# two points to plot the decision 
# boundary line as it changes 
for t in xrange(stime) :
    
    # Show evert 10th timestep
    if t%10 == 0:
        
        if dw[2,t] != 0 :
            # Evaluate x2 based on current weights
            x2 = -(dw[1,t]*x1 + dw[0,t])/dw[2,t]
        
            # Plot the changes in the boundary line during learning
            ax.plot(x1,x2, c='#cccccc', linewidth = 1, zorder = 1)

# Evaluate x2 ibased on final weights
x2 = -(w[1]*x1 + w[0])/w[2]

# Plot the learned boundary line
plot(x1,x2, c= '#000000', linewidth = 2, zorder = 1)

# Plot in red points belonging to the class
scatter(*P[(n_patterns/2):,:].T, s = 50,  c = '#ff8888', zorder = 2 )       
# Plot in blue points not belonging to the class 
scatter(*P[:(n_patterns/2),:].T, s = 50,  c = '#8888ff', zorder = 2 )

# Limits and labels of the plot
xlim( [lower_bound[0], upper_bound[0]] )
ylim( [lower_bound[1], upper_bound[1]] )
xlabel("$p_1$", size = 'xx-large')
ylabel("$p_2$", size = 'xx-large')

# Plot squared errors
ax = fig.add_subplot(122)
ax.set_title('Error')
ax.plot(squared_errors)

# Labels and ticks of the plot
xlabel("epochs", size = 'xx-large')
ylabel("SSE", size = 'xx-large')
xticks(range(epochs)) 

show()
Given the above code, generate some markdown instructions for it.

targets: 
Plotting the results of training
We plot the final decision boundary together with the history of the squared errors through epocs.
As you see below, the network finds a line that completely divides points belonging to the class from those not belonging to it. The perceptron can only define a linear boundary, thus it cannot find any correct solution. The background of the "Decision boundary" plot is all gray. This was due to the fact that the network is continously exploring all the possible inclinations of the boundary. You can see in the error plot how the curve of the error do not converge to zero. The network cannot find a minimum in this case.
inputs: 2 deque
Deques are a generalization of stacks and queues,Deques support thread-safe, memory efficient appends and pops from either side of the deque with approximately the same O(1) performance in either direction.


append(x)
Add x to the right side of the deque.


appendleft(x)
Add x to the left side of the deque.


clear()
Remove all elements from the deque leaving it with length 0.


count(x)
Count the number of deque elements equal to x.


extend(iterable) 
Extend the right side of the deque by appending elements from the iterable argument.


extendleft(iterable)
Extend the left side of the deque by appending elements from iterable. Note, the series of left appends results in reversing the order of elements in the iterable argument.


pop()
Remove and return an element from the right side of the deque. If no elements are present, raises an IndexError.


popleft()
Remove and return an element from the left side of the deque. If no elements are present, raises an IndexError.


remove(value)
Removed the first occurrence of value. If not found, raises a ValueError.


reverse()
Reverse the elements of the deque in-place and then return None.


rotate(n)
Rotate the deque n steps to the right. If n is negative, rotate to the left. Rotating one step to the right is equivalent to: d.appendleft(d.pop()).


maxlen
Maximum size of a deque or None if unbounded.


3 defaultdict
dictionary supply missing values

targets: s = [('yellow', 1), ('blue', 2), ('yellow', 3), ('blue', 4), ('red', 1)]
d = defaultdict(list)
for k, v in s:
    d[k].append(v)
d.items()
inputs: # Note the nature of the resulting index:
giss_temp_series.index

# It is an index made of 2 columns. Let's fix the fact that one of them doesn't have a name:
giss_temp_series.index = giss_temp_series.index.set_names(["year", "month"])

# We can now access deviations by specifying the year and month:
giss_temp_series[1980, "Jan"]
Given the above code, generate some markdown instructions for it.

targets: 
A side note: Multi-indexes
inputs: I am working on the file "Labs/Lab6/Lab6.ipynb".
The first task is:
So tuples are immutable, but not indestructible. Once we've defined a new one, we can assign it to x and overwrite the original if we really want to
Can you write Python code for it?

targets: 
x=y
x
inputs: I am working on the file "2-Numpy.ipynb".
The first task is:
Los índices negativos se cuentan desde el fin del arreglo (los índices positivos desde el comienzo):
Can you write Python code for it?

targets: 
A = np.array([1,2,3,4,5])

A[-1] # el último elemento del arreglo

A[-3:] # los últimos 3 elementos
inputs: "After using the show_topics method from the model, it will output the most probable words that appear in each topic. For the gensim library, the default printing behavior is to print a linear combination of the top words sorted in decreasing order of the probability of the word appearing in that topic. Thus words that appear towards the left are the ones that are more indicative of the topic.
We see our LDA model has given us a pretty intuitive result. Bank is the most influential word in both the topics and other words help define what kind of bank we are talking about. We can also use the function get_term_topics and get_document_topics to further evaluate our result. get_term_topics returns the odds of that particular word belonging to a particular topic. A few examples:"
Please write code following the instructions in jupyter notebook style.

targets: 
print(texts_model.get_term_topics('water'))
print(texts_model.get_term_topics('bank'))
inputs: 2. Save frame and display JPG here

targets: frame = hdmi.frame()
orig_img_path = '/home/xilinx/jupyter_notebooks/examples/data/orig.jpg'
frame.save_as_jpeg(orig_img_path)

Image(filename=orig_img_path)
inputs: I am working on the file "Exploration_of_data_in_MillionMusicSubset.ipynb".
The first task is:
Scatter plots of artist_familiarity vs year compared to artist_hotttnesss vs year
Can you write Python code for it?

targets: 
fig, ax = plt.subplots(nrows=2, ncols=2, sharex=True, sharey=True,
                       figsize=(15,12))

ax[0,0].scatter(songs.year, songs.artist_familiarity, marker='.')
ax[0,1].hexbin(songs.year, songs.artist_familiarity, cmap='viridis', gridsize=41, mincnt=1.0)

ax[1,0].scatter(songs.year, songs.artist_hotttnesss, marker='.')
ax[1,1].hexbin(songs.year, songs.artist_hotttnesss, cmap='viridis', gridsize=41, mincnt=1.0)
ax[-1,-1].set_xlim(1920,songs.year.max());
plt.subplots_adjust(wspace=0.02, hspace=0.05)
inputs: I am working on the file "P4B - Capitolo 2 - Persistenza, Unicode, OOP e DB.ipynb".
The first task is:
References python 2.7

IL Tutorial
Coding Style


Le funzioni builtins (su ipython >>> help(__builtins__))
Gli HowTo (tra cui "Unicode HowTo" e "Idioms and Anti-Idioms in Python")
The absolute minimum every developer must know about unicode and character sets no excuses!
Pragmatic Unicode
Tipi, operatori e comparazioni
String format mini-language
DB-API 2.0 (PEP 249)
Scope delle variabili (blog)
Test con PyTest

Feedback sono graditi

privati via luca@befair.it , cell: 3289639660
via telegram (sempre con il numero di cell)
se volete farli pubblici su www.befair.it
Can you write Python code for it?

targets: 
phrase = "happy-python-hacking!"
the_end = u" ".join([s.capitalize() for s in phrase.split("-")])
inputs: A function that has default arguments
Python allows you to have functions that have a default value for one or more arguments.  This allows you to call a function with fewer arguments than it is defined to allow (i.e., fewer than the maximum number of possible arguments).  In function definitions of this sort, the mandatory arguments (ones that do not have a default value) must come before the optional arguments (ones that have default values). This can be extremely helpful in many circumstances, and can be called in several ways:

Giving only the mandatory argument
Giving the mandatory argument plus one of the optional arguments
Giving the mandatory argument plus two or more of the optional arguments

There does not need to be a mandatory argument - all arguments can have defaults!  Note that if you are going to give more than just the mandatory argument, the optional arguments need to be given in the order they are defined.  In other words, you can't just give the mandatory argument and the second optional argument in a function with one mandatory argument and multiple optional arguments - you have to give all arguments up to the one that you want.

targets: # this version has one default value

def print_stuff(first,second='dog',third='monkey'):
    print(first,second,third)

print_stuff('cat')

print_stuff('cat','elephant')

print_stuff('donkey','wolf','mouse')

# this version has no default values!

def print_stuff_2(first='fish',second='bird',third='dinosaur'):
    print(first,second,third)

print_stuff_2()

print_stuff_2('newt')

print_stuff_2('vole','koala')

print_stuff_2('lion','armadillo','cow')
inputs: 13.4. Redistribution
Is Required: TRUE&nbsp;&nbsp;&nbsp;&nbsp;Type: ENUM&nbsp;&nbsp;&nbsp;&nbsp;Cardinality: 1.N
Which processes can redistribute sea ice (including thickness)?

targets: # PROPERTY ID - DO NOT EDIT !  
DOC.set_id('cmip6.seaice.dynamics.redistribution')  

# PROPERTY VALUE(S): 
# Set as follows: DOC.set_value("value")  
# Valid Choices: 
#      "Rafting"  
#      "Ridging"  
#      "Other: [Please specify]"  
DOC.set_value("Ridging")
inputs: "Can iterate this over a number of generations."
Please write code following the instructions in jupyter notebook style.

targets: 
generations = 500

def simulate():
    for i in range(generations):
        time_step()

simulate()

pop
inputs: CLeastAngleRegression requires the features to be normalized with a zero mean and unit norm. Hence we use two preprocessors: PruneVarSubMean and NormOne.

targets: #Preprocess data
preproc=PruneVarSubMean()
preproc.init(feats_train)
feats_train.add_preprocessor(preproc)
feats_train.apply_preprocessor()

preprocessor=NormOne()
preprocessor.init(feats_train)
feats_train.add_preprocessor(preprocessor)
feats_train.apply_preprocessor()

print "(No. of attributes, No. of samples) of data:"
print feats_train.get_feature_matrix().shape
inputs: I am working on the file "Model Evaluation/Model_Evaluation-exercise.ipynb".
The first task is:
2 - Fit a machine learning algorithm of your choice to the data, tuning hyperparameters using the Better Holdout Method.  Generate training and validation plots and comment on your results.
Can you write Python code for it?

targets: 
# in solution he encodes all variables once using df.apply(le.fit_transform)
inputs: But what does this UnicodeEncodeError mean?
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf7' in position 7984: ordinal not in range(128)

Well, to find out, we need to dig a bit deeper and check the corresponding MATLAB script. Because every SPM interface creates an executable MATLAB script, either in the current location or in the folder of the node. So what's written in this script?

targets: !cat /home/jovyan/work/notebooks/pyscript_realign.m
inputs: # initial wf: gaussian(x,n,x0,s0,w,v) 
# Gaussian centered at x0=0, width=w=1, velocity=v= 0, without nodes (s0=0)
c0=normaliza(gaussian(zp,Npoint,0,0,1,0)); # wf at t=0
#
# initial wf: thomas_fermi(x,n,x0,s0,gN,Ve) 
# Thomas Fermi for harmonic oscillator without nodes (s0=0)
#c0=normaliza(thomas_fermi(zp,Npoint,0,0,gint/NormWF,Vpot_R)); # wf at t=0
Given the above code, generate some markdown instructions for it.

targets: 
Choose initial state (wave function)
inputs: Create Pipeline

targets: # Create standardizer
standardizer = StandardScaler()

# Create logistic regression
logit = LogisticRegression()

# Create a pipeline that standardizes, then runs logistic regression
pipeline = make_pipeline(standardizer, logit)
inputs: "A new high performance and optimally cheap multivitamin. Saving the world one mixed integer program at a time
You have been tasked with developing a new superior multivitamin. You have been given free reign to select the ingredients and their relative amounts in the vitamin but you have been asked to keep the cost of the raw materials as low as possible.
Further complicating things, you must abide by some restrictions in the formula.

The formula should not have more than 20% of any one vitamin
The formula must have at least 10% Iron, Zinc, or Magnesium
The formula must have at least 20% Vitamin A, Vitamin C, or Vitamin D
Each vitamin that is used must account for at least 5% of the total
The formula may contain as few as 5 vitamins but no more than 10 vitamins
If the formula contains Magnesium it must also contain Calcium and Zinc
The formula must have one of the B vitamins either B6 or B12 but not both

The possible Ingredients and their per mg cost
{{HTML(df.to_html(index=False)) }}
You might see a simple greedy strategy to solve this, but the vitamin B constrains and the Magnesium/Zinc/Calcium constraints make things a bit more complicated. Instead of trial and error we can try and create this vitamin by writing a fairly simple mixed integer linear program. First we need to come up with an expression that captures our ultimate goal, in this case, to minimize the cost of raw materials.
We can calculate the total cost of the formula by adding up the individual cost of each vitamin in it.
$$\text{Total Cost} = \text{ sum over all the vitamins (cost of vitamin) * (percent of vitamin) } $$
First lets load the data we have and organize it so we can easily grab the cost of particular vitamin"
Please write code following the instructions in jupyter notebook style.

targets: 
df = pd.read_csv('vitamin_costs.csv')
vitamins = df.vitamin.values
vitamin_cost = df.set_index('vitamin').to_dict()['cost']
df.head()
inputs: "Convolutional Neural Network (CNN)
TODO 3: Update the CNN architecture
Try adjusting the # of filters (pattern types) and kernel size (size of the sliding window)"
Please write code following the instructions in jupyter notebook style.

targets: 
from tensorflow.keras.layers import AveragePooling1D

# TODO: Try adjusting the # of filters (pattern types) and kernel size (size of the sliding window)
model = Sequential([
    Conv1D(filters=32, kernel_size=3, input_shape=[n_input_steps, n_features]),
    Flatten(),
    Dense(n_output_steps)])

model.compile(optimizer='adam', loss='mae')

early_stopping = EarlyStopping(monitor='val_loss', patience=5)
_ = model.fit(x=X_train, y=y_train, validation_data=(X_test, y_test), epochs=epochs, callbacks=[early_stopping])

model.save('./cnn_export/')

preds = model.predict(X_test)
y_pred_cnn = inverse_scale(preds)

evaluate(y_pred_cnn)
inputs: # Set up the model
valueDate = Date(2016, 9, 17)
maximumDate = Date(2026, 9, 17)

dates = [Date(2016, 9, 17) , Date(2026, 9, 17)]
rates = [ 0.07, 0.07 ]
discountCurve = DatesAndRates(Currency.ZAR, valueDate, dates, rates, maximumDate)

numeraireModel = DeterminsiticCurves(discountCurve);
otherModels = List[Simulator]() # no model other than discounting for now.
coordinator = Coordinator(numeraireModel, otherModels, 1) # the magic ingredient that gets 
                                                          # models and products to work 
                                                          # together

print("A model is ready.")
Given the above code, generate some markdown instructions for it.

targets: 
Setting up a model
We haven't described how to make a model or exactly what it does but the following code is fairly easy to understand:
inputs: "Check the 2016 customers"
Please write code following the instructions in jupyter notebook style.

targets: 
a = known_behaviors[known_behaviors.Mortgage == 1]
b = known_behaviors[known_behaviors.Pension == 1]
c = known_behaviors[known_behaviors.Savings == 1]
print("Number of clients: %d" %len(known_behaviors))
print("Number of clients predicted to buy mortgage accounts: %d" %len(a))
print("Number of clients predicted to buy pension accounts: %d" %len(b))
print("Number of clients predicted to buy savings accounts: %d" %len(c))

known_behaviors["nb_products"] = known_behaviors.Mortgage + known_behaviors.Pension + known_behaviors.Savings

abc = known_behaviors[known_behaviors.nb_products > 1]
print("We have %d clients who bought several products" %len(abc))
abc = known_behaviors[known_behaviors.nb_products == 3]
print("We have %d clients who bought all the products" %len(abc))

products = ["Savings", "Mortgage", "Pension"]
inputs: "Convert training data to TFRecord"
Please write code following the instructions in jupyter notebook style.

targets: 
%cd models/research/slim

! ./prepare_checkpoint_and_dataset.sh --network_type mobilenet_v1
inputs: %%bq query
select
productorservicecode,
systemequipmentcode,
claimantprogramcode,
principalnaicscode,
sum(dollarsobligated) as dollarsobligated
from `fiery-set-171213.vrec.usa_spending_all`
where vendorname in ('LOCKHEED MARTIN CORPORATION')
group by 1,2,3,4
order by 5 DESC
limit 20

%%bq query
select
#mod_parent,
vendorname,
systemequipmentcode,
sum(dollarsobligated) as dollarsobligated
from `fiery-set-171213.vrec.usa_spending_all`
where productorservicecode in ('1510: AIRCRAFT, FIXED WING')
group by 1,2
order by 3 DESC
limit 20

%%bq query
select
vendorname,
systemequipmentcode,
claimantprogramcode,
principalnaicscode,
sum(dollarsobligated) as dollarsobligated
from `fiery-set-171213.vrec.usa_spending_all`
where productorservicecode in ('1510: AIRCRAFT, FIXED WING')
and contractingofficerbusinesssizedetermination in ('S: SMALL BUSINESS')
group by 1,2,3,4
order by dollarsobligated DESC
limit 20

%%bq query
select
*
from `gpqueries.contracts.raw`
where productorservicecode in ('1510: AIRCRAFT, FIXED WING')
and contractingofficerbusinesssizedetermination in ('S: SMALL BUSINESS')
limit 1

%%bq query
select 
claimantprogramcode,
principalnaicscode,
sum(dollarsobligated) as dollarsobligated
from `fiery-set-171213.vrec.usa_spending_all`
where contractingofficerbusinesssizedetermination in ("S: SMALL BUSINESS")
group by 1,2
order by dollarsobligated DESC
limit 10
Given the above code, generate some markdown instructions for it.

targets: 
Understanding where the budget is spent
inputs: I am working on the file "Miscellaneous/Adult Income Classification.ipynb".
The first task is:
KNN accuracy and time elapsed caculation
Can you write Python code for it?

targets: 
t6=time()
print ("KNN")
# knn = KNeighborsClassifier(n_neighbors=3)
knn = KNeighborsClassifier()
clf_knn=knn.fit(X_train, y_train)
print ("Acurracy: ", clf_knn.score(X_test,y_test) )
t7=time()
print ("time elapsed: ", t7-t6)
inputs: But what if we want to change the formatting?

targets: #ax stands for "axis", we'll use this object to change more settings
#we can also specify the image size, in inches, right here with the figsize argument
ax = tmp.plot.line(y='log_salary', figsize=(10,8))

#add a title to the chart
ax.set_title('Average salary by year born')

#label the axes
ax.set_ylabel('log(salary)')
ax.set_xlabel('Year born')

#set the ticks so they're not half years
#The range command makes a list of years starting in 1972 and counting by 2 up to (not including) 1993.
ax.set_xticks(range(1972, 1993, 2))

#show our plot
plt.show()
inputs: Adding depth and sparsity constraint on the encoded representations
In the previous example, the representations were only constrained by the size of the hidden layer (32). In such a situation, what typically happens is that the hidden layer is learning an approximation of PCA (principal component analysis). But another way to constrain the representations to be compact is to add a sparsity contraint on the activity of the hidden representations, so fewer units would "fire" at a given time. 
In Keras, this can be done by adding an activity_regularizer to our Dense layer:

targets: #autoencoder.reset_states()
#encoder.reset_states()
#decoder.reset_states()

from keras import regularizers
from keras import optimizers
from keras.regularizers import l2, activity_l1
from keras.layers import Input, Dense
from keras.models import Model

input_img = Input(shape=(784,))
encoded = Dense(128, activation='relu')(input_img)
encoded = Dense(64, activation='relu')(encoded)
encoded = Dense(32, activation='relu')(encoded)
decoded = Dense(64, activation='relu')(encoded)
decoded = Dense(128, activation='relu')(decoded)
decoded = Dense(784, activation='sigmoid')(decoded)

autoencoder = Model(input_img, decoded)
autoencoder.compile(optimizer='adadelta', 
                    loss='binary_crossentropy',
                    activity_regularizer=regularizers.l1(10e-5))

autoencoder.fit(x_train, x_train,
                nb_epoch=100,
                batch_size=256,
                shuffle=True,
                validation_data=(x_test, x_test))
inputs: I am working on the file "4_dlnd_language_translation/dlnd_language_translation.ipynb".
The first task is:
Process Decoding Input
Implement process_decoding_input using TensorFlow to remove the last word id from each batch in target_data and concat the GO ID to the begining of each batch.
Can you write Python code for it?

targets: 
def process_decoding_input(target_data, target_vocab_to_int, batch_size):
    """
    Preprocess target data for dencoding
    :param target_data: Target Placehoder
    :param target_vocab_to_int: Dictionary to go from the target words to an id
    :param batch_size: Batch Size
    :return: Preprocessed target data
    """
    
    go_id = target_vocab_to_int.get('<GO>')
    gos = tf.fill(dims=(batch_size,1), value=go_id)
    return tf.concat((gos, target_data[:,:-1]), axis=1)

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_process_decoding_input(process_decoding_input)
inputs: I am working on the file "Jupyter_notebooks/Notebook_4_Generating_Model_Directed_Evolution_of_UTRs_From_100_Native_and_Random_UTRs.ipynb".
The first task is:
Organize the data
reshape and combine the dataframes with the sequences and the predicted growth rates
Can you write Python code for it?

targets: 
# reshape the dataframes. Transpose, then convert into a Multiindex, then convert the index into new columns
ran_seqs_df = mut_random_seqs[0].T.unstack().reset_index()
ran_scores_df = mut_random_seqs[1].T.unstack().reset_index()

# rename the columns so that the names are somewhat descriptive
ran_seqs_df.rename(columns = {'level_0' : 'UTR_id', 'level_1' : 'Round', 0 : 'Evolved_Seq'}, inplace = True)
ran_scores_df.rename(columns = {'level_0' : 'UTR_id', 'level_1' : 'Round', 0 : 'Pred_Growth'}, inplace = True)

# merge two datasets, only include columns that aren't redundant
cols_to_use = ran_scores_df.columns.difference(ran_seqs_df.columns) # identify non-redundant columns
ran_seqs_scores = pd.merge(ran_seqs_df, ran_scores_df[cols_to_use], right_index = True, left_index = True)
inputs: "As you can see, these DataFrames are consistent with the format expected by solutionFactory."
Please write code following the instructions in jupyter notebook style.

targets: 
buyDf

consumeDf
inputs: I am working on the file "02. Parte 2/13. Clase 13/13Class NB.ipynb".
The first task is:
2. Modelo normal para los rendimientos
Can you write Python code for it?

targets: 
huber = sm.robust.scale.Huber()
returns_av, scale = huber(daily_returns)
mu=daily_returns.mean().AAPL
sigma=daily_returns.std().AAPL
ndays = 360
ntraj=10000
K1=160
K2=180

mu=daily_returns.mean().AAPL
sigma=daily_returns.std().AAPL
dates=pd.date_range('20170101',periods=ndays)
simret = pd.DataFrame(sigma*np.random.randn(ndays,ntraj)+mu,index=dates)
simdata=(data.loc['2016-12-30',:].AAPL)*np.exp(simret.cumsum())
ST=simdata.tail(1)
ind=np.digitize(ST.values[0],np.array([0, K1, K2, ST.values.max()+1]))
freq=(np.array([sum(ind==1),sum(ind==2),sum(ind==3)])/ntraj)*100

bar_labels = ['ST<K1', 'K1<ST<K2', 'ST>K2']
x_pos = list(range(len(bar_labels)))
plt.bar(x_pos,freq);
plt.grid();
plt.xticks(x_pos, bar_labels);
inputs: %%bash
bq --location=US mk -d demo
Given the above code, generate some markdown instructions for it.

targets: 
Train Model
With the relevant columns chosen to accomplish predictions, it is then possible to create and train the model in BigQuery. First, a dataset will be needed store the model.
inputs: I am working on the file "Python/5 Linear Regression/Polynomial-Regression.ipynb".
The first task is:
This seems to be working and the explained variance has improved. Let's try to use even higher order polynomials.  Hopefully, we can get a $100\%$ explained variance.
Can you write Python code for it?

targets: 
plot_nth_degree_polynomial(6)
inputs: "2 Categories
Categorical data is best represented by bar or pie charts. Reproduce the plots below using the object-oriented API of matplotlib, which is recommended for programming.
Question: What are the pros / cons of each plot ?
Tip: the matplotlib gallery is a convenient starting point."
Please write code following the instructions in jupyter notebook style.

targets: 
data = [10, 40, 25, 15, 10]
categories = list('ABCDE')

fig, axes = plt.subplots(1, 2, figsize=(15, 5))

# Right plot.
# axes[1].
# axes[1].

# Left plot.
# axes[0].
# axes[0].
inputs: %%csound 1
instr 1
asig asds

%%csound 1
instr 1
asig oscil 0.5, 440
outs asig, asig

%%csound 1
instr 1
asig oscil 0.5, 440
outs asig, asig
endin
Given the above code, generate some markdown instructions for it.

targets: 
Sending instruments
You can send instruments to a running csound engine with the %%csound magic. Any syntax errors will be displayed inline.
inputs: # Load COCO data from disk; this returns a dictionary
# We'll work with dimensionality-reduced features for this notebook, but feel
# free to experiment with the original features by changing the flag below.
data = load_coco_data(pca_features=True)

# Print out all the keys and values from the data dictionary
for k, v in data.iteritems():
  if type(v) == np.ndarray:
    print k, type(v), v.shape, v.dtype
  else:
    print k, type(v), len(v)
Given the above code, generate some markdown instructions for it.

targets: 
Load MS-COCO data
As in the previous notebook, we will use the Microsoft COCO dataset for captioning.
inputs: "Backends
PyMC3 has support for different ways to store samples during and after sampling, called backends, including in-memory (default), text file, and SQLite. These can be found in pymc.backends:
By default, an in-memory ndarray is used but if the samples would get too large to be held in memory we could use the sqlite backend:"
Please write code following the instructions in jupyter notebook style.

targets: 
from pymc3.backends import SQLite

with model_glm_logistic:
    backend = SQLite('trace.sqlite')
    trace = sample(5000, trace=backend)

summary(trace, vars=['x1', 'x2'])
inputs: # Compute the eccentricties
def ecc(q, v, t):
    Q = np.vstack([q, np.zeros(t.size)])
    V = np.vstack([v, np.zeros(t.size)])
    qT, vT = np.transpose(Q), np.transpose(V)
    return np.array( [orbit.Calc_e(qT[ii], vT[ii], (1.-beta)*G*M_Sun) for ii in range(t.size)])

e_slim2 = ecc(q_slim2, pi_slim2/m, t)
e_slim4 = ecc(q_slim4, pi_slim4/m, t)
e_slim6 = ecc(q_slim6, pi_slim6/m, t)
e_rk2 = ecc(q_rk2, v_rk2, t)
e_rk4 = ecc(q_rk4, v_rk4, t)

fig3 = plt.figure(figsize=(12,5), dpi=500)
ax3 = fig3.add_subplot(1,1,1)

ax3.set_xlim(0.01, 6000)
ax3.loglog(t, np.abs(e_slim2/e_slim6-1.), 'r-', linewidth=2.0, rasterized=True)
ax3.loglog(t, np.abs(e_slim4/e_slim6-1.), '-', color='orange', linewidth=2.0, rasterized=True)
ax3.loglog(t, np.abs(e_rk2/e_slim6-1.), 'g--', linewidth=2.0, rasterized=True)
ax3.loglog(t, np.abs(e_rk4/e_slim6-1.), 'b--', linewidth=2.0, rasterized=True)

ax3.set_xlabel('Time, $t$ [yr]', fontsize=18)
ax3.set_ylabel('Frac\'l Eccentricity Error, $\delta e/e_6$', fontsize=18)

ax3.text(20, 1.8e-3, r'$2^{nd}$ order Slimplectic', fontsize=15, color='black')
ax3.text(20, 0.5e-7, r'$4^{th}$ order Slimplectic', fontsize=15, color='black')
ax3.text(40, 1e-4, r'$4^{th}$ order RK', fontsize=15, color='blue', rotation = 10)
ax3.text(40, 1e0, r'$2^{nd}$ order RK', fontsize=15, color='g', rotation=10)

ax3.tick_params(axis='both', which='major', labelsize=16)
ax3.set_yticks([1e-12, 1e-9, 1e-6, 1e-3, 1e0])
ax3.text(0.015, 1e0, r'$\Delta t = 0.01$ yr', fontsize=18, color='black'); 

#fig3.savefig(plot_path + "PRDrag_Ecc_errorLong.pdf", transparent=True,bbox_inches='tight')
Given the above code, generate some markdown instructions for it.

targets: 
We can also look at how the orbital eccentricity changes depending on the integration scheme and order.
inputs: Below, we will plot the rolling mean values that will be used to calculate 1993's number of bombings. We will use a lookback window of 4 years to smooth out any random spikes that occur over time, and get a more general line to capture these trends over the time period of interest.

targets: ## Pandas has rolling mean function
pd.rolling_mean(bombing, 4)

## Using the function created above to show this visually
test_stationarity(bombing, 4)
inputs: # For 00, do nothing

# For 01, apply $X$
#shared.x(q[0])

# For 01, apply $Z$
#shared.z(q[0])

# For 11, apply $XZ$
superdense.z(sdq[0]) 
superdense.x(sdq[0])
superdense.barrier()
Given the above code, generate some markdown instructions for it.

targets: 
Alice now needs to decide what two bit message she wants to transmit to Bob, ($00$, $01$, $10$, or $11$), and perform the corresponding to transformation ($I$, $X$, $Z$ or $XZ$ respectively) to her qubit $q_A$ ($q_0$). In this case, she encodes $11$:
inputs: # Define a saturation nonlinearity as a simple function
def my_saturation(x):
    if abs(x) >= 1:
        return math.copysign(1, x)
    else:
        return x

amp_range = np.linspace(0, 2, 50)
plt.plot(amp_range, ct.describing_function(my_saturation, amp_range).real)
plt.xlabel("Amplitude A")
plt.ylabel("Describing function, N(A)")
plt.title("Describing function for a saturation nonlinearity");
Given the above code, generate some markdown instructions for it.

targets: 
User-defined, static nonlinearities
In addition to pre-defined nonlinearies, it is possible to computing describing functions for static nonlinearities.  The describing function for any suitable nonlinear function can be computed numerically using the describing_function function.
inputs: "Calculate torque in N-m."
Please write code following the instructions in jupyter notebook style.

targets: 
tau_ind = Pag / w_sync
inputs: # Download the model from GCS
download_from_gcs(PRETRAINED_KERAS_MODEL, 'pretrained_model.h5')

# Load the model, strategy.scope allows us to exploit multiple accelerators,
# just like we did during training.
with strategy.scope():
  model = tf.keras.models.load_model('pretrained_model.h5', compile=False)
  model.compile(
      optimizer=tf.train.AdamOptimizer(), loss=weighted_binary_crossentropy)
Given the above code, generate some markdown instructions for it.

targets: 
Then we'll download and load the keras model.
inputs: Concise test<a name="_concise test"></a>

targets: contingency_name_from_id = {
    0: 'Nível basal',
    1: 'Nível 0',
    2: 'Nível 1',
    3: 'Nível 2',    
}

def alert_trigger(dataset_id: int, year: int, territory_id: int):
    df = fluDB.get_data(dataset_id=dataset_id, scale_id=1, year=year, territory_id=territory_id)

    # If not obitoflu dataset (3), uses last 4 weeks, o.w. use 3:
    if dataset_id < 3:
        wdw = 4
    else:
        wdw = 3

    alert_zone = any(df.estimated_cases[-wdw:] > df.typical_high[-wdw:])
    data_increase = all(df.estimated_cases[-wdw:].values - df.estimated_cases[-(wdw+1):-1].values > 0)

    return(alert_zone & data_increase)


def contingency_level(year: int, territory_id: int):
    
    if alert_trigger(dataset_id=3, year=year, territory_id=territory_id):
        return(3)
    elif alert_trigger(dataset_id=2, year=year, territory_id=territory_id):
        return(2)
    elif alert_trigger(dataset_id=1, year=year, territory_id=territory_id):
        return(1)
    else:
        return(0)

contingency_name_from_id[contingency_level(year=year, territory_id=territory_id)]
inputs: # Fill in your code here.

Given the above code, generate some markdown instructions for it.

targets: 
Let's now try drawing the graph.
Exercise
Use nx.draw(my_graph) to draw the filtered graph to screen.
inputs: #Xtest = np.vstack([ testdatacand['ug'], testdatacand['gr'], testdatacand['ri'], testdatacand['iz'], testdatacand['zs1'], testdatacand['s1s2']]).T
Xtest = np.vstack([np.asarray(testdatacand[name]) for name in ['ug', 'gr', 'ri', 'iz', 'zs1', 's1s2']]).T

XStest = scaler.transform(Xtest)  # apply same transformation to test data
Given the above code, generate some markdown instructions for it.

targets: 
Scale the test data
inputs: %load solutions/random_number.py
Given the above code, generate some markdown instructions for it.

targets: 
Exercise: Random numbers
Generate a NumPy array of 1000 random numbers sampled from a Poisson distribution, with parameter lam=5. What is the modal value in the sample?
inputs: # dense input
Ydense  = np.random.rand(10, 20)
trainSession = smurff.TrainSession(burnin = 5, nsamples = 5)
trainSession.addTrainAndTest(Ydense)
trainSession.run()
Given the above code, generate some markdown instructions for it.

targets: 
Input to SMURFF
In this notebook we will look at how to provide input to SMURFF with dense and sparse matrices;
SMURFF accepts the following matrix files for train, test and side-info data:

for dense matrix or tensor input:
  numpy.ndarrays
for sparse matrices input:
  scipy Sparse matrices in COO, CSR or CSC format
for sparse tensors: a wrapper around a
  pandas.DataFrame

Let's have a look on how this could work.
Dense Train Input
inputs: "4.2. Create Finite Element space (Lagrange P1)"
Please write code following the instructions in jupyter notebook style.

targets: 
V = FunctionSpace(mesh, "Lagrange", 1)
inputs: !bq show --format=prettyjson -j fae46669-5e96-4744-9d2c-2b1b95fa21e7
Given the above code, generate some markdown instructions for it.

targets: 
Be sure to replace the job id with your own most recent job.
inputs: I am working on the file "02_basics_variables_ex2.ipynb".
The first task is:
Variable State
Can you write Python code for it?

targets: 
>>> session1 = tf.Session(graph=graph)
>>> session2 = tf.Session(graph=graph)
>>> x.initializer.run(session=session1)
>>> x.initializer.run(session=session2)

>>> increment_op.eval(session=session1)

>>> x.eval(session=session1)

>>> x.eval(session=session2)

>>> session1.close()
>>> session2.close()
inputs: 3rd q ring 0.00288861 (1/ Angstroms)

targets: fig, axes = plt.subplots(figsize=(6, 6))
axes.set_xlabel("K/<K>")
axes.set_ylabel("P(K)")
for i in range(3):
    art, = axes.plot(norm_bin_edges[i, 2][:-1], p_K[i, 2], '-o', label=str(time_steps_5[i])+" ms" )
    axes.set_xlim(0, 4)
    axes.legend()
plt.title("3rd q ring 0.00288861 (1/ Angstroms)")
plt.show()
inputs: "Comment:
Suprisingly, the cap color is not the great predictor of etability of a mushroom. Different colors have varied distributions among both classes. Just a few species with red and yellow caps tend to be poisonous more often, while brown, gray and white caps are more prevailant among etable mushrooms. However, one cannot draw any definitive conclusion and we need to investgate more charactristics."
Please write code following the instructions in jupyter notebook style.

targets: 
gills = pd.crosstab(index=df['classif'], columns=df["gill_color"])
print(gills)
#gill-color: black=k,brown=n,buff=b,chocolate=h,gray=g,green=r,orange=o,pink=p,purple=u,red=e, white=w,yellow=y
s1 = pd.crosstab(index=df['classif'], columns = df['gill_size'])
print(s1)
inputs: 5.4. Plotting the confidence intervals
On top of the previous figure (copy here your code from the previous section), plot functions
$${\mathbb E}\left{f({\bf x}^\ast)\mid{\bf s}\right}$$
and
$${\mathbb E}\left{f({\bf x}^\ast)\mid{\bf s}\right} \pm 2 \sqrt{{\mathbb V}\left{f({\bf x}^\ast)\mid{\bf s}\right}}$$
(i.e., the posterior mean of $f({\bf x}^\ast)$, as well as two standard deviations above and below).
It is possible to show analytically that this region comprises $95.45\%$ probability of the posterior probability $p(f({\bf x}^\ast)\mid {\bf s})$ at each ${\bf x}^\ast$.

targets: # Note that you can re-use code from sect. 4.2 to solve this exercise

# Plot the training points 
# plt.plot(X, Z.dot(true_w), 'b', label='True model', linewidth=2);
plt.plot(X0train, strain,'b.',markersize=2);
plt.xlim(xmin, xmax);
# </SOL>    

# Plot the posterior mean.
# mean_s = <FILL IN>
plt.plot(X, mean_s, 'g', label='Predictive mean', linewidth=2);

# Plot the posterior mean +- two standard deviations
# std_f = <FILL IN>

# Plot the confidence intervals.
# To do so, you can use the fill_between method
plt.fill_between(X.flatten(), (mean_s - 2*std_f).flatten(), (mean_s + 2*std_f).flatten(),
                 alpha=0.4, edgecolor='#1B2ACC', facecolor='#089FFF', linewidth=2)

# plt.legend(loc='best')
plt.xlabel('$x$',fontsize=14);
plt.ylabel('$s$',fontsize=14);
plt.show()
inputs: I am working on the file "1. Learning from data and optimization.ipynb".
The first task is:
Learning from data
In general, we have:

A dataset ${(\mathbf{x},y)}$ of $n$ examples. 
A target function $f_\mathbf{w}$, that we want to minimize, representing the discrepancy between our data and the model we want to fit. The model is represented by a set of parameters $\mathbf{w}$. 
The gradient of the target function, $g_f$. 

In the most common case $f$ represents the errors from a data representation model $M$. 
For example, to fit the model clould be to find the optimal parameters $\mathbf{w}$ that minimize the following expression:
$$ f_\mathbf{w} = \frac{1}{n} \sum_{i} (y_i - M(\mathbf{x}_i,\mathbf{w}))^2 $$
For example, $(\mathbf{x},y)$ can represent:

$\mathbf{x}$: the behavior of a "Candy Crush" player; $y$: monthly payments. 
$\mathbf{x}$: sensor data about your car engine; $y$: probability of engine error.
$\mathbf{x}$: finantial data of a bank customer; $y$: customer rating.


If $y$ is a real value, it is called a regression problem.
If $y$ is binary/categorical, it is called a classification problem. 

Let's suppose that our model is a one-dimensional linear model $M(\mathbf{x},\mathbf{w}) = w \cdot x $. 
Batch gradient descend
We can implement gradient descend in the following way (batch gradient descend):
Can you write Python code for it?

targets: 
import numpy as np
import random

# f = 2x
x = np.arange(10)
y = np.array([2*i for i in x])

# f_target = 1/n Sum (y - wx)**2
def target_f(x,y,w):
    return np.sum((y - x * w)**2.0) / x.size

# gradient_f = 2/n Sum 2wx**2 - 2xy
def gradient_f(x,y,w):
    return 2 * np.sum(2*w*(x**2) - 2*x*y) / x.size

def step(w,grad,alpha):
    return w - alpha * grad

def BGD_multi_step(target_f, 
                   gradient_f, 
                   x, 
                   y, 
                   toler = 1e-6):
    
    alphas = [100, 10, 1, 0.1, 0.001, 0.00001]
    w = random.random()
    val = target_f(x,y,w)
    i = 0
    while True:
        i += 1
        gradient = gradient_f(x,y,w)
        next_ws = [step(w, gradient, alpha) for alpha in alphas]
        next_vals = [target_f(x,y,w) for w in next_ws]
        min_val = min(next_vals)
        next_w = next_ws[next_vals.index(min_val)]   
        next_val = target_f(x,y,next_w)    
        if (abs(val - next_val) < toler):
            return w
        else:
            w, val = next_w, next_val
            
print '{:.6f}'.format(BGD_multi_step(target_f, gradient_f, x, y))

%%timeit
BGD_multi_step(target_f, gradient_f, x, y)

def BGD(target_f, gradient_f, x, y, toler = 1e-6, alpha=0.01):
    w = random.random()
    val = target_f(x,y,w)
    i = 0
    while True:
        i += 1
        gradient = gradient_f(x,y,w)
        next_w = step(w, gradient, alpha)
        next_val = target_f(x,y,next_w)    
        if (abs(val - next_val) < toler):
            return w
        else:
            w, val = next_w, next_val
            
print '{:.6f}'.format(BGD(target_f, gradient_f, x, y))

%%timeit
BGD(target_f, gradient_f, x, y)
inputs: %matplotlib inline

import matplotlib.pyplot as plt


plt.hist([1,2,3])
plt.show()

import numpy as np

x = np.arange(-20,20,0.1)

y = np.sin(x)

plt.plot(x,y)

a = tf.constant(100)
b = tf.constant(50)

add_op = a + b

v = tf.Variable(0)
let_op = tf.assign(v, add_op)

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    _, v_val = sess.run([let_op,v])
    print(v_val)
Given the above code, generate some markdown instructions for it.

targets: 
마크다운으로 메모 작성하기...!
마크 다운으로 메모를 작성합니다.
이번 절의 목표

jupyter notebook
시각적으로 기계 학습 살펴보기
inputs: I am working on the file "06_image-classification/dlnd_image_classification.ipynb".
The first task is:
Fully-Connected Layer
Implement the fully_conn function to apply a fully connected layer to x_tensor with the shape (Batch Size, num_outputs). Shortcut option: you can use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages for this layer. For more of a challenge, only use other TensorFlow packages.
Can you write Python code for it?

targets: 
def fully_conn(x_tensor, num_outputs):
    """
    Apply a fully connected layer to x_tensor using weight and bias
    : x_tensor: A 2-D tensor where the first dimension is batch size.
    : num_outputs: The number of output that the new tensor should be.
    : return: A 2-D tensor where the second dimension is num_outputs.
    """
    height = int(x_tensor.shape[1])
    deviation = 1/np.sqrt(num_outputs)
    
    W = tf.Variable(tf.truncated_normal([height, num_outputs],
                                        mean=0.0, stddev=deviation))
    b = tf.Variable(tf.truncated_normal([num_outputs],
                                        mean=0.0, stddev=deviation))
    fc1 = tf.add(tf.matmul(x_tensor, W), b)
    return tf.nn.relu(fc1)

tests.test_fully_conn(fully_conn)
inputs: %%cython

import numpy as np

cimport cython
cimport numpy as np

def feed_backward_comb_filter(signal, tau, alpha):
    """
    Filter the signal with a feed backward comb filter.

    :param signal: signal
    :param tau:    delay length
    :param alpha:  scaling factor
    :return:       comb filtered signal

    """
    if signal.ndim == 1:
        return feed_backward_comb_filter_1d(signal, tau, alpha)
    elif signal.ndim == 2:
        return feed_backward_comb_filter_2d(signal, tau, alpha)
    else:
        raise ValueError('signal must be 1d or 2d')

@cython.boundscheck(False)
def feed_backward_comb_filter_1d(np.ndarray[np.float_t, ndim=1] signal,
                                 unsigned int tau,
                                 float alpha):
    """
    Filter the signal with a feed backward comb filter.

    :param signal: signal
    :param tau:    delay length
    :param alpha:  scaling factor
    :return:       comb filtered signal

    """
    # y[n] = x[n] + α * y[n - τ]
    if tau <= 0:
        raise ValueError('tau must be greater than 0')
    # type definitions
    cdef np.ndarray[np.float_t, ndim=1] y = np.copy(signal)
    cdef unsigned int n
    # loop over the complete signal
    for n in range(tau, len(signal)):
        # add a delayed version of the output signal
        y[n] += alpha * y[n - tau]
    # return the filtered signal
    return y

@cython.boundscheck(False)
def feed_backward_comb_filter_2d(np.ndarray[np.float_t, ndim=2] signal,
                                 unsigned int tau,
                                 float alpha):
    """
    Filter the signal with a feed backward comb filter.

    :param signal: signal
    :param tau:    delay length
    :param alpha:  scaling factor
    :return:       comb filtered signal

    """
    # y[n] = x[n] + α * y[n - τ]
    if tau <= 0:
        raise ValueError('tau must be greater than 0')
    # type definitions
    cdef np.ndarray[np.float_t, ndim=2] y = np.copy(signal)
    cdef unsigned int d, n
    # loop over the dimensions
   ...
Given the above code, generate some markdown instructions for it.

targets: 
Ok, we are now on par with the feed forward Numpy variant -- or even a bit better :)
To get back the flexibility of the Python/Numpy solution to be able to handle signals of arbitrary dimension, we need to define a wrapper function (in pure Python):
inputs: # Define function hyperparameters
batch_size = 
img_size = 

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
# Call your function and get a dataloader
celeba_train_loader = get_dataloader(batch_size, img_size)

Given the above code, generate some markdown instructions for it.

targets: 
Create a DataLoader
Exercise: Create a DataLoader celeba_train_loader with appropriate hyperparameters.
Call the above function and create a dataloader to view images. 
* You can decide on any reasonable batch_size parameter
* Your image_size must be 32. Resizing the data to a smaller size will make for faster training, while still creating convincing images of faces!