Muennighoff/xP3x-sample huggingface dataset:
inputs: I am working on the file "Game of Life.ipynb".
The first task is:
The iteration step
Z: Should be a matrix of ones and zeros which represents the state of the board. The board will then be iterated and returned as a matrix of ones and zeros.

This function will add in the randomness according to the parameters listed above.
Can you write Python code for it?

targets: 
def iterate(Z):
    # Count neighbours
    N = (Z[0:-2,0:-2] + Z[0:-2,1:-1] + Z[0:-2,2:] +
         Z[1:-1,0:-2]                + Z[1:-1,2:] +
         Z[2:  ,0:-2] + Z[2:  ,1:-1] + Z[2:  ,2:])

    # Apply rules
    birth = (N==3) & (Z[1:-1,1:-1]==0)
    survive = ((N==2) | (N==3)) & (Z[1:-1,1:-1]==1)
    # Add randomess
    R_1 = (N==1) * (np.random.randint(0,k_1, np.array(Z.shape) - np.array([2,2])) == 0)
    R_2 = (N==2) * (np.random.randint(0,k_2, np.array(Z.shape) - np.array([2,2])) == 0)
    R_3 = (N==3) * (np.random.randint(0,k_3, np.array(Z.shape) - np.array([2,2])) == 0)
    R_4 = (N==4) * (np.random.randint(0,k_4, np.array(Z.shape) - np.array([2,2])) == 0)   
    Z[...] = 0
    Z[1:-1,1:-1][birth | survive] = 1
    Z[1:-1,1:-1][R_1] = 1
    Z[1:-1,1:-1][R_2] = 0
    Z[1:-1,1:-1][R_3] = 0
    Z[1:-1,1:-1][R_2] = 1
    return Z
inputs: I am working on the file "17-09-27-AWS Machine Learning A Complete Guide With Python/04 - Linear Regression/02 - ml_linear_examples.ipynb".
The first task is:
<h4>AWS ML Estimated/Predicted</h4>
Estimated/Predicted value is reported as 'score'. <br>
Tag is the row identifier
Can you write Python code for it?

targets: 
df_predicted.head()

df_predicted.columns = ["Row", "y_predicted"]

df_predicted.index = df_predicted.Row

df_predicted.head()
inputs: Average CTR
| Column  |  Description |
|---|---|
|avg_ctr_ad_id, avg_ctr_publisher_id, avg_ctr_advertiser_id, avg_ctr_campain_id, avg_ctr_entity_id_country … | Average CTR (#clicks / #views) given some categorical combinations and CTR confidence (details on Part II post). Eg. P(click  category01, category02).

targets: events.cache()
clicks_train.cache()
documents_meta.cache()
promoted_contents.cache()

avg_ctrs_by_ad_id = clicks_train.groupBy("ad_id").agg(F.avg("clicked").alias("avg_ctr_by_ad_id"))
%show avg_ctrs_by_ad_id

avg_ctrs_by_campaign_id = (clicks_train
.join(promoted_contents, on = "ad_id")
.groupBy("campaign_id")
.agg(F.avg("clicked").alias("avg_ctr_by_campaign_id")))

%show avg_ctrs_by_campaign_id

avg_ctrs_by_advertiser_id = (clicks_train
.join(promoted_contents, on = "ad_id")
.groupBy("advertiser_id")
.agg(F.avg("clicked").alias("avg_ctr_by_advertiser_id"))
.cache()
)

%show avg_ctrs_by_advertiser_id

avg_ctrs_by_document_id = (clicks_train
.join(promoted_contents, on = "ad_id")
.groupBy("document_id")
.agg(F.avg("clicked").alias("avg_ctr_by_document_id"))
.cache()
)

%show avg_ctrs_by_document_id

avg_ctrs_by_time = (events
 .join(documents_meta, on = "document_id", how = "left")
 .join(clicks_train, on = "display_id", how = "left")
 .join(promoted_contents, on = "ad_id", how = "left")
 .withColumn("total_clicks_by_ad_id"
    , F.expr("SUM(clicked) OVER (PARTITION BY ad_id ORDER BY timestamp ROWS BETWEEN UNBOUNDED PRECEDING AND -1 FOLLOWING)"))
 .withColumn("total_events_by_ad_id"
    , F.expr("COUNT(*) OVER (PARTITION BY ad_id ORDER BY timestamp ROWS BETWEEN UNBOUNDED PRECEDING AND -1 FOLLOWING)"))

 .withColumn("total_clicks_by_advertiser_id"
    , F.expr("SUM(clicked) OVER (PARTITION BY advertiser_id ORDER BY timestamp ROWS BETWEEN UNBOUNDED PRECEDING AND -1 FOLLOWING)"))
 .withColumn("total_events_by_advertiser_id"
    , F.expr("COUNT(*) OVER (PARTITION BY advertiser_id ORDER BY timestamp ROWS BETWEEN UNBOUNDED PRECEDING AND -1 FOLLOWING)"))

 .withColumn("total_clicks_by_campaign_id"
    , F.expr("SUM(clicked) OVER (PARTITION BY campaign_id ORDER BY timestamp ROWS BETWEEN UNBOUNDED PRECEDING AND -1 FOLLOWING)"))
 .withColumn("total_events_by_campaign_id"
    , F.expr("COUNT(*) OVER (PARTITION BY campaign_id ORDER BY timestamp ROWS BETWEEN UNBOUNDED PRECEDING AND -1 FOLLOWING)"))

...
inputs: #the LJ potential with the central bead keeps all the beads from simply collapsing into the center
system.non_bonded_inter[1, 0].lennard_jones.set_params(
    epsilon=eps_cs, sigma=sig_cs,
    cutoff=sig_cs*np.power(2.,1./6.), shift="auto")
#the LJ potential (WCA potential) between surface beads causes them to be roughly equidistant on the colloid surface
system.non_bonded_inter[1, 1].lennard_jones.set_params(
    epsilon=eps_ss, sigma=sig_ss,
    cutoff=sig_ss*np.power(2.,1./6.), shift="auto")

#the harmonic potential pulls surface beads towards the central colloid bead
col_center_surface_bond = interactions.HarmonicBond(k=3000., r_0=harmonic_radius)
system.bonded_inter.add(col_center_surface_bond)

Given the above code, generate some markdown instructions for it.

targets: 
The $periodicity$ parameter indicates that the system is periodic in all three
dimensions. Note that the lattice-Boltzmann algorithm requires periodicity in all three directions (although
this can be modified using boundaries, a topic not covered in this tutorial). 
4 Setting up the Raspberry
Setting up the raspberry is a non-trivial task. The main problem lies in creating a relatively
uniform distribution of beads on the surface of the colloid. In general one should take about 1 bead per lattice-Boltzmann grid
point on the surface to ensure that there are no holes in the surface. The behavior of the colloid can be further improved by placing
beads inside the colloid, though this is not done in this example script. In our example
we first define a harmonic interaction causing the surface beads to be attracted
to the center, and a Lennard-Jones interaction preventing the beads from entering the colloid. There is also a Lennard-Jones
potential between the surface beads to get them to distribute evenly on the surface.
inputs: I am working on the file "Notebooks/Tilting_Estimates_of_ATT.ipynb".
The first task is:
Hide Runtime warning messages to clean up output.
Can you write Python code for it?

targets: 
import warnings

def fxn():
    warnings.warn("runtime", RuntimeWarning)

with warnings.catch_warnings():
    warnings.simplefilter("ignore")
    fxn()
inputs: I am working on the file "Lab 7 - Time Evolution.ipynb".
The first task is:
The next line calls a Schrödinger equation solver (sesolve). The arguments are the Hamiltonian, the starting state $\lvert+z\rangle$ (pz), the time values, a list of operators, and the arguments to the function H_t. sesolve returns many things, but the expect method is most useful, it gives the expectation values of the four operators in the operator list. Notice the fourth operator is the $\lvert-z\rangle$ projection operator. It's expectation value is $P(\lvert-z\rangle,t)$
Can you write Python code for it?

targets: 
result5 = sesolve(H, pz, t, [Sx, Sy, Sz, mz*mz.dag()],args)
sx, sy, sz, Pmz = result5.expect
inputs: <h3>Matritzen Multiplikation</h3>

targets: ax = np.arange(10)
ay = np.array([ax,ax])
#Skalar multiplikation
ay*2

np.dot(ay,ay.reshape(10,2)) #Dot product
inputs: <br>
<br>
Increasing point size with distance from the origin
[back to top]

targets: import numpy as np
import matplotlib.pyplot as plt

fig = plt.figure(figsize=(8,6))

# Generating a Gaussion dataset:
# creating random vectors from the multivariate normal distribution 
# given mean and covariance 
mu_vec1 = np.array([0,0])
cov_mat1 = np.array([[1,0],[0,1]])
X = np.random.multivariate_normal(mu_vec1, cov_mat1, 500)

R = X**2
R_sum = R.sum(axis=1)
plt.scatter(X[:, 0], X[:, 1], 
            color='gray', 
            marker='o', 
            s=32. * R_sum,
            edgecolor='black',
            alpha=0.5)
plt.show()
inputs: # when running from the terminal
# python go_webserver.py

# here we load the server as a subprocess for demonstration purposes
webserver = subprocess.Popen(['python', '../go_webserver.py'])
time.sleep(5)  # make sure it loads completely
Given the above code, generate some markdown instructions for it.

targets: 
Load the database webserver.
inputs: # Add a 2nd polygon to "P"
xpts = [14,14,16,16]
ypts = [0,2,2,0]
P.add_polygon([xpts,ypts], layer = 1)

qp(P) # Quickplot the "P" with its 2 polygons
Given the above code, generate some markdown instructions for it.

targets: 
Now we're getting somewhere!  We've only had to make the polygon once, but we're able to reuse it as many times as we want.
Modifying the referenced geometry
Now, a question naturally follows this:  What happens when you change the original geometry that the reference points to?  In our case, our references in D all point to the Device P that with the original polygon.  Let's try adding a second polygon to P.
First we add the second polygon and make sure P looks like we expect:
inputs: 2°) Fitting the other cost data
In this section, we fit the data generated by the quadratic, exponential, and logarithmic cost function and recover the parameters $\text{RC}$ and $\theta_{11}$ and $\theta_{12}$. We do not illustrate the impact of misspecification for those cost functions.
A. Fitting the true quadratic cost function
We start by initializing a DynamicLogit instance, and fit the likelihood function to the data. 
Recall that in the quadratic model:
$$\text{RC} = 20$$
$$\theta_{11} = 0.5$$
$$\theta_{12} = 0.01$$

targets: quad_to_quad = DynamicLogit(quad_dataframe, "Choice", "State", p, quad_cost, npars=3)
quad_to_quad.fit_likelihood(x0=[20, 1, 1])
inputs: I am working on the file "0101_GettingStartedWithPython.ipynb".
The first task is:
A few imprtant notes:
* Indentation is not just here for better readability of the code but it is actually necessary for Python to correctly interpret the code.
* Though it is not necessary, we initiate seqSum = 0 here. Otherwise, if we run the code repeatedly we add to the previous total!
* value takes on every value in array seq. In the first loop value=10, second loop value=9, etc. 
Loops can be nested, too. Here's an example.
Can you write Python code for it?

targets: 
seq = seq.reshape(2, 5)
seq

seqSum = 0
row = seq.shape[0]
col = seq.shape[1]

for rowIndex in range(0, row):
    for colIndex in range(0, col):
        seqSum = seqSum + seq[rowIndex, colIndex]
        
seqSum
inputs: I am working on the file "Lecture-14-Ordinary-Differential-Equations.ipynb".
The first task is:
We substitute the initial condition and then solve for C1.  Afterwards we substitute C1 back into the general solution and plot the result.
Can you write Python code for it?

targets: 
particularSolution = solutionToEquation.subs([(current(t),0),(t,0)])
particularSolution

solutionSet = solveset(particularSolution,C1)
solutionList = [a for a in solutionSet]
solutionSet
inputs: I am working on the file "3 - Introducing Pandas.ipynb".
The first task is:
Count the unique values:
Can you write Python code for it?

targets: 
months = gig_months.value_counts()
months

# matplotlib has lots of options for customization
ax = months.sort_index().plot.bar(table=True, figsize=(10,5))
ax.get_xaxis().set_visible(False)
inputs: As you may have noticed, the vectorized implementation is much cleaner and more efficient. For bigger vectors/matrices, the differences in running time become even bigger. 
Note that np.dot() performs a matrix-matrix or matrix-vector multiplication. This is different from np.multiply() and the * operator (which is equivalent to  .* in Matlab/Octave), which performs an element-wise multiplication.
2.1 Implement the L1 and L2 loss functions
Exercise: Implement the numpy vectorized version of the L1 loss. You may find the function abs(x) (absolute value of x) useful.
Reminder:
- The loss is used to evaluate the performance of your model. The bigger your loss is, the more different your predictions ($ \hat{y} $) are from the true values ($y$). In deep learning, you use optimization algorithms like Gradient Descent to train your model and to minimize the cost.
- L1 loss is defined as:
$$\begin{align} & L_1(\hat{y}, y) = \sum_{i=0}^m|y^{(i)} - \hat{y}^{(i)}| \end{align}\tag{6}$$

targets: # GRADED FUNCTION: L1

def L1(yhat, y):
    """
    Arguments:
    yhat -- vector of size m (predicted labels)
    y -- vector of size m (true labels)
    
    Returns:
    loss -- the value of the L1 loss function defined above
    """
    
    ### START CODE HERE ### (≈ 1 line of code)
    diff = np.abs(np.subtract(yhat, y))
    loss = np.sum(diff)
    ### END CODE HERE ###
    
    return loss

yhat = np.array([.9, 0.2, 0.1, .4, .9])
y = np.array([1, 0, 0, 1, 1])
print("L1 = " + str(L1(yhat,y)))
inputs: "Build the Neural Network
You'll build the components necessary to build a RNN by implementing the following functions below:
- get_inputs
- get_init_cell
- get_embed
- build_rnn
- build_nn
- get_batches
Check the Version of TensorFlow and Access to GPU"
Please write code following the instructions in jupyter notebook style.

targets: 
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
from distutils.version import LooseVersion
import warnings
import tensorflow as tf

# Check TensorFlow Version
assert LooseVersion(tf.__version__) >= LooseVersion('1.0'), 'Please use TensorFlow version 1.0 or newer'
print('TensorFlow Version: {}'.format(tf.__version__))

# Check for a GPU
if not tf.test.gpu_device_name():
    warnings.warn('No GPU found. Please use a GPU to train your neural network.')
else:
    print('Default GPU Device: {}'.format(tf.test.gpu_device_name()))

def get_inputs():
    """
    Create TF Placeholders for input, targets, and learning rate.
    :return: Tuple (input, targets, learning rate)
    """
    inputs = tf.placeholder(tf.int32, shape=(None, None), name="input")
    targets = tf.placeholder(tf.int32, shape=(None, None), name="target")
    learning_rate = tf.placeholder(tf.float32, name="learning_rate")
    return inputs, targets, learning_rate


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_get_inputs(get_inputs)
inputs: I am working on the file "03-NLP/word-vectors.ipynb".
The first task is:
The arrows are different since we have selected two specific dimensions and not the reduced PCA ones and words are more spread, not necessarily clustered together.  
But note that similar words like 'village' and 'town' or 'country' and 'continent' tend to point in the same direction.
Also, note that 'sad' and 'happy' looks close to each other; however, the vectors point in opposite directions.
In this chart, one can figure out the angles and distances between the words. Some words are close in both kinds of similarity metrics.
Word distance
Now we will explore another peculiarity of the word vectors: how the distance between two vectors give an indication of the words similarity.
We plot the words 'sad', 'happy', 'town', and 'village' as in the previous chart. In this same chart, display the vector from 'village' to 'town' and the vector from 'sad' to 'happy'.
Can you write Python code for it?

targets: 
words = ['sad', 'happy', 'town', 'village']

embeddedWords = np.array([getVector(word) for word in words]) # Convert each word to its vector representation

fig, ax = plt.subplots(figsize = (10, 10)) # Create custom size image

col1 = 3 # Select the column for the x axe
col2 = 2 # Select the column for the y axe

# Print an arrow for each word
for word in embeddedWords:
    ax.arrow(0, 0, word[col1], word[col2], head_width=0.0005, head_length=0.0005, fc='r', ec='r', width = 1e-5)
    
# print the vector difference between village and town
village = getVector('village')
town = getVector('town')
diff = town - village
ax.arrow(village[col1], village[col2], diff[col1], diff[col2], fc='b', ec='b', width = 1e-5)

# print the vector difference between village and town
sad = getVector('sad')
happy = getVector('happy')
diff = happy - sad
ax.arrow(sad[col1], sad[col2], diff[col1], diff[col2], fc='b', ec='b', width = 1e-5)


ax.scatter(embeddedWords[:, col1], embeddedWords[:, col2]); # Plot a dot for each word

# Add the word label over each dot in the scatter plot
for i in range(0, len(words)):
    ax.annotate(words[i], (embeddedWords[i, col1], embeddedWords[i, col2]))


plt.show()
inputs: # geocode an address to lat-lng
address = '704 S Alvarado St, Los Angeles, California'
latlng = ox.geocode(address)
latlng
Given the above code, generate some markdown instructions for it.

targets: 
1b: Geocoding addresses to lat-lng
You can geocode addresses as well with OpenStreetMap, but it can be a little hit-or-miss compared to the data coverage of commercial closed-source services.
inputs: "Before we show any results we configure we further configure the graphic, by changing the appearance of the simulated lines. We can obtain line objects from any graphics instance with the result_lines property:"
Please write code following the instructions in jupyter notebook style.

targets: 
sim_graphics.result_lines
inputs: "A présent la base df_pocket est nettoyée et prête à être utilisée pour les analyses de textes. 
Analyse des données textuelles - TD-IDF, similarité cosine et n-grams
Le calcul tf-idf (term frequency–inverse document frequency) permet de calculer un score de proximité entre un terme de recherche et un document (c'est ce que font les moteurs de recherche). 
La partie tf calcule une fonction croissante de la fréquence du terme de recherche dans le document à l'étude, la partie idf calcule une fonction inversement proportionnelle à la fréquence du terme dans l'ensemble des documents (ou corpus). 
Le score total, obtenu en multipliant les deux composantes, permet ainsi de donner un score d'autant plus élevé que le terme est surréprésenté dans un document (par rapport à l'ensemble des documents). Il existe plusieurs fonctions, qui pénalisent plus ou moins les documents longs, ou qui sont plus ou moins smooth."
Please write code following the instructions in jupyter notebook style.

targets: 
corpus = { 
 'a' : "Mr. Green killed Colonel Mustard in the study with the candlestick. \
Mr. Green is not a very nice fellow.",
 'b' : "Professor Plum has a green plant in his study.",
 'c' : "Miss Scarlett watered Professor Plum's green plant while he was away \
from his office last week."
}
terms = {
 'a' : [ i.lower() for i in corpus['a'].split() ],
 'b' : [ i.lower() for i in corpus['b'].split() ],
 'c' : [ i.lower() for i in corpus['c'].split() ]
 }

from math import log

QUERY_TERMS = ['mr.', 'green']

def tf(term, doc, normalize=True):
    doc = doc.lower().split()
    if normalize:
        return doc.count(term.lower()) / float(len(doc))
    else:
        return doc.count(term.lower()) / 1.0


def idf(term, corpus):
    num_texts_with_term = len([True for text in corpus if term.lower() \
                              in text.lower().split()])
    try:
        return 1.0 + log(float(len(corpus)) / num_texts_with_term)
    except ZeroDivisionError:
        return 1.0
    
def tf_idf(term, doc, corpus):
    return tf(term, doc) * idf(term, corpus)

for (k, v) in sorted(corpus.items()):
    print(k, ':', v)
print('\n')

query_scores = {'a': 0, 'b': 0, 'c': 0}
for term in [t.lower() for t in QUERY_TERMS]:
    for doc in sorted(corpus):
        print('TF({}): {}'.format(doc, term), tf(term, corpus[doc]))
    print('IDF: {}'.format(term, ), idf(term, corpus.values()))
    print('\n')
    for doc in sorted(corpus):
        score = tf_idf(term, corpus[doc], corpus.values())
        print('TF-IDF({}): {}'.format(doc, term), score)
        query_scores[doc] += score
    print('\n')

print("Score TF-IDF total pour le terme '{}'".format(' '.join(QUERY_TERMS), ))
for (doc, score) in sorted(query_scores.items()):
    print(doc, score)
inputs: #read json files
with open('pageviews_desktop_201507-201709.json','r') as f:
    data_1 = json.load(f)
with open('pageviews_mobile-app_201507-201709.json','r') as f:
    data_2 = json.load(f)
with open('pageviews_mobile-web_201507-201709.json','r') as f:
    data_3= json.load(f)
with open('pagecounts_desktop-site_200801-201607.json','r') as f:
    data_4 = json.load(f)
with open('pagecounts_mobile-site_200801-201607.json','r') as f:
    data_5 = json.load(f)
    

#load json in datapfram
df_1 = pd.DataFrame(data_1['items'])
df_2 = pd.DataFrame(data_2['items'])
df_3 = pd.DataFrame(data_3['items'])
df_4 = pd.DataFrame(data_4['items'])
df_5 = pd.DataFrame(data_5['items'])
Given the above code, generate some markdown instructions for it.

targets: 
Step 2: Data Processing
We need to do somt proccessing on these json file data inorder to analysis. I conbine monthly mobile-app and monthly mobile-web to total mobile for pageviews.
First, read the json file sepreatedly
inputs: I am working on the file "Lectures/elegent_exception_handling/Exceptions Handling.ipynb".
The first task is:
We Can Proud of Ourselves 💃


Implemented restaurant recommendation 💪


Clean code 💄


<img src="https://github.com/eyaltrabelsi/my-notebooks/raw/master/Lectures/unicorn_rainbow.png" width="400"/>
Why Exception Handling? 🤨


Hardware can fail 🌲


Software often fail 🚪


How complex systems fail 🧩


<img src="https://github.com/eyaltrabelsi/my-notebooks/raw/master/Lectures/elegent_exception_handling/unicorn.png" width="400"/>
Unexceptable 😡
Lesson 1: We want to build a fault tolerant system.
Types of errors

error that can be detected at compile time
errors that can be deteled at run time
errors that can be infered
reproducieable erros
non reproduceable errors

We want our code to be safe 👷
Exception Handling to the Rescue 👨‍🚒


Detect errors 🕵


Do something about them 🔒


Naive Approach 👶


Log all exceptions 📝


Ignore all exceptions 🙈
Can you write Python code for it?

targets: 
def get_restaurant_recommendation(path):
    try:
        user = get_user(path)
        candidates = get_relevant_restaurants(user)
        pick_best_restaurants(candidates)
    except BaseException:
        logging.error("VERY UNINFORMATIVE INFORMATION")
        raise BaseException("VERY UNINFORMATIVE INFORMATION")
inputs: # Read drinks.csv (in the 'drinks_data' folder) into a DataFrame called 'drinks'

# Print the first 10 rows

# Examine the data types of all columns

# Print the 'beer_servings' Series

# Calculate the average 'beer_servings' for the entire dataset

# Print all columns, but only show rows where the country is in Europe

# Calculate the average 'beer_servings' for all of Europe

# Only show European countries with 'wine_servings' greater than 300

# Determine which 10 countries have the highest 'total_litres_of_pure_alcohol'

# Determine which country has the highest value for 'beer_servings'

# Count the number of occurrences of each 'continent' value and see if it looks correct

# Determine which countries do not have continent designations

# Determine the number of countries per continent. Does it look right?
Given the above code, generate some markdown instructions for it.

targets: 
Exercise: Working with the Drinks Data
(Be on the lookout for a curveball question)
inputs: I am working on the file "1-Osnove_programiranja.ipynb".
The first task is:
Osim ovom metodom, elemente liste moguće je brisati putem naredbe del i indeksa elementa:
Can you write Python code for it?

targets: 
lista5=['banana',5,98.95,'tekst',5]
del lista5[-1]
print lista5
inputs: "<div class="text-warning">
    <h3>Some methods not work in a remote experiment</h3>
</div>


session_config()
bot_data()

Raises an NotImplementedError when are called."
Please write code following the instructions in jupyter notebook style.

targets: 
remote.bot_data("matching_pennies", 1)
inputs: I am working on the file "04 Regression & Comparing Models.ipynb".
The first task is:
Note: the MSE goes down because of the log10 transform. However, that's exactly what we would have expected by definition.
Can you write Python code for it?

targets: 
mean_squared_error(preds, Y_test)
inputs: I am working on the file "ImportAppleHealthXML.ipynb".
The first task is:
Download the file from Google Drive
Ensure that the file downloaded is the latest file generated
Can you write Python code for it?

targets: 
for file1 in file_list:
        if file1['id'] == selection_id:
            print('Downloading this file: %s, id: %s createDate: %s' % (file1['title'], file1['id'], file1['createdDate']))
            file1.GetContentFile("healthextract/export.zip")
inputs: I am working on the file "Introduction_to_Pandas.ipynb".
The first task is:
Operations can be applied to Series without losing the data structure.
Use bool array to filter Series
Can you write Python code for it?

targets: 
fruit > 50

fruit[fruit > 50]
inputs: I am working on the file "ApacheMXNet/linear_regression_from_scratch.ipynb".
The first task is:
Data iterators
Once we start working with neural networks, we’re going to need to iterate through our data points quickly.
We’ll also want to be able to grab batches of k data points at a time, to shuffle our data.
In MXNet, data iterators give us a nice set of utilities for fetching and manipulating data.
In particular, we’ll work with the simple DataLoader class, that provides an intuitive way to use an ArrayDataset for training models.
We can load X and y into an ArrayDataset, by calling gluon.data.ArrayDataset(X, y).
It’s ok for X to be a multi-dimensional input array (say, of images) and y to be just a one-dimensional array of labels.
The one requirement is that they have equal lengths along the first axis, i.e., len(X) == len(y).  
Given an ArrayDataset, we can create a DataLoader which will grab random batches of data from an ArrayDataset.
We’ll want to specify two arguments.
First, we’ll need to say the batch_size, i.e., how many examples we want to grab at a time.
Second, we’ll want to specify whether or not to shuffle the data between iterations through the dataset.
Can you write Python code for it?

targets: 
batch_size = 4
train_data = gluon.data.DataLoader(gluon.data.ArrayDataset(X, y),
                                  batch_size=batch_size, shuffle=True)
inputs: I am working on the file "02_data_representation/embeddings.ipynb".
The first task is:
From this and the tokenizer we can extract the VOCAB_SIZE, DATASET_SIZE and MAX_LEN.
Can you write Python code for it?

targets: 
VOCAB_SIZE = len(tokenizer.index_word)
VOCAB_SIZE

DATASET_SIZE = tokenizer.document_count
DATASET_SIZE

MAX_LEN = max(len(sequence) for sequence in integerized_titles)
MAX_LEN
inputs: # Pattern shape
ncols = 3
nrows = 2

# Coordinates of the LCOS center in the pattern space
center_x = 100
center_y = 200

# Rotation of the LCOS grid versus the pattern space
rotation = 20

# Pitch for the multi-spot pattern
pitch_x = 80
pitch_y = 60
Given the above code, generate some markdown instructions for it.

targets: 
Pattern parameters:
inputs: # Aquifer properties
kD = 900 # m2/d
S = 0.25 # [-]

# Only for semi-confined (Hantush) flow we need the resistance of the overlying layer
c  = 300 # d
lamb = np.sqrt(kD * c) # characteristic length of semi-confined system.

# Parameters specifying the case
Q = 1200 # m3/d
r = 350 # m
t = 1.2 # d

# compute u from the ordinary parameters
u = r**2 * S / (4 * kD * t)

# compute the drawdown using function exp1 (=exponential integral)
s = Q/(4 * np.pi * kD) * exp1(u)

print('Drawdown s = {:.4g} m'.format(s))
Given the above code, generate some markdown instructions for it.

targets: 
Simple example, just compute the drawdown given $kD$, $S$, $r$ and $t$ using exp1(..)
inputs: I am working on the file "Day_00/02_Strings_and_FileIO/00 Strings in Python.ipynb".
The first task is:
Changing
Can you write Python code for it?

targets: 
some_kaiju = 'Godzilla, Space Godzilla, Mechagodzilla'
print re.sub('Godzilla', 'Gamera', some_kaiju)
print re.sub('(?i)Godzilla', 'Gamera', some_kaiju)
inputs: I am working on the file "Day 1 - Unit 2.2 DataFrame.ipynb".
The first task is:
Q6: How many countries participated in at least 25 Olympic games (100 years)? Return an integer.
Can you write Python code for it?

targets: 
len(og.query("Games >= 25").loc[:, "Country"])
inputs: Cluster subsample visualization

targets: x = features['serum_cholesterol_mg_per_dl']
y = features['resting_blood_pressure']

trace = [go.Scatter(
    x = x,
    y = y,
    name = 'data',
    mode = 'markers',
    hoverinfo = 'text',
    text = ['x: %s<br>y: %s' % (x_i, y_i) for x_i, y_i in zip(x, y)]
)]

layout = go.Layout(
    xaxis = dict({'title': xlab}),
    yaxis = dict({'title': ylab})
)

fig = go.Figure(data=trace, layout=layout)
iplot(fig, layout)
inputs: "1.2
12x12 matrix whose columns sum to 1, whose diagonals are 0 and off-diagonals are strictly positive. Use np.round(mat_name,2) to only have two decimals when printing."
Please write code following the instructions in jupyter notebook style.

targets: 
import numpy as np

s = 12
m = np.ones( (s,s) )
m[np.diag_indices(s)] = 0
for i in range(s):
    m[:,i] /= np.sum(m[:,i])
print(np.round(m,2))
inputs: # http://scikit-learn.org/stable/modules/clustering.html

dataset_array = df.values
print(dataset_array.dtype)  # kmeans needs to be homogeneous data type (here, float64)
print(dataset_array)

# do I need to normalize
# sklearn.preprocessing.StandardScaler
from sklearn import preprocessing

# http://scikit-learn.org/stable/modules/preprocessing.html
# first load scaler and train on given data set
scaler = preprocessing.StandardScaler().fit(df)

scaler.mean_

scaler.scale_

t0 = dt.datetime.utcnow()

# actually do normalization; can be reused for eg a training set
df_scaled = pandas.DataFrame(scaler.transform(df))

print('... finished in {}'.format(dt.datetime.utcnow() - t0))
Given the above code, generate some markdown instructions for it.

targets: 
Scale
inputs: # Figure 3
RMSE_spline_rses = np.sqrt(mean_squared_error(ll_2018.water_predicted, ll_2018.water))
RMSE_poly_rses = np.sqrt(mean_squared_error(dg_2018.water_predicted, dg_2018.water))

print('Dataset 2018 RSES')
print('Standard deviation with the method of LL 2012 is :'+str(np.round(RMSE_spline_rses,2)))
print('Standard deviation with the method of DG 2017 is :'+str(np.round(RMSE_poly_rses,2)))

plt.figure(figsize=(3.25, 6.5))
plt.subplot(2,1,1)

plt.plot(ll_2018.water_predicted,ll_2018.water,"o",mfc="none",mec="k",label="Iron-free")
plt.plot(ll_2018.water_predicted[np.where(data_liste_rses["FeO"]>0)],data_liste_rses["Water, wt%"].loc[data_liste_rses["FeO"]>0],"o",mfc="k",mec="k",label="Iron-bearing")

plt.plot([0,8.5],[0,8.5],"k--",alpha=0.5)
plt.xlabel(r"Raman [H$_2$O] / wt $\%$")
plt.ylabel(r"Reference [H$_2$O] / wt $\%$")
plt.title("A) LL2012 method")
plt.legend()

plt.subplot(2,1,2)
plt.plot(dg_2018.water_predicted,data_liste_rses["Water, wt%"],"s",mfc="none",mec="k",label="Iron-free")
plt.plot(dg_2018.water_predicted[np.where(data_liste_rses["FeO"]>0)],data_liste_rses["Water, wt%"].loc[data_liste_rses["FeO"]>0],"s",mfc="k",mec="k",label="Iron-bearing")

plt.plot([0,8.5],[0,8.5],"k--",alpha=0.5)

plt.xlabel(r"Raman [H$_2$O] / wt $\%$")
plt.ylabel(r"Reference [H$_2$O] / wt $\%$")
plt.title("B) DG2017 method")

plt.tight_layout()

plt.savefig("./Figure3.pdf")
Given the above code, generate some markdown instructions for it.

targets: 
Figure 3
Comparison of the known water content of basalts and Ca aluminosilicate glass standards with the Raman calculated values following the A) LL2012 and B) DG2017 protocols.
inputs: I am working on the file "A Beginners Guide to Python/20. Functions & Namespaces.ipynb".
The first task is:
Why does Python do this?
The main advantage to variable scope and namespaces is that as your code becomes progressively larger you don't have to keep on coming up with new names for objects. It is easy to imagine how constantly coming up with new names would, after a while adversely affect readability and make coding a lot slower (because with so many names you are going to continually forget which one you need to use). 
The other advantage is that if functions cannot change the content of another function (without your express blessing) then the chance for ugly bugs and/or nasty side-effects to occur is reduced.  
Finally, I'd like to point out that its not just functions that their own scope. In the code snippet below I have defined three list-comprehensions (not covered in this course) and printed them out. What I want you to focus on is how I used and reused the letter 'i' as a variable name several times, and that is okay because 'i' is defined in context ('scope') of each individual list comprehension. These means that all these 'i's' are completely separate from each other, and moreover, 'i' is not defined in the 'main program', which explains the NameError.
Can you write Python code for it?

targets: 
a = [i for i in range(10)]
b = [i for i in range(20)]
c = [i for i in "abcde"]

print(a, b, c, sep="\n")
print(i)
inputs: I am working on the file "Assignment1.ipynb".
The first task is:
Let's rename the column names:
Can you write Python code for it?

targets: 
df.columns = ["ID","Clump_Thickness","Size_Uniformity","Shape_Uniformity","Marginal_Adhesion","Epithelial_Size","Bare_Nucleoli","Bland_Chromatin","Normal_Nucleoli","Mitoses","Class"]

# Print the new header names:
df.columns
inputs: # define the integrand 
f_height_of_peak_weighted = seir.height_of_peak_weighted(meanmax)

# define integral bounds
lb = 1. # lower integral bound
ub = 80. # upper integral bound
Given the above code, generate some markdown instructions for it.

targets: 
<a id='section-task1'></a>
Task 1: Height of peak of infection
The height of the peak is the expected maximal number of simultaneously infected individuals 
$$
\mathbb{E}a\left[\max_t N_I(t, \alpha)\right] =\int{\Omega}\max_t N_I(t, \alpha)p(\alpha)\mathrm{d}\alpha.
$$
We use the Bayesian quadrature package of Emukit to emulate the integrand $\max_tN_I(t, \alpha)p(\alpha)$. 
Define user function $\max_t N_I(t, \alpha)p(\alpha)$ and integral bounds
We call the integrand $\max_t N_I(t, \alpha)p(\alpha)$ f_height_of_peak_weighted  since it computes the height of the infection peak weighted with the prior on $\alpha$. One call of f_height_of_peak_weighted will run num_gil Gillespie runs. The integral bounds have units $\alpha$, i.e., units of infection time divided by recovery time; they define the domain $\Omega = [lower_bounds, upper_bound]$. We choose 80 to be the upper bound since we assume that the infection rate will not be larger than 80 times the recovery rate.
inputs: Correlation metrics
Spearman correlation
Spearman correlation`
answers the simple question, every time $x$ increases, does $y$ increase also? If yes, then spearman correlation = 1.
Mathematically speaking, Spearman tells you whether $x$ and $y$ increase monotonically together (but not necessarily linearly!!)

Pearson correlation
Pearson Correlation answers the question, every time my $x$ decreases by some amount $a$, does $y$ decrease by an amount proportional to that, say $10a$ or $0.5a$, and is this amount constant?
$\rho_{x,y} = \frac{\mathrm{cov}(\vec{x}, \vec{y})}{\sigma_x, \sigma_y}$
Mathematically speaking, pearson tells you whether $x$ and $y$ are linear to each other.

Spearman vs Pearson
Spearman's correlation is related to Pearson because Spearman 
Spearman correlation = Pearson correlation on the ranks of the data.
Anscombe's quartet
Anscombe's quartet is a group of four datasets that have nearly identical statistical properties that we'll use for exploring distance and correlation metrics.

targets: # Read the file - notice it is a URL. pandas can read either URLs or files on your computer
anscombe = pd.read_csv("https://github.com/mwaskom/seaborn-data/raw/master/anscombe.csv")

# Say the variable name with no arguments to look at the data
anscombe
inputs: "As you can imagine, this can be handy when we have a large number of layers for which the actual names are not that meaningful. It also improves readability:"
Please write code following the instructions in jupyter notebook style.

targets: 
class MyCustomModule(nn.Module):
    def __init__(self, n_inputs, n_hidden, n_output_classes):
        super(MyCustomModule, self).__init__()
        self.p_keep = 0.7
        self.network = nn.Sequential(
            nn.Linear(n_inputs, n_hidden),
            nn.ReLU(),
            nn.Linear(n_hidden, 2*n_hidden),
            nn.ReLU(),
            nn.Linear(2*n_hidden, n_output_classes),
            # dropout argument is probability of dropping
            nn.Dropout(1 - self.p_keep),
            # applies softmax in the data dimension
            nn.Softmax(dim=1)
        )

    def forward(self, x):
        y = self.network(x)
        return y
inputs: # Create a pool of workers equaling cores on the machine
pool = ThreadPool() 

%%time

# Apply (map) some_function to the data using the pool of workers
results = pool.map(some_function, data)

# Close the pool
pool.close() 

# Combine the results of the workers
pool.join() 
Given the above code, generate some markdown instructions for it.

targets: 
Parallelism Approach
inputs: "A chance to speed up training and get bonus score

Try predicting next token probas at ALL ticks (like in the seminar part)
much more objectives, much better gradients
You may want to zero-out loss for first several iterations

The New World Order"
Please write code following the instructions in jupyter notebook style.

targets: 
seed = u"Каждый человек должен" #if you are using non-russian text corpora, use seed in it's language instead
sampling_fun = proportional_sample_fun
result_length = 300

generate_sample(sampling_fun,seed,result_length)


seed = u"В случае неповиновения"
sampling_fun = proportional_sample_fun
result_length = 300

generate_sample(sampling_fun,seed,result_length)


And so on at your will
inputs: I am working on the file "Coursera/Machine-learning-data-analysis/Course 3/Week_04/edit_CookingLDA_PA.ipynb".
The first task is:
Для такого большого количества классов это неплохая точность. Вы можете попроовать обучать RandomForest на исходной матрице частот слов, имеющей значительно большую размерность, и увидеть, что accuracy увеличивается на 10–15%. Таким образом, LDA собрал не всю, но достаточно большую часть информации из выборки, в матрице низкого ранга.
LDA — вероятностная модель
Матричное разложение, использующееся в LDA, интерпретируется как следующий процесс генерации документов.
Для документа $d$ длины $n_d$:
1. Из априорного распределения Дирихле с параметром alpha сгенерировать распределение над множеством тем: $\theta_d \sim Dirichlet(\alpha)$
1. Для каждого слова $w = 1, \dots, n_d$:
    1. Сгенерировать тему из дискретного распределения $t \sim \theta_{d}$
    1. Сгенерировать слово из дискретного распределения $w \sim \phi_{t}$.
Подробнее об этом в Википедии.
В контексте нашей задачи получается, что, используя данный генеративный процесс, можно создавать новые рецепты. Вы можете передать в функцию модель и число ингредиентов и сгенерировать рецепт :)
Can you write Python code for it?

targets: 
def generate_recipe(model, num_ingredients):
    theta = np.random.dirichlet(model.alpha)
    for i in range(num_ingredients):
        t = np.random.choice(np.arange(model.num_topics), p=theta)
        topic = model.show_topic(t, topn=model.num_terms)
        topic_distr = [x[1] for x in topic]
        terms = [x[0] for x in topic]
        w = np.random.choice(terms, p=topic_distr)
        print(w)

generate_recipe(lda2, 5)
inputs: I am working on the file "PNC_01Basics.ipynb".
The first task is:
ozone[0, :, 2, 3] * averagekernelm
Can you write Python code for it?

targets: 
ozone[0, :, 2, 3] * averagekernelm.T
inputs: "Addition, multiplication, and subtraction
Similar grids can be added, multiplied and subtracted using standard python operators. It is easily verified that the following sequence of operations return the same rotated grid as above:"
Please write code following the instructions in jupyter notebook style.

targets: 
grid_new = (2 * grid_l5m0_rot + grid_l5m2**2 - grid_l5m2 * grid_l5m2) / 2.0
grid_new.plot()

coeffs = grid_new.expand()
fig, ax = coeffs.plot_spectrum()
inputs: # 복원된 모델을 평가합니다
loss, acc = new_model.evaluate(test_images,  test_labels, verbose=2)
print('복원된 모델의 정확도: {:5.2f}%'.format(100*acc))

print(new_model.predict(test_images).shape)
Given the above code, generate some markdown instructions for it.

targets: 
복원된 모델은 원본 모델과 동일한 매개변수로 컴파일되어 있습니다. 이 모델을 평가하고 예측에 사용해 보죠:
inputs: I am working on the file "Loan_payment_feature_engineering.ipynb".
The first task is:
All these payments seem to have flat months indeed! As we assume that loans are payed monthly, the flat lines on the map appear to be missed payments. As order_info provides us with the heigth of every montly payment, we can plot the difference between the money ordered and eventually paid.
Also, all payments occur on the 12th of the month. Many customers who sign the contract of the loan before the 12th, therefore providing a missing payment in the data which may be artificial. We assume that loan payments start the next month therefore we bump the date by 12 days if the day in the month is before the 12th, removing our missed payments.
Can you write Python code for it?

targets: 
from datetime import datetime
payment_diff = transaction_scope.copy()
for acct in scope:
    loan_start = loan_info[loan_info['account_id']==acct]['date'].unique()[0]
    loan_date = datetime.strptime(loan_start, '%Y-%m-%d')
    if loan_date.day < 12:
        loan_date = datetime(loan_date.year, loan_date.month, loan_date.day + 12)
    loan_start = loan_date.strftime('%Y-%m-%d')
    loan_order = loan_info[loan_info['account_id']==acct]['payments'].unique()[0]
    payment_diff.loc[payment_diff.index > loan_start, acct] = payment_diff.loc[payment_diff.index > loan_start,acct] - loan_order
inputs: "(B) Counting characters. Is it faster to use the built-in function str.count() or to loop through a string and count characters manually? Compare the two by counting all the A's in all the sequences in fake.fasta using each method and comparing how long they take to run. 
(You do not need to output the counts)"
Please write code following the instructions in jupyter notebook style.

targets: 
# Method 1 (Manual counting)
seqDict = my_utils.read_fasta("fake.fasta")

start = time.time()

for seqID in seqDict:
    seq = seqDict[seqID]
    count = 0
    for char in seq:
        if char == "A":
            count += 1
            
end = time.time()
print end - start

# Method 2 (.count())
seqDict = my_utils.read_fasta("fake.fasta")

start = time.time()

for seqID in seqDict:
    seq = seqDict[seqID]
    count = seq.count("A")
            
end = time.time()
print end - start
inputs: Above, we created a new transform primitive that can be used with Deep Feature Synthesis by deriving a new primitive class using TransformPrimitive as a base and overriding get_function to return a function that calculates the feature. Additionally, we set the input data types that the primitive applies to and the return data type. Input and return data types are defined using a Woodwork ColumnSchema. A full guide on Woodwork logical types and semantic tags can be found in the Woodwork Understanding Types and Tags guide.
Similarly, we can make a new aggregation primitive using AggregationPrimitive.

targets: class Maximum(AggregationPrimitive):
    name = 'maximum'
    input_types = [ColumnSchema(semantic_tags={'numeric'})]
    return_type = ColumnSchema(semantic_tags={'numeric'})

    def get_function(self):
        def maximum(column):
            return max(column)

        return maximum
inputs: I am working on the file "NEU/Tejas_Bawaskar _DL/Deep Network Visualization.ipynb".
The first task is:
Visualizing the fifth convnet/decoded/output layer_5 with sample test image
Can you write Python code for it?

targets: 
i = 1
x = x_test[i:i+1]
check = np.squeeze(np.squeeze(np.array(convout5_f(x)),0),0)
check.shape

#Plotting conv_5

temp = x[0,:,:,:]
fig, axes = plt.subplots(1, 1, figsize=(3, 3))
plt.imshow(temp)
plt.show()


k = 0
while k < check.shape[2]:
    #plt.figure()
    #plt.subplot(231 + i)
    fig, axes = plt.subplots(2, 4, figsize=(5, 5))
    for i in range(2):
        for j in range(4):
            axes[i,j].imshow(check[:,:,k])
            k += 1
    #axes[0, 0].imshow(R, cmap='jet')
    #plt.imshow(check[:,:,i])

plt.show()
inputs: # look at unique company names in rounds df
# note that the column name in rounds file is different (company_permalink)
len(rounds.company_permalink.unique())

Given the above code, generate some markdown instructions for it.

targets: 
Thus, there are 66368 unique companies in the table and permalink is the unique primary key. Each row represents a unique company.
Let's now check whether all of these 66368 companies are present in the rounds file, and if some extra ones are present.
inputs: !cd 06-regsort-gpu/hybrid ; nvprof -s --profile-child-processes  ./run.sh
Given the above code, generate some markdown instructions for it.

targets: 
Local load and store transactions should have been reduced to zero.
The display of kernel time actually includes some memory transfer and synchronization. The real execution time is returned by the profiler when run in summary mode.
inputs: I am working on the file "DL0110EN/5.1.1dropoutPredictin.ipynb".
The first task is:
Create two model objects: model had no dropout and model_drop has a dropout probability of 0.5:
Can you write Python code for it?

targets: 
model=Net(2,300,2)
model_drop=Net(2,300,2,p=0.5)
inputs: I am working on the file "Data Wrangling with Python.ipynb".
The first task is:
Bonus Round: Questions we can ask of this data
Now, you can start asking questions of this data. What are some interesting questions that this data might answer for us, and how would do that?

Which congressional districts get the most funding?
Which agencies get the most funding in my state?
[With the help of additional data] Which congressional committees do congress folk sit on and is there a coorelation between the committees they sit on and the agencies or contractors that are getting contracts?
How removed is the location of the contractor vs the place where the work is being performed?
What interesting things might there be in the walshhealyact, servicecontractact, davisbaconact, and clingercohenact columns?
What is the distribution of size of vendor to number of contracts and also to the number of contracting dollars? Is any vendor an outlier?
What are the reasons for non-competed contracts?
What percentage of contracts are in each of these categories, and are the numbers reflective of the population as a whole:
issbacertifiedsmalldisadvantagedbusiness
womenownedflag
veteranownedflag
minorityownedbusinessflag
tribalgovernmentflag
ishispanicservicinginstitution
iswomenownedsmallbusiness
isecondisadvwomenownedsmallbusiness
isjointventurewomenownedsmallbusiness
isjointventureecondisadvwomenownedsmallbusiness



What other kinds of questions can we ask?
Can you write Python code for it?

targets: 
print table
inputs: "Aside: If you ever need true decimal storage with no loss of precision, turn to the Decimal package. Just be warned it will come at a price:"
Please write code following the instructions in jupyter notebook style.

targets: 
a_native = 1.0
b_native = 2.0

a_decimal = Decimal ('1.0')
b_decimal = Decimal ('2.0')

%timeit a_native + b_native
%timeit a_decimal + b_decimal
inputs: I am working on the file "Check_bug_correction.ipynb".
The first task is:
The same plot than before but using the new threshol
Can you write Python code for it?

targets: 
# Use new threshold for old data as well
cond_sel = ((merged['lr_5_old'] >= tnew) | (merged['lr_5_new'] >= tnew))
cond_nosel = ((merged['lr_5_old'] < tnew) & (merged['lr_5_new'] < tnew))
cond_change = ((merged['lr_5_old'] >= tnew) & (merged['lr_5_new'] < tnew))
plt.loglog(merged['lr_5_new'][cond_sel], (np.abs(diff_lr)/merged['lr_5_new']*100)[cond_sel], marker=",", ls="", label="Selected")
plt.loglog(merged['lr_5_new'][cond_nosel], (np.abs(diff_lr)/merged['lr_5_new']*100)[cond_nosel], marker=",", ls="", label="Rejected")
plt.loglog(merged['lr_5_new'][cond_change], (np.abs(diff_lr)/merged['lr_5_new']*100)[cond_change], marker="x", ls="", label="Changes")
plt.xlabel("log LR (new)")
plt.ylabel("log percent abs. LR diff")
plt.legend()