Muennighoff/xP3x-sample huggingface dataset:
inputs: # create the data in case you skipped the previous exercise

# a helper function to create the sine train set that can also add noise to the data
def noisy_sine_train_data(noise=None):
    x_train = np.linspace(0, 6, 10).reshape((-1, 1))
    y_train = np.sin(x_train)
    
    # If fixed, set the random seed so that the next call of the
    # random function always returns the same result
    if noise == 'fixed':
        np.random.seed(1)
    
    x_train += np.random.randn(len(x_train)).reshape((-1, 1)) / 5
    
    return x_train, y_train

def sine_test_data():
    x_test = 0.5 + np.arange(6).reshape((-1, 1))
    y_test = np.sin(x_test)
    return x_test, y_test

x_train, y_train = noisy_sine_train_data(noise='fixed')

# we randomly pick 3 data points to get a nice validation set
train_i = [0, 1, 3, 4, 6, 7, 9]
val_i = [2, 5, 8]

# create the train and validation sets
x_train_i = x_train[train_i, :]
y_train_i = y_train[train_i]
x_val_i = x_train[val_i, :]
y_val_i = y_train[val_i]

### BEGIN SOLUTION
pol_exp = sklearn.preprocessing.PolynomialFeatures(degree=4)
### END SOLUTION
# pol_exp = sklearn.preprocessing.PolynomialFeatures(degree= ? )

model = sklearn.linear_model.LinearRegression()
model.fit(pol_exp.fit_transform(x_train_i), y_train_i)

### BEGIN SOLUTION
train_score = model.score(pol_exp.fit_transform(x_train_i), y_train_i)
validation_score = model.score(pol_exp.fit_transform(x_val_i), y_val_i)
### END SOLUTION
# train_score = model.score( ? )
# validation_score = model.score( ? )

print(f'The R2 score of this model on the train set is: {train_score:.3f}')
print(f'The R2 score of this model on the validation set is: {validation_score:.3f}')
Given the above code, generate some markdown instructions for it.

targets: 
5. Over-fitting and Cross-Validation
What you have experienced above is called over-fitting and happens when your model learns the noise that is inherent in the data.
This problem was caused because there were to many parameters in the model. So the model was too advanced so that it became capable of learning the noise in the data by heart. Reducing the number of parameters solves this problem. But how do you know how many parameters is optimal?
(Another way to solve this problem is to use more data. Because if there are more data points in the data and if there is more noise, your model isn't able to learn all that noise anymore and you get a better result. Since it's often not possible to gather more data we will not take this approach.)
In the exercise above you had to set the number of polynomial functions to get a better result, but how can you estimate this in a reliable way without manually selection the optimal parameters?
5.1. Validation set
A common way to solve this problem is through the use of a validation set. This means that you use a subset of the training data to train your model on, and another subset of the training data to validate your parameters. Based on the score of your model on this validation set you can select the optimal parameter.
So use this approach to select the best number of polynomials for the noisy sine function.
inputs: # Create a scalar variable with the initial value 3.
v = tf.contrib.eager.Variable([3])

# Create a vector variable of shape [1, 4], with random initial values,
# sampled from a normal distribution with mean 1 and standard deviation 0.35.
w = tf.contrib.eager.Variable(tf.random_normal([1, 4], mean=1.0, stddev=0.35))

print("v:", v.numpy())
print("w:", w.numpy())
Given the above code, generate some markdown instructions for it.

targets: 
An alternative solution would be to reshape a into a 6x1 matrix and b into a 1x3 matrix, resulting in a 6x3 matrix after multiplication.
Variables, Initialization and Assignment
So far, all the operations we performed were on static values (tf.constant); calling numpy() always returned the same result. TensorFlow allows you to define Variable objects, whose values can be changed.
When creating a variable, you can set an initial value explicitly, or you can use an initializer (like a distribution):
inputs: # Problem 1 - Implement Min-Max scaling for grayscale image data
def normalize_grayscale(image_data):
    """
    Normalize the image data with Min-Max scaling to a range of [0.1, 0.9]
    :param image_data: The image data to be normalized
    :return: Normalized image data
    """
    # TODO: Implement Min-Max scaling for grayscale image data
    min = np.min(image_data)
    max = np.max(image_data)
    return .1 + ((image_data - min) * (.9 - .1)) / (max - min)

### DON'T MODIFY ANYTHING BELOW ###
# Test Cases
np.testing.assert_array_almost_equal(
    normalize_grayscale(np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 255])),
    [0.1, 0.103137254902, 0.106274509804, 0.109411764706, 0.112549019608, 0.11568627451, 0.118823529412, 0.121960784314,
     0.125098039216, 0.128235294118, 0.13137254902, 0.9],
    decimal=3)
np.testing.assert_array_almost_equal(
    normalize_grayscale(np.array([0, 1, 10, 20, 30, 40, 233, 244, 254,255])),
    [0.1, 0.103137254902, 0.13137254902, 0.162745098039, 0.194117647059, 0.225490196078, 0.830980392157, 0.865490196078,
     0.896862745098, 0.9])

if not is_features_normal:
    train_features = normalize_grayscale(train_features)
    test_features = normalize_grayscale(test_features)
    is_features_normal = True

print('Tests Passed!')

if not is_labels_encod:
    # Turn labels into numbers and apply One-Hot Encoding
    encoder = LabelBinarizer()
    encoder.fit(train_labels)
    train_labels = encoder.transform(train_labels)
    test_labels = encoder.transform(test_labels)

    # Change to float32, so it can be multiplied against the features in TensorFlow, which are float32
    train_labels = train_labels.astype(np.float32)
    test_labels = test_labels.astype(np.float32)
    is_labels_encod = True

print('Labels One-Hot Encoded')

assert is_features_normal, 'You skipped the step to normalize the features'
assert is_labels_encod, 'You skipped the step to One-Hot Encode the labels'

# Get randomized datasets for training and validation
train_features, valid_features, train_labels,...
Given the above code, generate some markdown instructions for it.

targets: 
<img src="image/Mean_Variance_Image.png" style="height: 75%;width: 75%; position: relative; right: 5%">
Problem 1
The first problem involves normalizing the features for your training and test data.
Implement Min-Max scaling in the normalize_grayscale() function to a range of a=0.1 and b=0.9. After scaling, the values of the pixels in the input data should range from 0.1 to 0.9.
Since the raw notMNIST image data is in grayscale, the current values range from a min of 0 to a max of 255.
Min-Max Scaling:
$
X'=a+{\frac {\left(X-X_{\min }\right)\left(b-a\right)}{X_{\max }-X_{\min }}}
$
If you're having trouble solving problem 1, you can view the solution here.
inputs: Aに対して行列積(np.dot)を行うと以下のように単位行列が作成されます。これにより正しく逆行列が計算されることがわかります。

targets: np.dot?

np.dot(A,AA)
inputs: Create a Spark DataFrame from Pandas:

targets: spark_df = context.createDataFrame(pandas_df)
inputs: I am working on the file "16--numpy.ipynb".
The first task is:
You can also create empty (uninitialized) arrays. What does the following produce?
Can you write Python code for it?

targets: 
A = np.empty ((3, 4)) # An empty 3 x 4 matrix
print A
inputs: "And here's the fraction of Group 1 that would be below the threshold, which I'll call the false negative rate."
Please write code following the instructions in jupyter notebook style.

targets: 
np.mean(sample2 < thresh)
inputs: "As no new Structure is build, the num_sub_systems must be updated by hand.
Otherwise this happens automatically."
Please write code following the instructions in jupyter notebook style.

targets: 
S.num_sub_systems = 2
inputs: "And now for the actual package..."
Please write code following the instructions in jupyter notebook style.

targets: 
import wwdata as ww
inputs: 1.2 Adding derived features
Similarly to the previous notebook, you can add derived features to your dataset, as demonstrated in the example below.

targets: # Add a column where the week is derived from the datetime column.
calls['week'] = calls.index.map(lambda observation_timestamp: 
                                observation_timestamp.week)

# Display the head of the new dataset.
calls.head(5)
inputs: "All the above results are very close to the b1 calculation results for the 57 cm reactor. So my question: how is the b1 diffusion coefficient calculated and how does it differ from calculation of DIFFCOEFF?
Serpent 1 and 2 results for the same input file are pretty much identical, which I suppose should be expected. However, the Serpent 2 result file does contain a lot more information.
3/9/17
Analog estimator: score the physical interactions (fission, capture, etc.) or event sequences that are simulated during the calculation. Analog estimates are directly related to the simulation process.
Note: in general when the chain of equalities ends, that generally means I've reached a point where all arguments are generated via Monte Carlo analog or implicit estimates, e.g. none of the arguments are derived quantities, e.g. if I search in collectresults.c, their first hit will be on the RHS of an equality sign. 
Default calculation of $D$:
\begin{equation}
D_g = \frac{\bar{r_g^2}}{6}\ \Sigma_{r,g}
\end{equation}
where $\bar{r_g^2}$ is the mean value of all scored square distances that neutrons have made within energy group $g$, which is an analog estimator. The starting points are the locations where neutrons have entered the group by fission or scattering from another (higher) energy group. The end points are the locations at which the neutrons are lost from the group either by absorption or scattering to another (lower) energy group.
Leakage calculation of $D$:
\begin{equation}
D_g = \frac{leak}{B_m^2}\
B_m^2 = \frac{k_{\infty} - 1}{L_g^2}\
L_g^2 = \frac{\bar{r_g^2}}{6}\
k_{\infty} = \frac{src}{loss} = \frac{\Sigma_{g,f}\bar{\nu}}{\Sigma_{g,f} + \Sigma_{g,c} - \Sigma_{n\rightarrow n}}
\end{equation}
where $leak$ comes directly from Monte Carlo scoring of neutrons exiting the simulation domain I believe.
P1 calculation of $D$:
\begin{equation}
D_g = \frac{1}{3\Sigma_{g,tr}}\
\Sigma_{g,tr} = \Sigma_{g,t} - \Sigma_{g,s1}
\end{equation}
B1 calculation of $D$ (note that this occurs in b1solver.c which is different from..."
Please write code following the instructions in jupyter notebook style.

targets: 
40 * 20000

(381.99 + 160.60) / 4

381.99 + 160.60
inputs: "Collect some data"
Please write code following the instructions in jupyter notebook style.

targets: 
# we'll use data from a job that collected tweets about parenting
tweet_bodies = [body for body in open('tweet_bodies.txt')]

# sanity checks
print(len(tweet_bodies))

# sanity checks
print(tweet_bodies[:10])

# lets do some quick deduplication
from duplicate_filter import duplicateFilter

## set the similarity threshold at 90%
dup_filter = duplicateFilter(0.9)

deduped_tweet_bodies = []
for id,tweet_body in enumerate(tweet_bodies):
    if not dup_filter.isDup(id,tweet_body):
        deduped_tweet_bodies.append(tweet_body)

pprint(deduped_tweet_bodies[:10])
inputs: $\implies$ Yay! It seems faster indeed!


Numba implementation
As this blog post explains it, we can also try to use Numba in a notebook cell.

targets: from numba import jit

@jit("int32(boolean[:])")
def lempel_ziv_complexity_numba_x(binary_sequence):
    """Lempel-Ziv complexity for a binary sequence, in Python code using numba.jit() for automatic speedup (hopefully)."""
    u, v, w = 0, 1, 1
    v_max = 1
    length = len(binary_sequence)
    complexity = 1
    while True:
        if binary_sequence[u + v - 1] == binary_sequence[w + v - 1]:
            v += 1
            if w + v >= length:
                complexity += 1
                break
        else:
            if v > v_max:
                v_max = v
            u += 1
            if u == w:
                complexity += 1
                w += v_max
                if w > length:
                    break
                else:
                    u = 0
                    v = 1
                    v_max = 1
            else:
                v = 1
    return complexity

def str_to_numpy(s):
    """str to np.array of bool"""
    return np.array([int(i) for i in s], dtype=np.bool)

def lempel_ziv_complexity_numba(s):
    return lempel_ziv_complexity_numba_x(str_to_numpy(s))
inputs: I am working on the file ".ipynb_checkpoints/skill_cluster4-checkpoint.ipynb".
The first task is:
Consider only n-grams separately:
Can you write Python code for it?

targets: 
unigram_occur = countOccur_ngram(n=1)
bigram_occur = countOccur_ngram(n=2)
trigram_occur = countOccur_ngram(n=3)

occur_df = pd.merge(unigram_occur, bigram_occur, on='job_id')
occur_df = pd.merge(occur_df, trigram_occur, on='job_id')

occur_df['n_skill'] = occur_df['n_1gram'] + occur_df['n_2gram'] + occur_df['n_3gram']
occur_df.head()

from my_util import *
n_skill_by_jd = pd.concat([quantile(occur_df['n_1gram']), quantile(occur_df['n_2gram']), quantile(occur_df['n_3gram'])])
n_skill_by_jd.index = ['n_unigram', 'n_bigram', 'n_trigram']
n_skill_by_jd

n_skill_by_jd.to_csv(REPORT_DIR + 'skill_by_jd.csv')

def plot2(w=7, h=10):    
    fig = plt.figure(figsize=(w,h))
    for i in range(3):
        n = i+1
        plt.subplot(3, 1, n)
        plt.subplots_adjust(hspace=.25)

        ngram_occur = occur_df['n_{}gram'.format(n)]
        m, bins, patches = plt.hist(x=ngram_occur, bins=np.unique(ngram_occur), align='left')
        plt.xlim(0, max(bins)+1)

        if (n==3):
            plt.xticks(range(11))
    #     plt.title('{}-gram'.format(n))
        plt.xlabel('# {}-gram skills'.format(n), fontsize='large')
        plt.ylabel('# JDs', fontsize='large')

    plt.show()
    return fig

fig2 = plot2()
fig2.savefig(REPORT_DIR + 'n_skill_in_jd.pdf')
fig2.savefig(REPORT_DIR + 'n_skill_in_jd.jpeg')
plt.close(fig2)
inputs: I am working on the file "Python/61_Registration_Introduction_Continued.ipynb".
The first task is:
Version 2
<ul>
<li> Multi scale - specify both scale, and how much to smooth with respect to original image.</li>
<li> Initial transformation modified in place, so in the end we have the same type of transformation in hand.</li>
</ul>
Can you write Python code for it?

targets: 
registration_method = sitk.ImageRegistrationMethod()

registration_method.SetMetricAsMattesMutualInformation(numberOfHistogramBins=50)
registration_method.SetMetricSamplingStrategy(registration_method.RANDOM)
registration_method.SetMetricSamplingPercentage(0.01)

registration_method.SetInterpolator(sitk.sitkLinear)

registration_method.SetOptimizerAsGradientDescent(
    learningRate=1.0, numberOfIterations=100
)  # , estimateLearningRate=registration_method.EachIteration)
registration_method.SetOptimizerScalesFromPhysicalShift()

final_transform = sitk.Euler3DTransform(initial_transform)
registration_method.SetInitialTransform(final_transform)
registration_method.SetShrinkFactorsPerLevel(shrinkFactors=[4, 2, 1])
registration_method.SetSmoothingSigmasPerLevel(smoothingSigmas=[2, 1, 0])
registration_method.SmoothingSigmasAreSpecifiedInPhysicalUnitsOn()

registration_method.AddCommand(
    sitk.sitkStartEvent, registration_callbacks.metric_start_plot
)
registration_method.AddCommand(
    sitk.sitkEndEvent, registration_callbacks.metric_end_plot
)
registration_method.AddCommand(
    sitk.sitkMultiResolutionIterationEvent,
    registration_callbacks.metric_update_multires_iterations,
)
registration_method.AddCommand(
    sitk.sitkIterationEvent,
    lambda: registration_callbacks.metric_plot_values(registration_method),
)

registration_method.Execute(
    sitk.Cast(fixed_image, sitk.sitkFloat32), sitk.Cast(moving_image, sitk.sitkFloat32)
)

print(
    f"Optimizer's stopping condition, {registration_method.GetOptimizerStopConditionDescription()}"
)
print(f"Final metric value: {registration_method.GetMetricValue()}")

# Save moving image after registration and view overlap using external viewer.
save_transform_and_image(
    final_transform,
    fixed_image,
    moving_image,
    os.path.join(OUTPUT_DIR, "finalAlignment-v2"),
)
inputs: "Avg adopters per seed comparison"
Please write code following the instructions in jupyter notebook style.

targets: 
draw_bars_comparison('Avg adopters per seeds', 'Avg adopters', np.array(get_avg_per_seed(eigen)+[(0, np.mean(eigen[:,1]))]))
inputs: "13. Defina y llame una función que reciva de parametro una lista de listas y solucione el problema 8"
Please write code following the instructions in jupyter notebook style.

targets: 
def max_list_of_lists(list_of_lists):
    size = len(list_of_lists)
    carpeta = list_of_lists[1]
    for i in range(size):
        if sum(list_of_lists[i]) > sum(carpeta):
            carpeta = list_of_lists[i]
    print(carpeta)

# Implementar la función
list_of_lists = [ [1,2,3], [4,5,6], [10,11,12], [7,8,9] ]
max_list_of_lists (list_of_lists)
inputs: Bonus: reading a CSV file from Google Docs
It used to be easy to anonymously read a public file directly from Google Docs, but now you need an API key. It's not too hard to set up, but you'll need to read some docs.
When you have an API key, put it here...

targets: key = "PUT YOUR KEY HERE"

import json

url = "https://sheets.googleapis.com/v4/spreadsheets/{id}/values/{sheet}"

meta = {"id": "1YlnEGT8uHpRllk7rjAgFFl8V6B5-kl02DBie11PjG9Q",
        "sheet": "Sheet1"
        }
url = url.format(**meta)

params = {"key": key}

r = requests.get(url, params=params)
j = json.loads(r.text)['values']
df = pd.DataFrame(j[1:], columns=j[0])
df.head()
inputs: I am working on the file ".ipynb_checkpoints/RNN_project-checkpoint.ipynb".
The first task is:
<a id='TODO_6'></a>
With your trained model try a few subsets of the complete text as input - note the length of each must be exactly equal to the window size.  For each subset us the function above to predict the next 100 characters that follow each input.
Can you write Python code for it?

targets: 
# TODO: choose an input sequence and use the prediction function in the previous Python cell to predict 100 characters following it
# get an appropriately sized chunk of characters from the text
start_inds = []

# load in weights
model.load_weights('model_weights/best_RNN_small_textdata_weights.hdf5')
for s in start_inds:
    start_index = s
    input_chars = text[start_index: start_index + window_size]

    # use the prediction function
    predict_input = predict_next_chars(model,input_chars,num_to_predict = 100)

    # print out input characters
    print('------------------')
    input_line = 'input chars = ' + '\n' +  input_chars + '"' + '\n'
    print(input_line)

    # print out predicted characters
    line = 'predicted chars = ' + '\n' +  predict_input + '"' + '\n'
    print(line)
inputs: .loc may suffer from lower performance as compared to .iloc due to the possibility of needing to map the label values into locations.

targets: # subset of first three rows
subset = sp500[:3].copy()
# get the location of the Price column
price_loc = sp500.columns.get_loc('Price')
# get the location of the MMM row
abt_row_loc = sp500.index.get_loc('ABT')
# change the price
subset.iloc[abt_row_loc,price_loc] = 1000
subset
inputs: 4. Get template QC atlas from database
Available templates in Database:
MSMLS_HILICz150mm_Annotation20190824_Template_EMA_Unlabeled_Positive\
MSMLS_HILICz150mm_Annotation20190824_Template_EMA_Unlabeled_Negative\
MSMLS_HILICz150mm_Annotation20190824_Template_QCv3_Unlabeled_Positive\
MSMLS_HILICz150mm_Annotation20190824_Template_QCv3_Unlabeled_Negative\
MSMLS_HILICz150mm_Annotation20190824_Template_ISv5_Unlabeled_Positive\
MSMLS_HILICz150mm_Annotation20190824_Template_ISv5_Unlabeled_Negative\
MSMLS_HILICz150mm_Annotation20190824_Template_ISv5_13C15N_Positive\
MSMLS_HILICz150mm_Annotation20190824_Template_ISv5_13C15N_Negative

targets: #Atlas File Name 
LCS = 'MSMLS' # Library Compound Set
CTY = 'HILICz150mm' # Chromatography
LR = 'Annotation20190824' # Library Run
RTS = 'Template' # RT space
CPD = 'QCv3' # Set of Compounds
LAB = 'Unlabeled' # Isolabeling
POL = 'Positive' # Polarity

QC_template_filename = "_".join((LCS,CTY,LR,RTS,CPD,LAB,POL))

atlases = metob.retrieve('Atlas',name=QC_template_filename,
                         username='vrsingan')
names = []
for i,a in enumerate(atlases):
    print(i,a.name,pd.to_datetime(a.last_modified,unit='s'),len(a.compound_identifications))

# #Alternatively use this block to create QC atlas from spreadsheet
# import datetime
#dp = reload(dp)

# LCS = 'MSMLS' # Library Compound Set
# CTY = 'HILICz150mm' # Chromatography
# LR = 'Annotation20190824' # Library Run
# RTS = 'Template' # RT space
# CPD = 'QCv3' # Set of Compounds
# LAB = 'Unlabeled' # Isolabeling
# POL = 'Positive' # Polarity
# DT = datetime.datetime.strftime(datetime.datetime.now(),'%Y%m%d')
# QC_template_filename = "_".join((LCS,CTY,LR,RTS,CPD,LAB,POL,DT))

#myAtlas = dp.make_atlas_from_spreadsheet('/global/project/projectdirs/metatlas/projects/1_TemplateAtlases/TemplateAtlas_HILICz150mm_Annotation20190824_QCv3_Unlabeled_Positive.csv',
#                                       QC_template_filename,
#                                        filetype='csv',
#                                        sheetname='',
#                                        polarity = 'positive',
#                                        store=True,
#                                       mz_tolerance = 20)
#atlases = dp.get_metatlas_atlas(name=QC_template_filename,do_print = True,most_recent=True)

myAtlas = atlases[-1]
atlas_df = ma_data.make_atlas_df(myAtlas)
atlas_df['label'] = [cid.name for cid in myAtlas.compound_identifications]
print myAtlas.name
print myAtlas.username
inputs: But that shouldn't actually blow your mind. Based on our discussion before, you can probably guess that the decorator notation is just shorthand for: 
say_bye = verbose( say_bye )
One place where this shorthand can come in particularly handy is when you need to stack a bunch of decorators. In place of nested decorators like this:
python
my_func = do_thing_a( add_numbers( subtract( verify( my_func ))))
We can write this as:
python
@do_thing_a
@add_numbers
@subtract
@verify
def my_func():
    # something useful happens here
Note that the order matters!

Ok, thank you, please come again.
THE END AGAIN

Ok, final round, I promise. 
Appendix
This is material that I originally intended to include in this RST (because it's relevant), but ultimately cut for clarity. You can come back and revisit it any time.
Scopes and namespaces
Roughly speaking, scope and namespace are the reason you can type some code (like variable_1 = 'dog') and then transparently use variable_1 later in your code. Sounds obvious, but easy to take for granted!
The concepts of scope and namespace in Python are pretty interesting and complex. Some time when you're bored and want to learn some details, have a read of this nice explainer or the official docs on the Python Execution Model. 
A short way to think about them is the following:

A namespace is a mapping from (variable) names to values; think about it like a Python dictionary, where our code can look up the keys (the variable names) and then use the corresponding values. You will generally have many namespaces in your code, they are usually nested (sort of like an onion), and they can even have identical keys (variable names). 
The scope (at a particular location in the code) defines in which namespace we look for variables (dictionary keys) when our code is executing.

While this RST isn't explicitly about scope, understanding these concepts will make it easier to read the code later on. Let's look at some examples.
There are two built-in functions that can aid in exploring the namespace...

targets: # -c option starts a new interpreter session in a subshell and evaluates the code in quotes. 
#  here, we just assign the value 3 to the variable x and print the global namespace
! python -c 'x=3; print( globals() )'
inputs: "Abstract syntax trees
This is made possible because siuba represents lazy expressions with abstract syntax trees.
Fast grouped operations and SQL queries are just the beginning. This allows people to produce a whole range of interesting tools!
Siuba's lazy expressions consist of a Symbolic and Call class.
Symbolic is used to quickly create lazy expressions."
Please write code following the instructions in jupyter notebook style.

targets: 
# ASTs for transforming
from siuba.siu import Symbolic, Call, strip_symbolic

_ = Symbolic()

sym = _.a.mean() + _["b"]
sym
inputs: "Crear un arreglo de 10 unos"
Please write code following the instructions in jupyter notebook style.

targets: 
np.ones(10)
inputs: D
In this question, you'll write a function acount_lines that performs the same operation as before, except in the case that the output file already exists: in this case, you'll append the line count to the file instead of overwriting it, thus preserving any existing previous line counts.
Each new appended line count should be on its own line in the output file. You may need to manually insert newline characters, which are a backslash followed by the letter n: \n
Your function should return True on success, and False if an error occurred.
NOTE: Your function should be able to handle errors gracefully! If an error occurs when trying to read from the file or write to the output file, your function should return False.

targets: if os.path.exists("q1data/out_again.txt"):
    os.remove("q1data/out_again.txt")

assert acount_lines("q1data/file1.txt", "q1data/out_again.txt")
assert os.path.exists("q1data/out_again.txt")
assert int(open("q1data/out_again.txt", "r").read()) == 3

assert acount_lines("q1data/file2.txt", "q1data/out_again.txt")
assert os.path.exists("q1data/out_again.txt")
assert int("".join(open("q1data/out_again.txt", "r").read().split("\n"))) == 34

r1 = None
try:
    r1 = acount_lines("yet/another/nonexistent/path.txt", "meaningless")
except:
    assert False
else:
    assert not r1

r2 = None
try:
    r2 = acount_lines("q1data/file2.txt", "/this/should/throw/an/error.txt")
except:
    assert False
else:
    assert not r2
inputs: !ls -lh data

import numpy as np
from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Dropout, Flatten, Dense
from keras import applications

# dimensions of our images.
img_width, img_height = 150, 150

train_data_dir = 'data/train'
validation_data_dir = 'data/validation'
nb_train_samples = 2000
nb_validation_samples = 800
epochs = 50
batch_size = 16

# build the VGG16 network
model = applications.VGG16(include_top=False, weights='imagenet')

model.summary()
Given the above code, generate some markdown instructions for it.

targets: 
This script goes along the blog post
"Building powerful image classification models using very little data"
from blog.keras.io.
It uses data that can be downloaded at:
https://www.kaggle.com/c/dogs-vs-cats/data
In our setup, we:
- created a data/ folder
- created train/ and validation/ subfolders inside data/
- created cats/ and dogs/ subfolders inside train/ and validation/
- put the cat pictures index 0-999 in data/train/cats
- put the cat pictures index 1000-1400 in data/validation/cats
- put the dogs pictures index 12500-13499 in data/train/dogs
- put the dog pictures index 13500-13900 in data/validation/dogs
So that we have 1000 training examples for each class, and 400 validation examples for each class.
In summary, this is our directory structure:
data/
    train/
        dogs/
            dog001.jpg
            dog002.jpg
            ...
        cats/
            cat001.jpg
            cat002.jpg
            ...
    validation/
        dogs/
            dog001.jpg
            dog002.jpg
            ...
        cats/
            cat001.jpg
            cat002.jpg
            ...
inputs: Construct the reward array R and the transition probability array Q:

targets: # Number of actions; 0: stay put, 1: new job, 2: new life
m = 3

# Reward and transition probability arrays
R = np.empty((n, m))
Q = np.zeros((n, m, n))

# Stay put
R[:, 0] = theta[s_indices2d[:, 0]] + epsilon[s_indices2d[:, 1]]
Q[np.arange(n), 0, np.arange(n)] = 1

# New job
R[:, 1] = theta[s_indices2d[:, 0]] + G_mean
for i in range(N):
    Q[i*N:(i+1)*N, 1, i*N:(i+1)*N] = G_probs

# New life
R[:, 2] = F_mean + G_mean
Q[:, 2] = F_probs.reshape(N, 1).dot(G_probs.reshape(1, N)).ravel()
inputs: "<font color='red'>Вопрос 6 (2 балла).</font> Постройте новый признак – BMI (Body Mass Index). Для этого надо вес в килограммах поделить на квадрат роста в метрах. Нормальными считаются значения BMI от 18.5 до 25. Выберите верные утверждения.
<font color='red'>Утверждения:</font>
- Медианный BMI по выборке превышает норму
- У женщин в среднем BMI ниже, чем у мужчин
- У здоровых в среднем BMI выше, чем у больных
- В сегменте здоровых и непьющих мужчин в среднем BMI ближе к норме, чем в сегменте здоровых и непьющих женщин"
Please write code following the instructions in jupyter notebook style.

targets: 
df['BMI'] = df.weight / ((df.height / 100) ** 2)
print("median: " + str(df.BMI.median()))
print("women mean: " + str(df[df.gender == 1].BMI.mean()))
print("men mean: " + str(df[df.gender == 2].BMI.mean()))
print("healty mean: " + str(df[df.cardio == 0].BMI.mean()))
print("sick mean: " + str(df[df.cardio == 1].BMI.mean()))
print("teetotal healthy men mean: " + str(df[(df.gender == 2) & (df.alco == 0) & (df.cardio == 0)].BMI.mean()))
print("teetotal healthy women mean: " + str(df[(df.gender == 1) & (df.alco == 0) & (df.cardio == 0)].BMI.mean()))
inputs: I am working on the file "Intro ML Semcomp/semcomp17_ml/.ipynb_checkpoints/semcomp17_ml_answer-checkpoint.ipynb".
The first task is:
Now we will perform gradient descent update procedure according to the following formula:
$$W = W - k\frac{\partial L}{\partial W}$$
Here we use fixed number of iterations and learning rate.
Can you write Python code for it?

targets: 
"""step 4: gradient descent - update loop"""
np.random.seed(0)
W = np.random.rand(1,X_train.shape[0]).astype(np.float32)   #W has size 1xD, randomly initialised
k = 1e-8                                #learning rate
niters = 160                            #number of training iterations

#visualisation settings
vis_interval = niters/50
loss_collections = []
plt.close()
plt.ion()
fig = plt.figure(1,figsize=(16, 4))
axis_loss = fig.add_subplot(131)
axis_data = fig.add_subplot(132)
for i in xrange(niters):
    Y_train = F_Regression(X_train,W)               #compute hypothesis function aka. predicted values
    loss = Loss_Regression(Y_train, Z_train)         #compute loss
    dLdW = dLdW_Regression(X_train, Y_train, Z_train) #compute gradient
    W = W - k*dLdW                            #update
    loss_collections.append(loss)
    if (i+1)% vis_interval == 0:
        ml_utils.plot_loss(axis_loss, range(i+1),loss_collections, "loss = " + str(loss))
        ml_utils.plot_scatter_and_line(axis_data, X_train, Z_train, W, "iter #" + str(i))
        fig.canvas.draw()
print "Learned parameters ", W.squeeze()
inputs: Beam search decoding

targets: # Set up beam search.
beam_decoder = Search(
    trax.models.Reformer, model_weights,
    beam_size=4,
    alpha=0.6,  # For length normalization, set to 0.6 following Vaswani et al.
    eos_id=1,  # The stop token has id 1 in the vocabulary we use.
    max_decode_len=146,
    )

pred_ids = []
preds = []
BATCH_SIZE = 1024
for start in range(0, input_ids.shape[0], BATCH_SIZE):
  print(start, '/', input_ids.shape[0], flush=True)
  batch = input_ids[start:start+BATCH_SIZE]
  seqs, scores = beam_decoder.decode(batch, batch_size=BATCH_SIZE)
  # Select highest scoring output.
  batch_pred_ids = seqs[:, -1]
  pred_ids.append(batch_pred_ids)
  preds.extend([
      tokenizer.decode(pred.tolist(), strip_extraneous=True)
      for pred in batch_pred_ids
  ])

bleu = sacrebleu.corpus_bleu(preds, [refs], lowercase=True, tokenize='intl')
print(bleu)
inputs: "Building the office structure
We store the office points in a namedtuple called Point with the properties x and y to hold the co-ordinates."
Please write code following the instructions in jupyter notebook style.

targets: 
from collections import namedtuple
Point = namedtuple('Point', ('x', 'y'))
inputs: # FINAL LOSS: 0.226 - Seq length 20, LR 0.001, Epochs 200
# FINAL LOSS: 0.141 - Seq length 20, LR 0.001, Epochs 250
# FINAL LOSS: 0.012 - Seq length 200, LR 0.01, Epochs 500 - Text generated equals to the data

# Number of Epochs
num_epochs = 250
# Batch Size
batch_size = 64
# RNN Size
rnn_size = 256
# Embedding Dimension Size
embed_dim = 300
# Sequence Length
seq_length = 20
# Learning Rate
learning_rate = 0.001
# Show stats for every n number of batches
show_every_n_batches = 50

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
save_dir = './save'
Given the above code, generate some markdown instructions for it.

targets: 
Neural Network Training
Hyperparameters
Tune the following parameters:

Set num_epochs to the number of epochs.
Set batch_size to the batch size.
Set rnn_size to the size of the RNNs.
Set embed_dim to the size of the embedding.
Set seq_length to the length of sequence.
Set learning_rate to the learning rate.
Set show_every_n_batches to the number of batches the neural network should print progress.
inputs: # Get the unique names of the original Macaulay2016 clusters
cluster_names = macaulay2016_metadata.cluster.unique()
# Sort them in alphabetical order so that they're in the order we want
cluster_names.sort()

# Map the cluster name to an integer number
cluster_name_to_integer = dict(zip(cluster_names, range(len(cluster_names))))

paper_cluster_integers = macaulay2016_metadata.cluster.map(cluster_name_to_integer)
paper_cluster_integers.head()

macaulay2016_palette = [macaulay2016_cluster_to_color_from_paper[x] for x in cluster_names]

from sklearn.metrics import confusion_matrix

confusion = pd.DataFrame(confusion_matrix(paper_cluster_integers, labels), 
                         index=cluster_names)
confusion.index.name = 'Macaulay2016 Labels'
confusion.columns.name = 'K-Means Predicted'

confusiongrid = sns.clustermap(confusion, annot=True, fmt='d', figsize=(4, 4),
               col_cluster=False, row_cluster=False, 
               row_colors=macaulay2016_palette, col_colors=kmeans_palette)

# rotate the ylabels to be horizontal instead of vertical
plt.setp(confusiongrid.ax_heatmap.get_yticklabels(), rotation=0);
Given the above code, generate some markdown instructions for it.

targets: 
Exercise 1
Change the number of clusters to 20 and use the "husl" palette for coloring
Evaluating clustering
How do we evaluate the clusters that we found versus the clusters from the paper?
inputs: "3.4 Generating a Training Dataset
The following code generates a training dataset (separated into positive and negative classes according to a random line in $\mathcal{X}$) and plots it."
Please write code following the instructions in jupyter notebook style.

targets: 
def plot_data(fig,plot_id,X,y=None,w_arr=None,my_x=None,title=None):
    ax = fig.add_subplot(plot_id)
    if y is None:
        ax.plot(X[:,1],X[:,2],'gx')
    else:
        ax.plot(X[y > 0,1],X[y > 0,2],'b+',label='Positive (+)')
        ax.plot(X[y < 0,1],X[y < 0,2],'ro',label='Negative (-)')
    ax.set_xlim(-1,1)
    ax.set_ylim(-1,1)
    ax.grid(True)
    if w_arr is not None:
        if isinstance(w_arr,list) is not True:
            w_arr=[w_arr]
        for i,w in enumerate(w_arr):
            if i==0:
                draw_line(ax,w,'g-',label='Theoretical')
            else:
                draw_line(ax,w,'g--')
    if my_x is not None:
        ax.plot([my_x[0]],[my_x[1]],'kx',markersize=10)
    if title is not None:
        ax.set_title(title)
    ax.legend(loc='best',frameon=True)

def create_dataset(N,make_plot=True,seed=None):
    X = generate_data(N,seed=seed)
    w_theoretical = get_random_line()
    y = get_hypothesis(X,w_theoretical)
    if make_plot is True:
        fig = plt.figure(figsize=(7,5))
        plot_data(fig,111,X,y,w_theoretical,title="Initial Dataset")
    return X,y,w_theoretical
inputs: "Actually, the deviation is 1/100 of this, so let's adjust..."
Please write code following the instructions in jupyter notebook style.

targets: 
x=data['Year']
y=data['J-D']/100.0
plot(x,y,'-o')
xlabel('Year')
ylabel('Temperature Deviation')
inputs: I am working on the file "Data Science From Scratch/7 - Hypothesis And Inference.ipynb".
The first task is:
Assuming a normal distribution, we conclude that we are 95% confident that the interval below includes the true $p$:
Can you write Python code for it?

targets: 
normal_two_sided_bounds(0.95, mu, sigma)
inputs: %matplotlib inline
Given the above code, generate some markdown instructions for it.

targets: 
Instruct the notebook to draw plot graphics inline:
inputs: I am working on the file "MBA_S17/Oliveira_CrimeRentNYC.ipynb".
The first task is:
Merge Crime and Precinct Data
Can you write Python code for it?

targets: 
# Merge the two Dataframes to include Zipcode

Combo = pd.merge(Crime, Precinct,          # left and right df's
                 how='inner',              # Inner Merge 
                 left_on='Precinct',       # link with this variable/column 
                 right_on='Precinct',      # link with this variable/column
                indicator = True)

Combo.dtypes

Combo.head()

# Update Crime df with data from Combo and remove indicator

Crime = Combo [['Crime', 'YTD_Current', 'YTD_Prior', 'Precinct', 'Zip', 'borough','Neighborhood']]

Crime.head()
inputs: # PROPERTY ID - DO NOT EDIT !  
DOC.set_id('cmip6.toplevel.radiative_forcings.greenhouse_gases.N2O.provision')  

# PROPERTY VALUE(S): 
# Set as follows: DOC.set_value("value")  
# Valid Choices: 
#      "N/A"  
#      "M"  
#      "Y"  
#      "E"  
#      "ES"  
#      "C"  
#      "Other: [Please specify]"  
# TODO - please enter value(s)

Given the above code, generate some markdown instructions for it.

targets: 
14. Radiative Forcings --&gt; Greenhouse Gases --&gt; N2O
Nitrous oxide forcing
14.1. Provision
Is Required: TRUE&nbsp;&nbsp;&nbsp;&nbsp;Type: ENUM&nbsp;&nbsp;&nbsp;&nbsp;Cardinality: 1.N
How this forcing agent is provided (e.g. via concentrations, emission precursors, prognostically derived, etc.)
inputs: Build the Neural Network
Apply the functions you implemented above to:
- Apply embedding to input_data using your get_embed(input_data, vocab_size, embed_dim) function.
- Build RNN using cell and your build_rnn(cell, inputs) function.
- Apply a fully connected layer with a linear activation and vocab_size as the number of outputs.
Return the logits and final state in the following tuple (Logits, FinalState)

targets: def build_nn(cell, rnn_size, input_data, vocab_size, embed_dim=300):
    """
    Build part of the neural network
    :param cell: RNN cell
    :param rnn_size: Size of rnns
    :param input_data: Input data
    :param vocab_size: Vocabulary size
    :param embed_dim: Number of embedding dimensions
    :return: Tuple (Logits, FinalState)
    """
    embed =  get_embed(input_data, vocab_size, embed_dim = 300)
    # TODO: Imprplement Function
    output, final_state = build_rnn(cell, embed)
    predictions = tf.contrib.layers.fully_connected( output , vocab_size, activation_fn=None)
    return predictions , final_state


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_build_nn(build_nn)
inputs: Build the Neural Network
Apply the functions you implemented above to:
- Apply embedding to input_data using your get_embed(input_data, vocab_size, embed_dim) function.
- Build RNN using cell and your build_rnn(cell, inputs) function.
- Apply a fully connected layer with a linear activation and vocab_size as the number of outputs.
Return the logits and final state in the following tuple (Logits, FinalState)

targets: def build_nn(cell, rnn_size, input_data, vocab_size, embed_dim):
    """
    Build part of the neural network
    :param cell: RNN cell
    :param rnn_size: Size of rnns
    :param input_data: Input data
    :param vocab_size: Vocabulary size
    :param embed_dim: Number of embedding dimensions
    :return: Tuple (Logits, FinalState)
    """
    embeddings = get_embed(input_data, vocab_size, embed_dim)
    output, final_state = build_rnn(cell, embeddings)
    logits = tf.contrib.layers.fully_connected(output, vocab_size, activation_fn = None,
                                              weights_initializer=tf.truncated_normal_initializer(stddev=1/np.sqrt(2)),
                                              biases_initializer=tf.zeros_initializer())
    return logits, final_state


"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_build_nn(build_nn)
inputs: I am working on the file "Copia_de_Copia_de_Preprocesamiento_y_Red_test.ipynb".
The first task is:
Cargar base de datos
Can you write Python code for it?

targets: 
# Cargando base de datos Alerce.
data = pd.read_pickle('/content/drive/My Drive/Supernova/ALERCE_stamps.pkl') # Base de datos original
#data = pd.read_pickle('/content/drive/My Drive/Supernova/ALERCE_stamps_1k.pkl') # Base de datos reducida

imgs = data['images'] # Lista de 31727 imagenes
labels = data['labels'] # Lista de 31727 labels

img_0 = imgs[0] # Primera imagen de imgs, ndarray 63x63, 3 canales
label_0 = labels[0] # Primer label, entero

np.shape(imgs)
inputs: Add attributes for supplementary edges

targets: for e in g.edges(data=True):
    way_id = e[2].get('id').split('-')[0]
    if way_id in edge_ids_to_add:
        g_t.add_edge(e[0], e[1], **e[2])
        g_t.add_node(e[0], lat=g.node[e[0]]['lat'], lon=g.node[e[0]]['lon'])
        g_t.add_node(e[1], lat=g.node[e[1]]['lat'], lon=g.node[e[1]]['lon'])
    if way_id in edge_ids_to_remove:
        if g_t.has_edge(e[0], e[1]):
            g_t.remove_edge(e[0], e[1])
            
for n in nx.isolates(g_t.copy()):
    g_t.remove_node(n)
inputs: I am working on the file "14/Homework-14-Ronga.ipynb".
The first task is:
Okay, it's far too big to even look at. Let's try to get a list of features from a new CountVectorizer that only takes the top 100 words.
Can you write Python code for it?

targets: 
count_vectorizer = CountVectorizer(stop_words='english', max_features=100)
X = count_vectorizer.fit_transform(speeches_df['content'])
X

pd.DataFrame(X.toarray(), columns=count_vectorizer.get_feature_names()).head()
inputs: I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Position_Concentration_Risk/notebook.ipynb".
The first task is:
Final Point
Be invested in as many uncorrelated assets as possible. In finance this is known as diversification. If you have a pricing model, price everything and invest accordingly. This concept is explained in the Long-Short Equity Lecture.
Capital Constraints
Because of transaction costs, you need to have certain minimum amounts of capital to invest in large numbers of assets. Therefore sometimes you are unable to invest in hundreds or thousands. In this case you should still try to maximize your portfolio size, keeping in mind that if you have a portfolio of size 20, you can still find 20 relatively uncorrelated assets and that's better than nothing.
Now Let's Explain with Math Rather Than Pictures
One of the key aspects of modern portfolio theory is that by combining multiple assets into a portfolio, you can reduce the entire package's overall risk. Since we represent the volatility of an asset by its standard deviation, we can easily show this mathematically.
Say that we have two assets in a portfolio, $S_1$ and $S_2$, with weights $\omega_1$ and $\omega_2$ such that $\omega_1 + \omega_2 = 1$. Call the portfolio $P$ and say that $S_1$ and $S_2$ have mean and standard deviation $\mu_1, \sigma_1$ and $\mu_2, \sigma_2$ respectively. We can calculate the value of $P$ easily.
$$ P = \omega_1 S_1 + \omega_2 S_2 $$
Now we set $\mu_P$ as the return of the portfolio $P$. It is simple to calculate the expected return of this portfolio:
$$ E[\mu_P] = E[\omega_1 \mu_1 + \omega_2 \mu_2] = \omega_1 E[\mu_1] + \omega_2 E[\mu_2] $$
As you can see, the expected return of the overall portfolio can be directly determined using the expected returns of the assets in the portfolio as well as their associated weights. Similarly, we can use these same characteristics to determine the overall risk of the portfolio, $\sigma_p$. First, we calculate the variance of the portfolio, $\sigma_p^2 = VAR[P]$. Then we say that the correlation between $S_1$ and $S_2$ is $COR[S_1,S_2] = \frac{COV[S_1,S_2]}{\sigma_1\sigma_2} = \rho_{12}$....
Can you write Python code for it?

targets: 
stocks = np.array([100, 75])
mean_returns = np.array([4, 6])

r_12 = 0.20
cov_12 = r_12 * 0.05 * 0.08
covariance_matrix = np.array([[0.05**2,cov_12],[cov_12,0.08**2]])

weights = np.array([0.7, 0.3])

P = np.dot(weights, stocks.T)
inputs: %%R -i workDir

inFile = paste(c(workDir, 'target_genome_index.txt'), collapse='/')

tbl.target = read.delim(inFile, sep='\t', header=F)
colnames(tbl.target) = c('OTUId', 'genome_file', 'genome_ID', 'X', 'Y', 'Z')
tbl.target = tbl.target %>% distinct(OTUId)


cat('Number of target OTUs: ', tbl.target$OTUId %>% unique %>% length, '\n')
cat('----------\n')
tbl.target %>% head(n=3)
Given the above code, generate some markdown instructions for it.

targets: 
Getting list of target taxa
inputs: "Authenticate your GCP account
If you are using AI Platform Notebooks, your environment is already
authenticated. Skip this step.
If you are using Colab, run the cell below and follow the instructions
when prompted to authenticate your account via oAuth.
Otherwise, follow these steps:


In the GCP Console, go to the Create service account key
   page.


From the Service account drop-down list, select New service account.


In the Service account name field, enter a name.


From the Role drop-down list, select
   Machine Learning Engine > AI Platform Admin and
   Storage > Storage Object Admin.


Click Create. A JSON file that contains your key downloads to your
local environment.


Enter the path to your service account key as the
GOOGLE_APPLICATION_CREDENTIALS variable in the cell below and run the cell."
Please write code following the instructions in jupyter notebook style.

targets: 
import sys

# If you are running this notebook in Colab, run this cell and follow the
# instructions to authenticate your GCP account. This provides access to your
# Cloud Storage bucket and lets you submit training jobs and prediction
# requests.

if 'google.colab' in sys.modules:
  from google.colab import auth as google_auth
  google_auth.authenticate_user()

# If you are running this notebook locally, replace the string below with the
# path to your service account key and run this cell to authenticate your GCP
# account.
else:
  %env GOOGLE_APPLICATION_CREDENTIALS ''
inputs: I am working on the file "0.13/_downloads/plot_read_inverse.ipynb".
The first task is:
Show result on 3D source space
Can you write Python code for it?

targets: 
lh_points = inv['src'][0]['rr']
lh_faces = inv['src'][0]['use_tris']
rh_points = inv['src'][1]['rr']
rh_faces = inv['src'][1]['use_tris']
from mayavi import mlab  # noqa

mlab.figure(size=(600, 600), bgcolor=(0, 0, 0))
mesh = mlab.triangular_mesh(lh_points[:, 0], lh_points[:, 1], lh_points[:, 2],
                            lh_faces, colormap='RdBu')
mesh.module_manager.scalar_lut_manager.reverse_lut = True

mesh = mlab.triangular_mesh(rh_points[:, 0], rh_points[:, 1], rh_points[:, 2],
                            rh_faces, colormap='RdBu')
mesh.module_manager.scalar_lut_manager.reverse_lut = True
inputs: Constructing and estimating the model
The next step is to formulate the econometric model that we want to use for forecasting. In this case, we will use an AR(1) model via the SARIMAX class in statsmodels.
After constructing the model, we need to estimate its parameters. This is done using the fit method. The summary method produces several convenient tables showing the results.

targets: # Construct the model
mod = sm.tsa.SARIMAX(endog, order=(1, 0, 0), trend='c')
# Estimate the parameters
res = mod.fit()

print(res.summary())
inputs: <h2>Preprocessing Text</h2>

<p> After looking at our raw text, we know that there are a number of textual attributes that we will need to address before we can ultimately represent our text as quantified features. Using some built in string functions, we can address the character encoding and mixed capitalization.

targets: article[:500].decode('utf-8').lower()