Muennighoff/xP3x-sample huggingface dataset:
inputs: "Building complete data set, with the small caveat that we don't seek to load all of the positions for time constraints (for now at least)."
Please write code following the instructions in jupyter notebook style.

targets: 
max_step = 1000
start_step = 1000
stride = 10
size_data = max_step*nbC
data = np.zeros( (max_step*nbC, size_off+1, nb_type ), dtype=float )
for step in np.arange(start_step,stride*max_step+start_step,stride):
    # Distance from all atoms (saves time?)
    matrix = computeDistanceMatrix( positions[step,:,:], cell_lengths)
    for carbon in range(nbC):
        # Data Adress
        add_data = int((step-start_step)/stride)*nbC + carbon
        # C-C
        for carbon2 in range(nbC):
            # Gaussians at atomic site
            data[ add_data, 0, 0 ] += np.exp( -(matrix[carbon,carbon2]*matrix[carbon,carbon2])/(2*sigma_C*sigma_C) )
            # Gaussians with small displacement from site
            if carbon != carbon2:
                for i in range(size_off):
                    dist = getDistance( positions[step, carbon2, :], (positions[step,carbon,:]+positions_offset[i,:])%cell_lengths[0], cell_lengths )
                    data[ add_data, i+1, 0 ] += np.exp( -(dist*dist)/(2*sigma_C*sigma_C) )
        # C-O
        for oxygen in range(nbC,nb_atoms):
            # Gaussians at atomic site
            data[ add_data, 0, 1 ] += np.exp( -(matrix[carbon,oxygen]*matrix[carbon,oxygen])/(2*sigma_O*sigma_O) )
            # Gaussians with small displacement from site
            for i in range(size_off):
                dist = getDistance( positions[step, oxygen,:], (positions[step,carbon,:]+positions_offset[i,:])%cell_lengths[0], cell_lengths )
                data[ add_data, i+1, 1 ] += np.exp( -(dist*dist)/(2*sigma_O*sigma_O) )
inputs: I am working on the file "Closure a dekorator.ipynb".
The first task is:
Ako tento hack opravit?

zabalime to cele do funkcie
zmenime mutable objekt za immutable
definujeme nonlokalnu premennu
vratime vnutornu funkciu
Can you write Python code for it?

targets: 
counter = [0] # Toto je ta globalny mutable objekt 
def measure_add(a, result, counter=counter):
    if counter[0] % 200000 == 0:
        print_memory_usage()
    counter[0] = counter[0] + 1
    return a + result

# toto tu mam en pre kontrolu, aby som nespravil chybu
def make_adder():
    counter = 0
    def adder(a, result):
        nonlocal counter
        if counter % 2000000 == 0:
            print_memory_usage()
        counter += 1
        return a+result
    return adder
inputs: I am working on the file "Code/notebooks/bootcamp_timeseries_update.ipynb".
The first task is:
Exercise: write a python function named days_until that accepts one argument (a datetime.date) and returns the number of days between today and that date. Apply your function to 

May 5, 2017 (day the UG project is due)
Your birthday
Can you write Python code for it?

targets: 
def days_until(date):
    today = dt.date.today()
    numb_days = date - today
    
    return numb_days.days

project_due = dt.date(2017, 5, 5)

days_until(project_due)
inputs: %%output filename="macaw_plot" fig="png" holomap="gif"
parrot = hv.RGB.load_image('../assets/macaw.png')
parrot
Given the above code, generate some markdown instructions for it.

targets: 
Exporting specific files
During interactive exploration in the IPython Notebook, your results are always visible within the notebook itself, but you can explicitly request that any IPython cell is also exported to an external file on disk:
inputs: I am working on the file "05_CorrectiveSolutions-6nodes-2loops.ipynb".
The first task is:
Compute H matrix
Can you write Python code for it?

targets: 
S1.compute_H()

S1.H.subs(datav)
inputs: A 'many-to-one' zone connectivity occurred in this time step relative to the previous time step, meaning neither speciation or extinction were triggered by zone change. (See zone connectivity table above.)
The same two taxa of time 0 exist in time 1. Their geographic range is all that changed.

targets: se.taxa_data_frame
inputs: "4. Beam Pipeline"
Please write code following the instructions in jupyter notebook style.

targets: 
import apache_beam as beam


def run_pipeline(runner, opts):
    
    print("Sink train data files: {}".format(Params.TRANSFORMED_TRAIN_DATA_FILE_PREFIX))
    print("Sink data files: {}".format(Params.TRANSFORMED_EVAL_DATA_FILE_PREFIX))
    print("Temporary directory: {}".format(Params.TEMP_DIR))
    print("")
    
    
    with beam.Pipeline(runner, options=opts) as pipeline:
        with impl.Context(Params.TEMP_DIR): 
    
            ###### analyze & transform train #########################################################
            if(runner=='DirectRunner'):
                print("")
                print("Transform training data....")
                print("")
            
            step = 'train'
            source_query = get_source_query(step)
            
            # Read raw train data from BQ and cleanup
            raw_train_data = (
              pipeline
              | '{} - Read Data from BigQuery'.format(step) >> beam.io.Read(beam.io.BigQuerySource(query=source_query, use_standard_sql=True))
              | '{} - Extract Features'.format(step) >> beam.Map(get_features)
            )
            
            # create a train dataset from the data and schema
            raw_train_dataset = (raw_train_data, raw_metadata)
            
            # analyze and transform raw_train_dataset to produced transformed_train_dataset and transform_fn
            transformed_train_dataset, transform_fn = (
                raw_train_dataset 
                | '{} - Analyze & Transform'.format(step) >> impl.AnalyzeAndTransformDataset(preprocessing_fn)
            )
            
            # get data and schema separately from the transformed_train_dataset
            transformed_train_data, transformed_metadata = transformed_train_dataset

            # write transformed train data to sink
            _ = (
                transformed_train_data 
                | '{} - Write Transformed Data as tfrecords'.format(step) >> beam.io.tfrecordio.WriteToTFRecord(
                   ...
inputs: # With cropping then resizing then YUV
i = 0
for camera in ["left", "center", "right"]:
    image = load_img("data/"+data_frame.iloc[7100][camera].strip())
    image = img_to_array(image).astype(np.uint8)
    image = image[55:135, :, :]
    image = imresize(image, (32, 16, 3))
    yuv = cv2.cvtColor(image.astype("uint8"), cv2.COLOR_RGB2YUV)
    hsv[:, :, 0] = hsv[:, :, 0] * 0
    hsv[:, :, 1] = hsv[:, :, 1] * 0
    plt.subplot(1, 3, i+1)
    plt.imshow(yuv)
    plt.axis('off')
    plt.title(camera)
    i += 1
Given the above code, generate some markdown instructions for it.

targets: 
Converted to YUV colour space and showing only the V channel
inputs: I am working on the file "Code/IPython/bootcamp_pandas-clean.ipynb".
The first task is:
Exercise. 

Construct a list of countries with countries = weo[['ISO', 'Country']]; that is, without applying the drop_duplicates method.  How large is it?  How many duplicates have we dropped?  
What are the country codes (ISO) for Argentina and the United States?  
What are the variable codes (WEO Subject Code) for government debt (gross debt, percent of GDP) and net lending/borrowing (also percent of GDP)?  

Comment. Now that we have the country and variable codes, we can be more explicit about what we want.  We want observations with those country and variable codes.  
We work up to the solution one step at a time.  
Comparisons for series
We can construct comparisons for series (dataframe columns) much as we did with simple variables.  The difference is that we get a complete column of True/False responses, not just one.  
Mutiple comparisons have a different syntax than we saw earlier: and is replaced by &amp;, and or is replaced by |.  And when we have more than one comparison, we need to enclose them in parentheses.  
Examples.  Consider the comparisons:  

small['Units'] == 'National currency'
small['2011'] &gt;= 100
(small['Units'] == 'National currency') &amp; (small['2011'] &gt;= 100)
(small['Units'] == 'National currency') | (small['2011'] &gt;= 100)

Remind yourself what the &amp; and | do.
Can you write Python code for it?

targets: 
small['Units'] == 'National currency'

small['2011'] >= 100

(small['Units'] == 'National currency') & (small['2011'] >= 100)

(small['Units'] == 'National currency') | (small['2011'] >= 100)
inputs: 8.Prüfe mit einer Funktionen, wieviele Grossbuchstaben in folgendem Satz zu finden sind.

targets: satz = "In Oesterreich zeichnet sich ein Rechtsrutsch ab. OeVP und FPOe haben stark zugelegt. Gemaess der neusten Hochrechnung ist die Partei von Sebastian Kurz mit 31,6 Prozent der Stimmen Wahlsiegerin, auf Platz zwei folgt die SPÖ (26,9 Prozent) vor der FPOe (26,0 Prozent)."

def counting_caps(XXXXX):
    XXXXXX = 0
    for XXXXXX in elem:
        if XXXXXX.isupper():
            XXXXXXX += 1
    return  XXXXXXXX

 counting_caps(satz)
inputs: I am working on the file "MVideo/training model_2 (keras==2.0.8).ipynb".
The first task is:
Результаты
Результаты на тестовой выборке (20% от исходных данных)
целевые переменные не нормировались!
Can you write Python code for it?

targets: 
get_score(model, X_test, y_test, sparse=50)
inputs: I am working on the file "Issue_Embeddings/notebooks/08_Train_Repo_Specific_IssueLabeler.ipynb".
The first task is:
Do pre-processing (such as markdown parsing) to prepare data for model.
Can you write Python code for it?

targets: 
parsed_df = iw.process_df(combined_sig_df)

holdout_text = parsed_df[holdout_mask]

lang_model_predict = np.stack(holdout_text.text.apply(lambda x: tcl.predict(x)[2].numpy()[pred_mask]).values)

lang_model_predict.shape

len(iw.learn.data.classes)

np.array(iw.learn.data.classes)[pred_mask]

lang_model_predict_df = pd.DataFrame(lang_model_predict)
lang_model_predict_df.columns = np.array(iw.learn.data.classes)[pred_mask]
lm_df = lang_model_predict_df[[x for x in label_columns if x in lang_model_predict_df.columns]]
inputs: "Baseline model
Note that this is a very simple model to demonstrate training.
Suggestion for a stronger model:  Replace the convnet encoder with a point-based encoder like Transformer or Point Pillars and feed it points from the scene with rich features.  See function _sample_agent_points() in occupancy_flow_renderer.py for ideas on how to obtain such points and features.
Relevant publications:

Scene Transformer
Occupancy Flow Fields"
Please write code following the instructions in jupyter notebook style.

targets: 
# Number of channels output by the model.
# Occupancy of currently-observed vehicles: 1 channel.
# Occupancy of currently-occluded vehicles: 1 channel.
# Flow of all vehicles: 2 channels.
NUM_PRED_CHANNELS = 4

def _make_model_inputs(
    timestep_grids: occupancy_flow_grids.TimestepGrids,
    vis_grids: occupancy_flow_grids.VisGrids,
) -> tf.Tensor:
  """Concatenates all occupancy grids over past, current to a single tensor."""
  model_inputs = tf.concat(
      [
          vis_grids.roadgraph,
          timestep_grids.vehicles.past_occupancy,
          timestep_grids.vehicles.current_occupancy,
          tf.clip_by_value(
              timestep_grids.pedestrians.past_occupancy +
              timestep_grids.cyclists.past_occupancy, 0, 1),
          tf.clip_by_value(
              timestep_grids.pedestrians.current_occupancy +
              timestep_grids.cyclists.current_occupancy, 0, 1),
      ],
      axis=-1,
  )
  return model_inputs

def _make_model(
    model_inputs: tf.Tensor,
    config: occupancy_flow_metrics_pb2.OccupancyFlowTaskConfig,
) -> tf.keras.Model:
  """Simple convolutional model."""
  inputs = tf.keras.Input(tensor=model_inputs)

  encoder = tf.keras.applications.ResNet50V2(
      include_top=False, weights=None, input_tensor=inputs)

  num_output_channels = NUM_PRED_CHANNELS * config.num_waypoints
  decoder_channels = [32, 64, 128, 256, 512]

  conv2d_kwargs = {
      'kernel_size': 3,
      'strides': 1,
      'padding': 'same',
  }

  x = encoder(inputs)

  for i in [4, 3, 2, 1, 0]:
    x = tf.keras.layers.Conv2D(
        filters=decoder_channels[i],
        activation='relu',
        name=f'upconv_{i}_0',
        **conv2d_kwargs)(
            x)
    x = tf.keras.layers.UpSampling2D(name=f'upsample_{i}')(x)
    x = tf.keras.layers.Conv2D(
        filters=decoder_channels[i],
        activation='relu',
        name=f'upconv_{i}_1',
        **conv2d_kwargs)(
            x)

  outputs = tf.keras.layers.Conv2D(
      filters=num_output_channels,
      activation=None,
     ...
inputs: I am working on the file "3.Machine_learning/2.family_registered_habr.ipynb".
The first task is:
При больших значениях альфа можно посмотреть, на отбор признаков в действии

For large alpha values, you can look at the selection of features in action
Can you write Python code for it?

targets: 
#large
Rid=linear_model.Lasso (alpha = 0.01)
Rid.fit(X_train, y_train)
print('\n Appha:', Rid.alpha)
print(' Coefficients:', Rid.coef_)
print(' Score:', Rid.score(X_test,y_test))
inputs: "$ \frac { d\vec{v} } { d\alpha_1} = \frac {1}{R} \vec{n} = k \vec{n} $
Derivative of vectors
$ \vec{u} = u_v \vec{v} + u_n\vec{n} $
$ \frac { d\vec{u} } { d\alpha_1} =  \frac { d(u_v\vec{v}) } { d\alpha_1} + \frac { d(u_n\vec{n}) } { d\alpha_1} = 
 \frac { du_n } { d\alpha_1} \vec{n} + u_n \frac { d\vec{n} } { d\alpha_1} + \frac { du_v } { d\alpha_1} \vec{v} + u_v \frac { d\vec{v} } { d\alpha_1} = \frac { du_n } { d\alpha_1} \vec{n} - u_n k \vec{v} + \frac { du_v } { d\alpha_1} \vec{v} + u_v k \vec{n}$
Then
$ \frac { d\vec{u} } { d\alpha_1} = \left( \frac { du_v } { d\alpha_1} - u_n k \right) \vec{v} + \left( \frac { du_n } { d\alpha_1} + u_v k \right) \vec{n}$
$ \frac { d\vec{u} } { d\alpha_2} = \frac { d(u_n\vec{n}) } { d\alpha_2} + \frac { d(u_v\vec{v}) } { d\alpha_2} = 
 \frac { du_n } { d\alpha_2} \vec{n} + u_n \frac { d\vec{n} } { d\alpha_2} + \frac { du_v } { d\alpha_2} \vec{v} + u_v \frac { d\vec{v} } { d\alpha_2} = \frac { du_n } { d\alpha_2} \vec{n} + \frac { du_v } { d\alpha_2} \vec{v} $
Base Vectors $\vec{R}_1, \vec{R}_2, \vec{R}_3$"
Please write code following the instructions in jupyter notebook style.

targets: 
R_alpha=r+alpha3*n
R_alpha

R1=R_alpha.diff(alpha1)
R2=R_alpha.diff(alpha2)
R3=R_alpha.diff(alpha3)

R1

R1x = R1.dot(N.i)

simplify(R1x)

R1z = R1.dot(N.k)
simplify(R1z)

R2

R3
inputs: <h1> 3. Make sure outputs do not clobber each other </h1>

We append the trial-number to the output directory.

targets: !grep -A 5 "trial" taxifare/trainer/task.py
inputs: I am working on the file "Notebooks/Unused/Resizing.ipynb".
The first task is:
Resize the image with default downsampling:
Can you write Python code for it?

targets: 
%timeit im.resize((hsize, vsize))
resized_im = im.resize((hsize, vsize), Image.NEAREST)
inputs: <a name="equilibrium"></a>
Equilibrium
Equilibrium is at 0.693.  Why?  Consider what the cost is measuring, the binary cross entropy.  If we have random guesses, then we have as many 0s as we have 1s.  And on average, we'll be 50% correct.  The binary cross entropy is:
\begin{align}
\sum_i \text{X}_i * \text{log}(\tilde{\text{X}}_i) + (1 - \text{X}_i) * \text{log}(1 - \tilde{\text{X}}_i)
\end{align}
Which is written out in tensorflow as:
python
(-(x * tf.log(z) + (1. - x) * tf.log(1. - z)))
Where x is the discriminator's prediction of the true distribution, in the case of GANs, the input images, and z is the discriminator's prediction of the generated images corresponding to the mathematical notation of $\tilde{\text{X}}$.  We sum over all features, but in the case of the discriminator, we have just 1 feature, the guess of whether it is a true image or not.  If our discriminator guesses at chance, i.e. 0.5,  then we'd have something like:
\begin{align}
0.5 * \text{log}(0.5) + (1 - 0.5) * \text{log}(1 - 0.5) = -0.693
\end{align}
So this is what we'd expect at the start of learning and from a game theoretic point of view, where we want things to remain.  So unlike our previous networks, where our loss continues to drop closer and closer to 0, we want our loss to waver around this value as much as possible, and hope for the best.

targets: equilibrium = 0.693
margin = 0.2
inputs: Analyze the mesh fission rate tally

targets: # Find the mesh tally with the StatePoint API
tally = sp.get_tally(name='mesh tally')

# Print a little info about the mesh tally to the screen
print(tally)
inputs: # Apply the Classifier we trained to the test data (which, remember, it has never seen before)
clf.predict(test[features])
Given the above code, generate some markdown instructions for it.

targets: 
Huzzah! We have done it! We have officially trained our random forest Classifier! Now let's play with it. The Classifier model itself is stored in the clf variable.
Apply Classifier To Test Data
If you have been following along, you will know we only trained our classifier on part of the data, leaving the rest out. This is, in my humble opinion, the most important part of machine learning. Why? Because by leaving out a portion of the data, we have a set of data to test the accuracy of our model!
Let's do that now.
inputs: "Az ábra paramétereit (title, ylabel stb.) a matplotlibben megszokott módon állíthatjuk be.
Elsőként növeljük meg alapértelmezetten a tengelyfeliratokat:"
Please write code following the instructions in jupyter notebook style.

targets: 
rcParams["font.size"]=15
inputs: "Building a Coding Tree
The module heapq provides priority queues.  The api is given at https://docs.python.org/3/library/heapq.html.
We will use two methods:
* heapq.heappush(H, p)
  pushes the priority p onto the heap.  For this to be possible, p has to be an object of a class that provides the 
  method __lt__.  The expression p.__lt__(q) is True if p is smaller than q.
* heapq.heappop(H) 
  removes and returns the element from the heap H that has the highest priority, i.e. the smallest element.
The implementation of this module represents heaps as arrays.  Therefore, 
* to test whether a heap H is empty we can write H == [] since [] is the empty heap.
* to create an empty heap we write H = [],
* to get the element with the highest priority in the heap H we write H[0],
* to get the number of elements in H we can write len(H)."
Please write code following the instructions in jupyter notebook style.

targets: 
import heapq

H = []
heapq.heappush(H, 7)
heapq.heappush(H, 1)
heapq.heappush(H, 0)
heapq.heappush(H, 6)
H

a = heapq.heappop(H)
print('a = ', a)
H
inputs: # parameters: number of samples and according length of acf
N = int( 1e2 )
N_acf_range = np.arange( - N + 1, N, 1 )

# number of realizations for averaging    
N_real = int( 1e2 )

# number of freq. points and freq. range
N_freq = 512            
Ome = np.linspace(-np.pi, np.pi, N_freq)


# filtering noise?!
filtered = 1
Given the above code, generate some markdown instructions for it.

targets: 
Parameters
inputs: I am working on the file "FADL1/L3CA2_rossmann.ipynb".
The first task is:
1.5.3 Test
Can you write Python code for it?

targets: 
m = md.get_learner(emb_szs, len(df.columns)-len(cat_vars), 
                   0.04, 1, [1000,500], [0.001,0.01], y_range=y_range)
λr = 1e-3

m.fit(λr, 3, metrics=[exp_rmspe])

m.fit(λr, 3, metrics=[exp_rmspe], cycle_len=1)

m.save('val0')

# m.load('val0')

x,y = m.predict_with_targs()

exp_rmspe(x,y)

pred_test = m.predict(True)

pred_test = np.exp(pred_test)

joined_test['Sales'] = pred_test

csv_fn = f'{PATH}tmp/sub.csv'

joined_test[['Id','Sales']].to_csv(csv_fn, index=False)

FileLink(csv_fn)
inputs: I am working on the file "Algorithm/Itertools.ipynb".
The first task is:
islice returns selcted item by index
Can you write Python code for it?

targets: 
from itertools import * 
print('Stop at 5')
for i in islice(range(100), 5):
    print(i, end =' ')
print('\n')

print('start at 5, and stop at 10')
for i in islice(range(100), 5, 10):
    print(i, end=' ')
print('\n')

print('by ten to 100')
for i in islice(range(100), 0,100, 10):
    print(i, end=' ')
print('\n')
inputs: "As you can see, our we have now stored the same raster data into two different types of variables:
-raster is a Gdal Dataset 
-rasterArray is a numpy array 
The GDAL library also comes with a module gdal_array that works as an interface between NumPy and GDAL. Gdal_array reads and writes raster files to and from NumPy arrays directly from file:"
Please write code following the instructions in jupyter notebook style.

targets: 
from osgeo import gdal_array
#from osgeo import gdalnumeric

# Read raster data as numeric array from file
rasterArray = gdal_array.LoadFile(filepath)
inputs: I am working on the file "Data Science From Scratch/8 - Gradient Descent.ipynb".
The first task is:
Using The Gradient
Here is an example of using gradient descent to find the minimum of a 3-dimensional vector:
Can you write Python code for it?

targets: 
def dot(v, w):
    """v_1 * w_1 + ... + v_n * w_n"""
    return sum(v_i * w_i for v_i, w_i in zip(v, w))

def vector_subtract(v, w):
    """subtracts corresponding elements"""
    return [v_i - w_i for v_i, w_i in zip(v, w)]

def sum_of_squares(v):
    """v_1 * v_1 + ... + v_n * v_n"""
    return dot(v, v)

def squared_distance(v, w):
    """(v_1 - w_1) ** 2 + ... + (v_n - w_n) ** 2"""
    return sum_of_squares(vector_subtract(v, w))

def distance(v, w):
    return math.sqrt(squared_distance(v, w))

def step(v, direction, step_size):
    """move step_size in the direction from v"""
    return [v_i + step_size * direction_i for v_i, direction_i in zip(v, direction)]

def sum_of_squares_gradient(v):
    return [2 * v_i for v_i in v]

# pick a random starting point
v = [random.randint(-10,10) for i in range(3)]

tolerance = 0.0001

while True:
    gradient = sum_of_squares_gradient(v) # compute the gradient at v
    next_v = step(v, gradient, -0.01) # take a negative gradient step
    if distance(next_v, v) < tolerance: # stop if we're converging
        break
    v = next_v # continue if we're not
v
inputs: "Another benefit of this kind of model is that the last convolutional layer has to learn to classify each part of the image (since there's only an average pooling layer after). Let's create a function that grabs the output of this layer (which is the 4th-last layer of our model)."
Please write code following the instructions in jupyter notebook style.

targets: 
l = lrg_model.layers
conv_fn = K.function([l[0].input, K.learning_phase()], l[-4].output)

def get_cm(inp, label):
    conv = conv_fn([inp,0])[0, label]
    return scipy.misc.imresize(conv, (360,640), interp='nearest')
inputs: 3. 利用PyTorch的Tensor和autograd实现
两层网络的反向求导比较容易，但如果层数加多，在手动求导就会变得很复杂。因此深度学习平台都提供了自动求导功能，PyTorch的Autograd中的自动求导功能可以使反向求导简捷且灵活。要注意的是计算图的构建需要用autograd中的Variable将需要并入计算图中的变量进行封装，并设置相关属性。

targets: import torch as T
from torch.autograd import Variable

#先定义网络结构: batch_size, Input Dimension, Hidden Dimension, Output Dimension 
N, D_in, D_hidden, D_out = 10, 20, 30, 5  

#随机生成输入和输出数据, 并用Variable对输入输出进行封装，同时在计算图形中不要求求导
x = Variable(T.randn(N, D_in), requires_grad=False)
y = Variable(T.randn(N, D_out), requires_grad=False)

#对输入层和输出层的参数进行初始化，并用Variable封装，同时要求求导
w1 = Variable(T.randn(D_in, D_hidden), requires_grad=True)
w2 = Variable(T.randn(D_hidden, D_out), requires_grad=True)

learning_rate = 0.001

#循环更新参数，每个循环前向和反向各计算一次
for i in xrange(50):
    
    # 计算前向通道
    #mm should also work as x is a matrix. The matrix multiplication will be summarized in another post
    h_linear = x.matmul(w1)     #10x20 and 20x30 produce 10x30, which is the shape of h_linear
    h_relu = h_linear.clamp(min=0) #note one have to use np.maximum but not np.max, 10x30
    y_pred = h_relu.matmul(w2)  #10x30 and 30x5 produce 10x5
    
    #定义代价函数
    loss = 0.5 * (y_pred - y).pow(2).sum() #sum squared error as loss
    
    loss.backward()
    
    
    #梯度下降法更新参数
    w1.data -= learning_rate * w1.grad.data  #note that we are updating the 'data' of Variable w1
    w2.data -= learning_rate * w2.grad.data
    
    #PyTorch中，将grad中的值在循环中进行累积，当不须此操作时，应清零
    w1.grad.data.zero_()
    w2.grad.data.zero_()
inputs: I am working on the file "Mixmatch/streetview_dataset/parse_data_to_tfrecord_main.ipynb".
The first task is:
streetview_v3-test
This test set uses all TC11 images as positive cases, and handpick UCF images with confidence < 0.2 as negative cases.
Can you write Python code for it?

targets: 
# Filter UCF images and copy files for furthur handpicking test set.
for i in range(int(0.9 * INPUT_RECORD_CNT), INPUT_RECORD_CNT):
    file_name = "./streetlearn_detections_tfexample-" + str(i).zfill(5) + "-of-01000.tfrecord"
    parsed_image_dataset = read_tfrecord(os.path.join(INPUT_RECORD_DIR, file_name))
    filter_image_with_confidence_threshold(parsed_image_dataset, INPUT_UCF_IMG_DIR, OUTPUT_UCF_IMG_DIR, 0.2)

# Build streetview_v3_64-test. Write test tfrecord from TC11 dataset and filtered UCF
test_writer = tf.io.TFRecordWriter(OUTPUT_RECORD_DIR + OUTPUT_TEST_RECORD_FILENAME)
test_image_path = os.path.join(INPUT_TC11_IMG_DIR, 'img')
write_tfrecord_from_images(test_image_path, 1, test_writer)
write_tfrecord_from_images(INPUT_UCF_FILTER_IMG_DIR, 0, test_writer)
test_writer.close()
inputs: I am working on the file "ImageSimilarity_and_ClusterDemo.ipynb".
The first task is:
To check whether the file search has succeeded, display some found images to get a feeling of the data we are going to deal with.
Can you write Python code for it?

targets: 
img1=imread(jpegFilePaths[0])
img2=imread(jpegFilePaths[1388])
img3=imread(jpegFilePaths[1389])

#Creates two subplots and unpacks the output array immediately
f, (ax1, ax2,ax3) = plt.subplots(1, 3, sharex='all', sharey='all')
ax1.axis('off')
ax2.axis('off')
ax3.axis('off')

ax1.set_title("Image #0")
ax1.imshow(img1)
ax2.set_title("Image #1388")
ax2.imshow(img2)
ax3.set_title("Image #1389")
ax3.imshow(img3)
inputs: Below we plot the smoothed probability of the economy operating in a low-production state, and again include the NBER recessions for comparison.

targets: fig, ax = plt.subplots(figsize=(12,3))

ax.plot(res_filardo.smoothed_marginal_probabilities[0])
ax.fill_between(usrec.index, 0, 1, where=usrec['USREC'].values, color='gray', alpha=0.2)
ax.set_xlim(dta_filardo.index[6], dta_filardo.index[-1])
ax.set(title='Smoothed probability of a low-production state');
inputs: "0. General note


This notebook shows an example of how to conduct equation of state fitting for the pressure-volume-temperature data using pytheos.


Advantage of using pytheos is that you can apply different pressure scales and different equations without much coding.  


We use data on SiC from Nisr et al. (2017, JGR-Planet).


1. Global setup"
Please write code following the instructions in jupyter notebook style.

targets: 
import numpy as np
import uncertainties as uct
import pandas as pd
from uncertainties import unumpy as unp
import matplotlib.pyplot as plt
import pytheos as eos
inputs: "And now applying this on some real data
These exercises are based on the PyCon tutorial of Brandon Rhodes (so all credit to him!) and the datasets he prepared for that. You can download these data from here: titles.csv and cast.csv and put them in the /data folder.
cast dataset: different roles played by actors/actresses in films

title: title of the film
name: name of the actor/actress
type: actor/actress
n: the order of the role (n=1: leading role)"
Please write code following the instructions in jupyter notebook style.

targets: 
cast = pd.read_csv('data/cast.csv')
cast.head()

titles = pd.read_csv('data/titles.csv')
titles.head()
inputs: Broadcasting
Broadcasting is Numpy's terminology for performing mathematical operations between arrays of different shapes. If certain assumptions hold, the smaller of the two arrays is said to be broadcast to the size of the larger array in order to make the two arrays compatible so  element-to-element operations can be performed. <a href="http://eli.thegreenplace.net/2015/broadcasting-arrays-in-numpy/">Read more</a>.

targets: macro_nutrients = np.array([[0.3, 2.5, 3.5],
                            [2.9, 27.5, 0],
                            [0.4, 1.3, 23.9],
                            [14.4, 6, 2.3]])
calories_per_macro = np.array([9, 4, 4])
macro_nutrients * calories_per_macro

arr4 = np.arange(0, 10)
arr4

arr4[0:4] = 10
arr4
inputs: "3.3- Defining the choice set for each latent class"
Please write code following the instructions in jupyter notebook style.

targets: 
# Set the available alternatives for each latent class
# Each array entials the alternatives available in the choice set for each latent class 
# NOTE: By default the code does not require the user to specify the choice set
#       for each class. The code assumes that all alternatives are available in the choice
#       set for each latent class.
# We are assuming that the bike alternative does not exist in the choice set for latent class 1
# We are assuming that all alternatives are available for latent class 2
avail_alts = (np.array([1,2,4,5,6]),
              np.array([1,2,3,4,5,6]))
inputs: I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Random_Variables/notebook.ipynb".
The first task is:
Now let's see how a theoretical normal curve fits against the actual values.
Can you write Python code for it?

targets: 
x = np.linspace(-(sample_mean + 4 * sample_std_dev), (sample_mean + 4 * sample_std_dev), len(returns))
sample_distribution = ((1/(sample_std_dev * 2 * np.pi)) * 
                       np.exp(-(x - sample_mean)*(x - sample_mean) / (2 * sample_std_dev * sample_std_dev)))
plt.hist(returns.price, bins = 20, normed = True);
plt.plot(x, sample_distribution)
plt.xlabel('Value')
plt.ylabel('Occurrences');
inputs: Calculate the Average Noise
Note: this will compute the average spectrogram using all of the "noise" spectrograms in the primary_small data. This wil take a bit of time

targets: zz = zipfile.ZipFile(mydatafolder + '/' + 'primary_small.zip')
csv_data = zz.namelist()[1:]
        
data = []
N = 1
for i,v in enumerate(csv_data):
    d = zz.open(v).read()
    aca = ibmseti.compamp.SimCompamp(d)
    if aca.header()['signal_classification'] == 'noise':
        spec = aca.get_spectrogram()
        if len(data) == 0:
            data = spec
        else:
            data = data + spec
            N += 1
data = data/N
print "done"

# This is what the average noise plot looks like
fig, ax = plt.subplots(figsize=(10, 5))   
ax.imshow(np.log(data), aspect = 0.5*float(data.shape[1]) / data.shape[0])

# This is the power spectrum of the image above.
plt.plot(np.sum(data, axis=0))
inputs: I am working on the file "Data_analysis/SNP-indel-calling/dadi/MODIFIED_SPECTRA/01_modified_spectra_synthesis.ipynb".
The first task is:
This is good convergence.
The best-fit ancient migration model had a log likelihood of -12156. So only one log likelihood unit worse.
Can you write Python code for it?

targets: 
ll_s = 12156.061309
ll_c = 12155.538032
D = 2 * (ll_s - ll_c)
D

# calculate p-value for Chi-square dist.
# the weights specify a weighted sum of chi^2 distributions with 0 and 1 d.o.f
# this is because Tsc is 0 in the split_asym_mig_iso model and at the boundary of the parameter space
p = dadi.Godambe.sum_chi2_ppf(D, weights=(0.5, 0.5))
p
inputs: Answer: 79.35%.

Adding the feature Age as a condition in conjunction with Sex improves the accuracy by a small margin more than with simply using the feature Sex alone. Now it's your turn: Find a series of features and conditions to split the data on to obtain an outcome prediction accuracy of at least 80%. This may require multiple features and multiple levels of conditional statements to succeed. You can use the same feature multiple times with different conditions. 
Pclass, Sex, Age, SibSp, and Parch are some suggested features to try.
Use the survival_stats function below to to examine various survival statistics.
Hint: To use mulitple filter conditions, put each condition in the list passed as the last argument. Example: ["Sex == 'male'", "Age &lt; 18"]

targets: vs.survival_stats(data, outcomes, 'Sex', ["Sex == 'male'", "Pclass == 1"])
inputs: I am working on the file "Digital Text Analysis.ipynb".
The first task is:
Tokenization
In the previous paragraph, we have adopted an extremely crude definition of a 'word', namely as a string of characters that doesn't contain any whitespace. There are of course many problems that arise if we use such a naive definition. Can you think of some?
In computer science, and computational linguistics in particular, people have come up with much smarter ways to divide texts into words. This process is called tokenization, which refers to the fact that this process divides strings of characters into a list of more meaningful tokens. One interesting package which we can use for this, is nltk (the Natural Language Toolkit), a package which has been specifically designed to deal with language problems. First, we have to import it, since it isn't part of the standard library of Python:
Can you write Python code for it?

targets: 
import nltk
inputs: "Average Every 15 Mins Hour 2-3 After ROP Exam"
Please write code following the instructions in jupyter notebook style.

targets: 
print result3
inputs: ## create a baba object linked to a data file and newick tree
bb = ipa.baba(data=locifile, newick=newick)

## generate all possible abba-baba tests meeting a set of constraints
bb.generate_tests_from_tree(
    constraint_dict={
        "p4": ["32082_przewalskii", "33588_przewalskii"],
        "p3": ["33413_thamno"],
    })

## run all tests linked to bb 
bb.run(ipyclient)
Given the above code, generate some markdown instructions for it.

targets: 
Short tutorial: calculating abba-baba statistics
To give a gist of what this code can do, here is a quick tutorial version, each step of which we explain in greater detail below. We first create a 'baba' analysis object that is linked to our data file, in this example we name the variable bb. Then we tell it which tests to perform, here by automatically generating a number of tests using the generate_tests_from_tree() function. And finally, we calculate the results and plot them.
inputs: I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Calculate a boolean array whether the predicted classes for the images are correct.
Can you write Python code for it?

targets: 
def correct_prediction(images, labels, cls_true):
    # Calculate the predicted labels.
    pred_labels = predict_labels(images=images)

    # Calculate the predicted class-number for each image.
    cls_pred = np.argmax(pred_labels, axis=1)

    # Create a boolean array whether each image is correctly classified.
    correct = (cls_true == cls_pred)

    return correct
inputs: ".. sidebar:: The middle column of the Events array
MNE-Python events are actually *three* values: in between the sample
number and the integer event code is a value indicating what the event
code was on the immediately preceding sample. In practice, that value is
almost always `0`, but it can be used to detect the *endpoint* of an
event whose duration is longer than one sample. See the documentation of
:func:`mne.find_events` for more details.

If you don't provide the name of a STIM channel, :func:~mne.find_events
will first look for MNE-Python config variables &lt;tut-configure-mne&gt;
for variables MNE_STIM_CHANNEL, MNE_STIM_CHANNEL_1, etc. If those are
not found, channels STI 014 and STI101 are tried, followed by the
first channel with type "STIM" present in raw.ch_names. If you regularly
work with data from several different MEG systems with different STIM channel
names, setting the MNE_STIM_CHANNEL config variable may not be very
useful, but for researchers whose data is all from a single system it can be
a time-saver to configure that variable once and then forget about it.
:func:~mne.find_events has several options, including options for aligning
events to the onset or offset of the STIM channel pulses, setting the minimum
pulse duration, and handling of consecutive pulses (with no return to zero
between them). For example, you can effectively encode event duration by
passing output='step' to :func:mne.find_events; see the documentation
of :func:~mne.find_events for details. More information on working with
events arrays (including how to plot, combine, load, and save event arrays)
can be found in the tutorial tut-event-arrays.
Reading embedded events as Annotations
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Some EEG/MEG systems generate files where events are stored in a separate
data array rather than as pulses on one or more STIM channels. For example,
the EEGLAB format stores events as a collection of arrays in the :file:.set
file. When reading those files, MNE-Python will automatically convert..."
Please write code following the instructions in jupyter notebook style.

targets: 
testing_data_folder = mne.datasets.testing.data_path()
eeglab_raw_file = os.path.join(testing_data_folder, 'EEGLAB', 'test_raw.set')
eeglab_raw = mne.io.read_raw_eeglab(eeglab_raw_file)
print(eeglab_raw.annotations)
inputs: I am working on the file "Finance-Banking-Related/Lending Club Loan Data - Exploratory Data Analysis [Part I, Python notebook, kaggle].ipynb".
The first task is:
F1. Nominal Categorical Attributes
We chose to define as nominal categorical attributes the variables below:

'home_ownership': The home ownership status provided by the borrower during registration (Values: 'ANY', 'MORTGAGE', 'NONE', 'OTHER', 'OWN', 'RENT').
'pymnt_plan': Indicates if a payment plan has been put in place for the loan (Values: 'y'/'n').
'purpose': A category provided by the borrower for the loan request (Values: 'car', 'credit_card', 'debt_consolidation', 'educational', ..., 'renewable_energy', 'small_business', 'vacation', 'wedding').
'zip_code': The first 3 numbers of the zip code provided by the borrower in the loan application (Values: US Zip Codes).
'addr_state': The state provided by the borrower in the loan application (Values: US State Name).
'initial_list_status': The initial listing status of the loan (Values: 'w'/'f').
'policy_code': Publicly available products ('policy_code'='1'), New products not publicly available ('policy_code'='2').
'application_type': Indicates whether the loan is an individual application or a joint application with two co-borrowers (Values: 'INDIVIDUAL'/'JOINT').

Among them, these two cannot be important for the response variable outcome, 'loan_status':

'pymnt_plan' (having only 10 'y' values, and all the remaining ones being 'n')
'policy_code' (there are no products which are not publicly available, i.e. 'policy code': '2')

Note, that the 'application_type' attribute which indicates whether a loan is an individual or a joint application 
has only 511 'JOINT' applications, whereas all the remaining ones (886868!) have been made by individuals. However, we choose to keep this variable for the time being, and check below if the response variable appears any difference among the two types of loans.
Can you write Python code for it?

targets: 
pprint(nominal_categ_attribs)

categ_df[nominal_categ_attribs].info()

for attrib in nominal_categ_attribs:
    s = 'Nominal Categorical Attribute: "%s":' % attrib
    print(s)
    print('-' * (len(s)+3))
    pprint(categ_df.loc[:3, attrib])
    s = '\nValue Counts:'
    print(s)
    print('-' * (len(s)+3))
    pprint(pd.value_counts(categ_df[attrib]))
    print('\n')
inputs: "<h3> SELECTION DOWNLOAD </h3>"
Please write code following the instructions in jupyter notebook style.

targets: 
for nc in selected_netCDFs:
    last_idx_slash = file_name.rfind('/')
    ncdf_file_name = file_name[last_idx_slash+1:]
    folders = file_name.split('/')[3:len(file_name.split('/'))-1]
    host = file_name.split('/')[2] #distribution unit
    
    ftp=ftplib.FTP(host,user,password) 
    for folder in folders:
    ftp.cwd(folder)
    
    local_file = open(ncdf_file_name, 'wb')
    ftp.retrbinary('RETR '+ncdf_file_name, local_file.write)
    local_file.close()
    ftp.quit()
inputs: "Create a dashboard with Data Studio
Follow the steps in this section to create an interactive, shareable dashboard of the forecasted data by using Data Studio. 
Create a view containing the data for the dashboard
Create a view that concatenates the historical and forecasted data. The SQL before the UNION ALL clause selects the historical data. The SQL after the UNION ALL clause uses ML.FORECAST to generate the forecasted value and the prediction interval. The query uses different fields for history_value and forecasted_value so that you can plot them in different colors in a following step."
Please write code following the instructions in jupyter notebook style.

targets: 
%%bigquery --params $ARIMA_PARAMS  --project $PROJECT_ID 

CREATE OR REPLACE VIEW bqmlforecast.outputdata_datastudio AS (
  SELECT
    date AS timestamp,
    item_name,
    total_amount_sold AS history_value,
    NULL AS forecast_value,
    NULL AS prediction_interval_lower_bound,
    NULL AS prediction_interval_upper_bound
  FROM
    bqmlforecast.training_data
  UNION ALL
  SELECT
    EXTRACT(DATE
    FROM
      forecast_timestamp) AS timestamp,
    item_name,
    NULL AS history_value,
    forecast_value,
    prediction_interval_lower_bound,
    prediction_interval_upper_bound
  FROM
    ML.FORECAST(MODEL bqmlforecast.arima_model,
      STRUCT(30 AS horizon, 0.9 AS confidence_level)) 
  ORDER BY timestamp
  )
inputs: Assign a Series to a column (note if assigning a list or array, the length must match the DataFrame, unlike a Series):

targets: unempl = pd.Series([6.0, 6.0, 6.1], index=[2, 3, 4])
df_3['unempl'] = unempl
df_3
inputs: I am working on the file "LP/Introduction-to-linear-programming/Introduction to Linear Programming with Python - Part 5.ipynb".
The first task is:
Let's take a look at the optimal production schedule output for each month from each factory. For ease of viewing we'll output the data to a pandas DataFrame.
Can you write Python code for it?

targets: 
x=production[1, 'A']
type(x)

output = []
for month, factory in production:
    var_output = {
        'Month': month,
        'Factory': factory,
        'Production': production[(month, factory)].varValue,
        'Factory Status': factory_status[(month, factory)].varValue
    }
    output.append(var_output)
output_df = pd.DataFrame.from_records(output).sort_values(['Month', 'Factory'])
output_df.set_index(['Month', 'Factory'], inplace=True)
output_df
inputs: # Example 1, numerically.
def f_SIR(t,y,alpha,beta):
    S = y[0]
    I = y[1]
    R = y[2]
    
    Sp = -alpha*I*S
    Ip = alpha*I*S-beta*I
    Rp = beta*I
    
    return np.array([Sp,Ip,Rp])

# Initial condition
N = 1e6
alpha = 1e-6
beta = 1/3

S0 = 9e5
I0 = 1e5
y0 = np.array([S0, I0, N-S0-I0])

T = 20

# time where we want your solution
t = np.linspace(0, T, 100)
sol = solve_ivp(f_SIR, [0,T], y0, t_eval=t, args=(alpha,beta))

plt.figure()
plt.plot(t, sol.y[0], 'b', label='S(t)')
plt.plot(t, sol.y[1], 'r', label='I(t)')
plt.plot(t, sol.y[2], 'g', label='R(t)')
plt.legend(loc='right', bbox_to_anchor=(1.3, 0.5))
plt.xlabel('t')
plt.ylabel('number of individuals')
plt.grid(True)
plt.show()
Given the above code, generate some markdown instructions for it.

targets: 
2.2.2.- SIR models in epidemiology
inputs: "Astropy tables
There is great deal of flexibility in the way that a table can be initially constructed:

Read an existing table from a file or web URL
Add columns of data one by one
Add rows of data one by one

From an existing data structure in memory:


List of data columns

Dict of data columns
List of row dicts
Numpy homgeneous array or structured array
List of row records

See the documentation section on Constructing a table for the gory details and plenty of examples."
Please write code following the instructions in jupyter notebook style.

targets: 
t = Table()
t['name'] = ['larry', 'curly', 'moe', 'shemp']
t['flux'] = [1.2, 2.2, 3.1, 4.3]
inputs: I am working on the file "Heteroscedasticity.ipynb".
The first task is:
P値は0.003より有意水準1%でも帰無仮説BP=0を棄却することができ、不均一分散であると結論することができます。
例題6-3
「k0602.csv」を用いた不均一分散のデータである場合の回帰分析。
不均一分散が存在する場合、変数を対数化することによって不均一分散の状態の解消を行います。
先ほど読み込んだ「k0602.csv」のデータから変数の対数化を行います。
dataの中に新しいコラム「lnX」「lnY」を追加します。
Can you write Python code for it?

targets: 
data["lnX"] = np.log(data["X"])
data["lnY"] = np.log(data["Y"])
data

# 説明変数設定
X = data['lnX']
X = sm.add_constant(X)
X

# 被説明変数設定
Y = data['lnY']
Y

# OLSの実行(Ordinary Least Squares: 最小二乗法)
model = sm.OLS(Y,X)
results = model.fit()
print(results.summary())
# グラフ生成
plt.plot(data["lnX"], data["lnY"], 'o', label="data")
plt.plot(data["lnX"], results.fittedvalues, label="OLS")
plt.xlim(min(data["lnX"])-0.5, max(data["lnX"])+0.5)
plt.ylim(min(data["lnY"])-0.5, max(data["lnY"])+0.5)
plt.title('Example6-3')
plt.legend(loc=2)
plt.show()

# BPテスト
BP = smsdia.het_breushpagan(results.resid, results.model.exog)
print('Breusch-Pagan test : ')
print('BP = ', BP[0])
print('p-value = ', BP[1])
inputs: I am working on the file "31_ML-IV/ML_4.ipynb".
The first task is:
Homework
Training Your Final Classifiers!
Once you have parameters that you're happy with for each dataset, go ahead and make one final classifier. You must set the random state, and your code must produce a classifier named d1_classifier for the disease 1 dataset and d2_classifier from the disease two dataset. Each one should assume that data have been loaded into variables named X1 and X2 respectively for features, and y1 and y2 for labels. You can assume that these have already been loaded (just like we did above). You don't have to report everything you try below, but include the training score and testing score (using classifier.score) and accuracy + 2x standard deviation from cross validation (as in prelab) for your final classifiers.
Can you write Python code for it?

targets: 
# CODE TO CREATE THE d1_classifier

# CODE TO CREATE THE d2_classifier
inputs: # we can unfreeze and this will train the entire network
learn.unfreeze()
# bn_freeze stops updates of 
learn.bn_freeze(True) # important for large models --> stops updates of BN moving avgs
                      # use for models w/ ~ > 34 layers, & larger imgs & reg.s of intrst & norml imgs
%time learn.fit([1e-5, 1e-4, 1e-2], 1, cycle_len=1)

# We use Test Time Augmentation to ensure we get the best predictions we can
%time log_preds, y = learn.TTA()

# Finally, this gives us about 99.6% accuracy
metrics.log_loss(y, np.exp(log_preds)), accuracy(log_preds, y)
Given the above code, generate some markdown instructions for it.

targets: 
We didn't specify precompute=True because it only makes it a little faster for this first set. So we skipped it. It's a shortcut that caches some of the intermediate steps that don't have to be recalculated each time. NOTE: when we use precomputed activations: data augmentation will not work.
So if you specfy data augmentation, such as w/ aug_tfms=..., and also precompute=True fastai will not actually do any data augmentation.
NOTE: if you're using big / deep model like ResNet50 or ResNeXt101 on a dataset v.similar to ImageNet (like this cats/dogs datset; ie: side-on photos of standard objects of a similar size to ImageNet: 200~500 pxls), you should probably run learn.bn_freeze(True) after learn.unfreeze().
What this is doing is causing the BatchNorm moving averages to stop updating.
(Not currently supported by any library besides fastai, and is v.important)
Needed to restart kernel & set batch-size to 16 to fit it in Gfx Memory
rerun lines: tfms = ...; data = ...; learn = ...;
We run one more epoch, training the entire network: