I am working on the file "Iris_DataSet.ipynb".
The first task is:
Visualizing the data
Visualizing the data can help us better understand the data and make use of it. The following block of code will create a plot of sepal length (x-axis) vs sepal width (y-axis). The colors of the datapoints correspond to the labeled species of iris for that example.
After plotting, look at the data. What do you notice about the way it is arranged?
Can you write Python code for it?

# Plot the data

# Sepal length and width
X_sepal = X[:,:2]
# Get the minimum and maximum values with an additional 0.5 border
x_min, x_max = X_sepal[:, 0].min() - .5, X_sepal[:, 0].max() + .5
y_min, y_max = X_sepal[:, 1].min() - .5, X_sepal[:, 1].max() + .5

plt.figure(figsize=(8, 6))

# Plot the training points
plt.scatter(X_sepal[:, 0], X_sepal[:, 1], c=y, cmap=cmap_bold)
plt.xlabel('Sepal length (cm)')
plt.ylabel('Sepal width (cm)')
plt.title('Sepal width vs length')

# Set the plot limits
plt.xlim(x_min, x_max)
plt.ylim(y_min, y_max)

plt.legend(labelList, labelNames)

plt.show()


I am working on the file "Isosurface-in-volumetric-data.ipynb".
The first task is:
Although our 3D data  is defined in  $[-2,2]^3$, the function measure.marching_cubes_lewiner returns verts translated such that they belong
to the parallelipiped $[0,4]^3$, provided that the spacing key in this function is the same as the spacing
of voxels in the initial parallelipiped,
i.e. spacing=(X[1,0, 0]-X[0,0,0], Y[0,1, 0]-Y[0,0,0], Z[0,0, 1]-Z[0,0,0]):
Can you write Python code for it?

verts, faces = measure.marching_cubes_lewiner(surf_eq, 1.2,
                                              spacing=(X[1,0, 0]-X[0,0,0], Y[0,1, 0]-Y[0,0,0],
                                                       Z[0,0, 1]-Z[0,0,0]))[:2]
title = 'Isosurface in volumetric data'


I am working on the file "Isosurface-in-volumetric-data.ipynb".
The first task is:
Define a meshgrid on our volume and the function that for the (iso)surface equation:
Can you write Python code for it?


I am working on the file "Issue_Embeddings/notebooks/01_AcquireData.ipynb".
The first task is:
Cached pre-processed data
Since ~19M GitHub issues take a long time to pre-process, the pre-processed files are available here:
https://storage.googleapis.com/issue_label_bot/pre_processed_data/1_processed_csv/processed_part00{00-99}.csv
Partition Data Into Train/Validation Set
Set aside random 10 files (out of 100) as the Validation set
Can you write Python code for it?

from pathlib import Path
from random import shuffle

# shuffle the files
p = Path('/ds/IssuesLanguageModel/data/1_processed_csv/')
files = p.ls()
shuffle(files)

# show a preview of files
files[:5]

valid_df = pd.concat([pd.read_csv(f) for f in files[:10]]).dropna().drop_duplicates()
train_df = pd.concat([pd.read_csv(f) for f in files[10:]]).dropna().drop_duplicates()

print(f'rows in train_df:, {train_df.shape[0]:,}')
print(f'rows in valid_df:, {valid_df.shape[0]:,}')

valid_df.to_hdf('/ds/IssuesLanguageModel/data/2_partitioned_df/valid_df.hdf')
train_df.to_hdf('/ds/IssuesLanguageModel/data/2_partitioned_df/train_df.hdf')


I am working on the file "Issue_Embeddings/notebooks/01_AcquireData.ipynb".
The first task is:
Download And Pre-Process Data
We download the data from GCP and pre-process this data before saving to disk.
Can you write Python code for it?

from fastai.text.transform import ProcessPoolExecutor, partition_by_cores
import numpy as np
from fastai.core import parallel
from itertools import chain

transforms = transform_pre_rules + defaults.text_pre_rules

def process_dict(dfdict, _):
    """process the data, but allow failure."""
    t = compose(transforms)
    title = dfdict['title']
    body = dfdict['body']
    try:
        text = 'xxxfldtitle '+ t(title) + ' xxxfldbody ' + t(body)
    except:
        return None
    return {'url': dfdict['url'], 'text':text}

def download_data(i, _):
    """Since the data is in 100 chunks already, just do the processing by chunk."""
    fn = f'https://storage.googleapis.com/issue_label_bot/language_model_data/{str(i).zfill(12)}.csv.gz'
    dicts = [process_dict(d, 0) for d in pd.read_csv(fn).to_dict(orient='rows')]
    df = pd.DataFrame([d for d in dicts if d])
    df.to_csv(f'/ds/IssuesLanguageModel/data/1_processed_csv/processed_part{str(i).zfill(4)}.csv', index=False)
    return df


I am working on the file "Issue_Embeddings/notebooks/02_fastai_DataBunch.ipynb".
The first task is:
Read in Data
You can download the above saved dataframes (in pickle format) from Google Cloud Storage:
train_df.hdf (9GB):
https://storage.googleapis.com/issue_label_bot/pre_processed_data/2_partitioned_df/train_df.hdf
valid_df.hdf (1GB)
https://storage.googleapis.com/issue_label_bot/pre_processed_data/2_partitioned_df/valid_df.hdf
Can you write Python code for it?

# note: download the data and place in right directory before running this code!

valid_df = pd.read_hdf(Path('../data/2_partitioned_df/valid_df.hdf'))
train_df = pd.read_hdf(Path('../data/2_partitioned_df/train_df.hdf'))

print(f'rows in train_df:, {train_df.shape[0]:,}')
print(f'rows in valid_df:, {valid_df.shape[0]:,}')

train_df.head(3)


I am working on the file "Issue_Embeddings/notebooks/02_fastai_DataBunch.ipynb".
The first task is:
Specify path for saving language model artifacts
Can you write Python code for it?


I am working on the file "Issue_Embeddings/notebooks/03_Create_Model.ipynb".
The first task is:
1. Load DataBunch
You have to redefine or import any custom you defined in the last step, because the data bunch is going to look for that.
Can you write Python code for it?

def pass_through(x):
    return x

data_lm = load_data(path, bs=120)


I am working on the file "Issue_Embeddings/notebooks/03_Create_Model.ipynb".
The first task is:
2. Instantiate Language Model
We are going to use the awd_lstm with the default parameters listed in the config file below:
Can you write Python code for it?

learn = language_model_learner(data=data_lm,
                               arch=AWD_LSTM,
                               pretrained=False)


I am working on the file "Issue_Embeddings/notebooks/03_Create_Model.ipynb".
The first task is:
Define callbacks
Can you write Python code for it?

escb = EarlyStoppingCallback(learn=learn, patience=5)
smcb = SaveModelCallback(learn=learn)
rpcb = ReduceLROnPlateauCallback(learn=learn, patience=3)
sgcb = ShowGraph(learn=learn)
csvcb = CSVLogger(learn=learn)

callbacks = [escb, smcb, rpcb, sgcb, csvcb]


I am working on the file "Issue_Embeddings/notebooks/04_Inference.ipynb".
The first task is:
Create an InferenceWrapper object
Can you write Python code for it?

wrapper = InferenceWrapper(model_path='/ds/lang_model/models_22zkdqlr/',
                           )

issue_string = '# hello abacadabra world \nA second line **something bold**.'

pooledfeat = wrapper.get_pooled_features(issue_string)
print(pooledfeat)
print(pooledfeat.shape)

rawfeat = wrapper.get_raw_features(issue_string)
print(rawfeat)
print(rawfeat.shape)


I am working on the file "Issue_Embeddings/notebooks/04_Inference.ipynb".
The first task is:
Export Minimal Model State For Inference
Can you write Python code for it?


I am working on the file "Issue_Embeddings/notebooks/04_Inference.ipynb".
The first task is:
Load learner object
Note: you don't have to do this over and over again, you just have to call learn.export() to save the learner after you have loaded everything.
Can you write Python code for it?

data_lm = load_data(data_path, bs=96)

learn = language_model_learner(data=data_lm,
                               arch=AWD_LSTM,
                               model_dir=model_path,
                               pretrained=False)


I am working on the file "Issue_Embeddings/notebooks/04_Inference.ipynb".
The first task is:
Part II:  Load Minimal Model For Inference
Can you write Python code for it?

from inference import InferenceWrapper, pass_through


I am working on the file "Issue_Embeddings/notebooks/04_Inference.ipynb".
The first task is:
Predict the next  5 words
We don't actually use this functionality, but it is interesting to see for those who are curious what the output of a langauge model is.  Recall that we are using the encoder of the language model to extract features from GitHub issues.
Can you write Python code for it?

wrapper.learn.predict('I am having trouble opening a', 5)


I am working on the file "Issue_Embeddings/notebooks/04b_Inference-Batch.ipynb".
The first task is:
Create an InferenceWrapper object
Can you write Python code for it?

wrapper = InferenceWrapper(model_path='/ds/Issue-Embeddings/notebooks',
                           model_file_name='trained_model_22zkdqlr.hdf')


I am working on the file "Issue_Embeddings/notebooks/04b_Inference-Batch.ipynb".
The first task is:
Download a test dataset
The test dataset has 2,000 GitHub Issues in the below format:
Can you write Python code for it?

testdf = pd.read_csv(f'https://storage.googleapis.com/issue_label_bot/language_model_data/000000000000.csv.gz').head(8000)

testdf.head(3)


I am working on the file "Issue_Embeddings/notebooks/04b_Inference-Batch.ipynb".
The first task is:
Notes:
There is a 10x speedup for inference by chunking the data into batches of similar length (to minimize padding) and passing that through the GPU.
In order to get a further speed improvement we must utilize pad_packed_sequence.  We leave this a future exercise to optimize batching more.
Test
This section tests that the embeddings retrieved from the one-at-a time approach are sufficently close to the embeddings from the batching approach
Can you write Python code for it?

assert np.allclose(emb_single_combined, embeddings, atol=1e-5)


I am working on the file "Issue_Embeddings/notebooks/04b_Inference-Batch.ipynb".
The first task is:
Perform Batch Inference
Why Batch-Inference?  When there are a large number of issues for which you want to retrieve document embedddings, batch inference on a gpu (should be) significantly faster than on a cpu.
Generate Embeddings From Pre-Trained Language Model
See help for wrapper.df_to_emb:
Can you write Python code for it?


I am working on the file "Issue_Embeddings/notebooks/05_EvaluateEmbeddings.ipynb".
The first task is:
Compare Transfer Learning vs. Regular Methods
Can you write Python code for it?

import datarobot as dr
from datarobot import UserCV
dr.Client(token='something-something', endpoint='https://app.datarobot.com/api/v2')

def get_metrics(modelobj):
    return modelobj.metrics['AUC']['holdout']

projects = [p for p in dr.Project.list() if p.project_name.startswith('lm_')]

'hamel'.replace('am', 'gg')

label = []
category = []
auc = []

for proj in projects:
    print(f'getting metrics for {proj.project_name}')
    models = [m for m in proj.get_models() if m.sample_pct > 45]
    baseline_model = sorted([m for m in models if m.featurelist_name == 'text only'], key=get_metrics, reverse=True)[0]
    deep_model = sorted([m for m in models if m.featurelist_name != 'text only'], key=get_metrics, reverse=True)[0]
    baseline_auc = get_metrics(baseline_model)
    deep_auc = get_metrics(deep_model)
    label.extend([proj.project_name.replace('lm_', '')] * 2)
    category.extend(['baseline', 'deep'])
    auc.extend([baseline_auc, deep_auc])

import pandas as pd
compare_df = pd.DataFrame({'label': label,
                          'category': category,
                          'auc': auc})

pivot = compare_df.pivot(index='label', columns='category', values='auc')
pivot['winner'] = pivot.apply(lambda x: 'deep' if x.deep > x.baseline else 'baseline', axis=1)
pivot['abs diff'] = pivot.apply(lambda x: abs(x.deep - x.baseline), axis=1)
pivot['label count'] = [c[x] for x in pivot.index.values]
pivot.sort_values(by=['label count'], ascending=False)

wrapper

len(wrapper.learn.data.vocab.itos)

pivot.to_hdf('pivot_df.hdf')

import pandas as pd

score_df = pd.read_hdf('score_df.hdf')

score_df.set_index('label', inplace=True)

score_df.columns = ['deep2']

new_pivot = pivot.join(score_df, how='left')[['baseline', 'deep', 'deep2', 'label count']]

def winner(x):
    if x.baseline > x.deep and x.baseline > x.deep2:
        return 'baseline'
    elif x.deep > x.deep2:
        return 'deep'
    elif x.deep2 > x.deep:
        return 'deep2'

new_pivot.dropna(inplace=True)

new_pivot['winner'] = new_pivot.apply(lambda...


I am working on the file "Issue_Embeddings/notebooks/05_EvaluateEmbeddings.ipynb".
The first task is:
Count Labels
Can you write Python code for it?

from collections import Counter
c = Counter()

for row in df.labels:
    c.update(row)

print(f'There are {len(c.keys())} unique labels in kubernetes/kubernetes')
nsig = sum(['sig/' in x for x in list(c.keys())])
print(f"number of sig labels: {nsig}")


I am working on the file "Issue_Embeddings/notebooks/05_EvaluateEmbeddings.ipynb".
The first task is:
Explore The Data
Question from @cblecker
@Hamel Husain that's how often a PR/issue has two different sig labels on it?
Can you write Python code for it?

def count_sig(l):
    return(sum(['sig/' in x for x in l]))

from matplotlib.ticker import PercentFormatter
sig_counts = df.labels.apply(lambda x: count_sig(x))

plt.hist(sig_counts, weights=np.ones(len(sig_counts)) / len(sig_counts))
plt.gca().yaxis.set_major_formatter(PercentFormatter(1))
plt.title(f'Distribution of # of sig/ labels for kubernetes/kubernetes\n {len(sig_counts):,} issues pulled from GHArchive.')
plt.show()


I am working on the file "Issue_Embeddings/notebooks/05_EvaluateEmbeddings.ipynb".
The first task is:
Get Embeddings and Repeat
Can you write Python code for it?

import pandas as pd
from inference import InferenceWrapper, pass_through
import os
import torch
from torch.cuda import empty_cache
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"]="0"

wrapper = InferenceWrapper(model_path='/ds/lang_model/models_uxgcl1e1/',
                           model_file_name='trained_model_uxgcl1e1.hdf')

empty_cache()

combined_sig_df = pd.read_hdf('combined_sig_df.hdf')
# text = wrapper.process_df(combined_sig_df)
# text.to_hdf('textlm_df.hdf')

text = pd.read_hdf('textlm_df.hdf')
assert text['text'].isna().sum() == 0

features = []

from tqdm.auto import tqdm

with torch.no_grad():
    for t in tqdm(text['text'].values):
        feat = wrapper.get_pooled_features(t).cpu()
        features.append(feat)
        empty_cache()

feat_matrix = torch.cat(features, dim=0).numpy()

feat_matrix = feat_matrix[:, :1600]

feat_df = pd.DataFrame(feat_matrix)
feat_df.columns = ['f_' + str(x) for x in feat_df.columns]
feat_df.to_csv('feat_df.csv', index=False)

feat_df = pd.read_csv('feat_df.csv')

lm_combined_df = pd.concat([combined_sig_df.reset_index(drop=True),
                            feat_df.reset_index(drop=True)], axis=1)

import datarobot as dr
from datarobot import UserCV
ucv = UserCV(user_partition_col='part', cv_holdout_level=6, seed=123)
dr.Client(token='something', endpoint='https://app.datarobot.com/api/v2')

def create_dr_proj(label):
    temp_df = lm_combined_df[['title', 'body', 'part', label] + list(feat_df.columns)]
    proj = dr.Project.create(sourcedata=temp_df,
                             project_name='lm_'+label,
                            )
    proj.set_target(label,
                    positive_class=1,
                    partitioning_method=ucv,
                    target_type='Binary',
                    mode=dr.AUTOPILOT_MODE.QUICK,
                    worker_count=9,
                    max_wait=600000)

    proj.unlock_holdout()

    return proj

proj_list_lm = []

for i, label in enumerate(sig_labels):
    try:
       ...


I am working on the file "Issue_Embeddings/notebooks/05_EvaluateEmbeddings.ipynb".
The first task is:
Obtain Baseline With Automated Machine Learning
Can you write Python code for it?

def part_assign():
    i = randint(1, 10)
    if i <=5:
        return i
    else:
        return 6

combined_sig_df = pd.concat([sig_df.reset_index(), indicator_matrix.reset_index()], axis=1)
combined_sig_df['part'] = combined_sig_df.repo.apply(lambda x: part_assign())
combined_sig_df.to_hdf('combined_sig_df.hdf')

combined_sig_df = pd.read_hdf('combined_sig_df.hdf')

#! pip install datarobot

import datarobot as dr
from datarobot import UserCV
from fastai.core import parallel
from datarobot import Blueprint

ucv = UserCV(user_partition_col='part', cv_holdout_level=6, seed=123)
dr.Client(token='something-something', endpoint='https://app.datarobot.com/api/v2')

def create_dr_proj(label):
    temp_df = combined_sig_df[['title', 'body', 'part', label]]
    proj = dr.Project.create(sourcedata=temp_df,
                             project_name=label,
                            )
    proj.set_target(label,
                positive_class=1,
                partitioning_method=ucv,
                target_type='Binary',
                mode=dr.AUTOPILOT_MODE.MANUAL,
                worker_count=9,
                max_wait=600000)

    bps = proj.get_blueprints()
    bp = [b for b in bps if 'Nystroem' in str(b)][0]
    proj.train(bp, sample_pct=49.8)
    proj.unlock_holdout()

    return proj

proj_list = []

for i, label in enumerate(sig_labels):
    try:
        print(f'creating project {i}: {label}')
        proj = create_dr_proj(label)
        proj_list.append(proj)
    except:
        pass

predictions = []
for proj in proj_list:
    print(f'getting predictions for holdout set for {str(proj)}')
    label = proj.target.replace('_', '-')
    temp_df = combined_sig_df[['title', 'body', 'part', label]]
    temp_df = temp_df[temp_df.part == 6]
    ds = proj.upload_dataset(temp_df)
    m = proj.get_models()[0]
    predict_job = m.request_predictions(ds.id)
    yhat = predict_job.get_result_when_complete()
    predictions.append({label: yhat['positive_probability']})

result = {}
for d in predictions:
   ...


I am working on the file "Issue_Embeddings/notebooks/05_EvaluateEmbeddings.ipynb".
The first task is:
See correlation among labels
Can you write Python code for it?

min_freq = 30

def contains_sig(l):
    if not l:
        return False
    else:
        # make sure there are at least 10 issues labeled with that value
        return max(['sig/' in x and c[x] >=min_freq for x in l])

sig_df = df[df.labels.apply(lambda x: contains_sig(x))]
print(f'{sig_df.shape[0]:,} issues have sig/ labels')

sig_labels = [k for k in c.keys() if c[k] >= min_freq and 'sig/' in k]
print(f'{len(sig_labels)} sig labels that have at least {min_freq} issues')

# build an indicator matrix

indicator = []

for l in sig_df.labels.values:
    zer = np.zeros(len(sig_labels))
    mask = [sig_labels.index(x) for x in l if x in sig_labels]
    zer[mask] = 1
    indicator.append(zer[None, :])

indicator_matrix = pd.DataFrame(np.concatenate(indicator, axis=0), columns=sig_labels).astype(int)
corr_grid = indicator_matrix.T.dot(indicator_matrix)

for i, x in enumerate(corr_grid):
    corr_grid.iloc[i][i:] = 0

import seaborn as sns
import matplotlib.pyplot as plt
#cmap = sns.diverging_palette(220, 10, as_cmap=True)

#normalize correlation grid
for label in corr_grid:
    corr_grid.loc[label] = corr_grid.loc[label] / c[label]

plt.figure(figsize=(16, 14))
plt.title('Co-Occurence Matrix')
sns.heatmap(corr_grid, square=True, vmin=0, vmax=.4, mask=corr_grid<=0.05)


I am working on the file "Issue_Embeddings/notebooks/05_EvaluateEmbeddings.ipynb".
The first task is:
Sig/ Labels
Can you write Python code for it?

sig_labels = [x for x in list(c.keys()) if 'sig/' in x]
for l in sig_labels:
    print(f'{l}: {c[l]}')


I am working on the file "Issue_Embeddings/notebooks/05_EvaluateEmbeddings.ipynb".
The first task is:
Top 50 Labels
Can you write Python code for it?

c.most_common(50)

len([(k, c[k]) for k in c if c[k] >= 100])


I am working on the file "Issue_Embeddings/notebooks/06_FineTune.ipynb".
The first task is:
Explore The Data
Count Labels
Filter Issues Again To Remove Those w/o Labels occuring at least 50 times
Can you write Python code for it?

from collections import Counter
c = Counter()

for row in df.labels:
    c.update(row)

min_threshold = 50
min_threshold_labels = [k for k in c if c[k] >= min_threshold]

print(f'{len(min_threshold_labels)} labels that occur at least {min_threshold} times.')

df['labels'] = df.labels.apply(lambda x: [l for l in x if l in min_threshold_labels])
df = df[df.labels.apply(lambda x: x != [])]
print(f'Number of labeled issues after filtering again: {df.shape[0]:,}')


I am working on the file "Issue_Embeddings/notebooks/06_FineTune.ipynb".
The first task is:
Filter Labels By YAML File
Can you write Python code for it?

import base64
import requests
import yaml

def get_current_labels(url="https://raw.githubusercontent.com/kubernetes/test-infra/master/label_sync/labels.yaml"):
    """
    Get list of valid issue labels (b/c labels get deprecated over time).
    See: https://kubernetes.slack.com/archives/C1TU9EB9S/p1561570627363100
    """
    req = requests.get(url)
    yml = yaml.safe_load(req.content)
    return [x['name'] for x in yml['default']['labels']]

current_labels = get_current_labels()
# remove deprecated labels
df.labels = df.labels.apply(lambda x: [l for l in x if l in current_labels])
# filter out issues without any labels
df = df[df.labels.apply(lambda x: x != [])]
print(f'Number of labeled issues after filtering: {df.shape[0]:,}')

df.head(1)


I am working on the file "Issue_Embeddings/notebooks/06_FineTune.ipynb".
The first task is:
Join the labels back onto the parsed text
Can you write Python code for it?

assert parsed_df.shape[0] == df.shape[0]
ml_df = pd.concat([df.reset_index(drop=True), parsed_df], axis=1)[['text', 'labels']]
# must delimit the labels by something (here using space) for fastai
ml_df['labels'] = ml_df.labels.apply(lambda x: ' '.join(x))
assert len(ml_df) == len(parsed_df) == len(df)

ml_df.head(2)

ml_df.to_hdf('ml_df.hdf')


I am working on the file "Issue_Embeddings/notebooks/06_FineTune.ipynb".
The first task is:
Manual Learning Rate Annealing
Can you write Python code for it?


I am working on the file "Issue_Embeddings/notebooks/06_FineTune.ipynb".
The first task is:
Number of Issues By Time (after filtering)
(Issues were filtered out in previous years due to having deprecated issue labels)
Can you write Python code for it?

df['year'] = df.last_time.apply(lambda x: x.year)
df.groupby('year')['body'].count()


I am working on the file "Issue_Embeddings/notebooks/06_FineTune.ipynb".
The first task is:
Remaining Issues
Can you write Python code for it?

for l in min_threshold_labels:
    print(f'{l}: {c[l]}')


I am working on the file "Issue_Embeddings/notebooks/07_Get_Repo_TrainingData_BigQuery.ipynb".
The first task is:
Clean Data

Remove labels that are deprecated and no longer specified here.
Remove bot commands that contain the label name (otherwhise this will be data leakage).
Can you write Python code for it?

def get_current_labels(url="https://raw.githubusercontent.com/kubernetes/test-infra/master/label_sync/labels.yaml"):
    """
    Get list of valid issue labels (b/c labels get deprecated over time).
    See: https://kubernetes.slack.com/archives/C1TU9EB9S/p1561570627363100
    """
    req = requests.get(url)
    yml = yaml.safe_load(req.content)
    return [x['name'] for x in yml['default']['labels']]

# remove deprecated labels
current_labels = get_current_labels()
df.labels = df.labels.apply(lambda x: [l for l in x if l in current_labels])

# get rid of bot commands
df['body'] = df.body.apply(lambda x: re.sub('(/sig|/kind|/status/triage/|priority) \S+', '', str(x)))


I am working on the file "Issue_Embeddings/notebooks/07_Get_Repo_TrainingData_BigQuery.ipynb".
The first task is:
Filter Issues
Remove Issues that:
- Do not have any labels
- Do not contain at least one label that has a minimum frequency of 25
Can you write Python code for it?

# remove labels that do not occur at least 25 times
c = Counter()

for row in df.labels:
    c.update(row)

min_threshold = 25
min_threshold_labels = [k for k in c if c[k] >= min_threshold]

df['labels'] = df.labels.apply(lambda x: [l for l in x if l in min_threshold_labels])

# filter out issues without any labels
df = df[df.labels.apply(lambda x: x != [])]

# remove extreanous columns
df = df[['last_time', 'repo', 'title', 'body', 'labels']]

print(f'Number of labeled issues after filtering and cleaning: {df.shape[0]:,}')

df.head()


I am working on the file "Issue_Embeddings/notebooks/08_Train_Repo_Specific_IssueLabeler.ipynb".
The first task is:
Do pre-processing (such as markdown parsing) to prepare data for model.
Can you write Python code for it?

parsed_df = iw.process_df(combined_sig_df)

holdout_text = parsed_df[holdout_mask]

lang_model_predict = np.stack(holdout_text.text.apply(lambda x: tcl.predict(x)[2].numpy()[pred_mask]).values)

lang_model_predict.shape

len(iw.learn.data.classes)

np.array(iw.learn.data.classes)[pred_mask]

lang_model_predict_df = pd.DataFrame(lang_model_predict)
lang_model_predict_df.columns = np.array(iw.learn.data.classes)[pred_mask]
lm_df = lang_model_predict_df[[x for x in label_columns if x in lang_model_predict_df.columns]]


I am working on the file "Issue_Embeddings/notebooks/08_Train_Repo_Specific_IssueLabeler.ipynb".
The first task is:
Fit the model
Can you write Python code for it?

model_pipeline.fit(x_train_df, y_train)

best_model = model_pipeline.named_steps['gridsearchcv'].best_estimator_
print(f'The best model from grid search is:\n=====================================\n{best_model}')


I am working on the file "Issue_Embeddings/notebooks/08_Train_Repo_Specific_IssueLabeler.ipynb".
The first task is:
Part 2: Fine Tune Original Language Model
With FastAI & Pytorch
Note: this model was completely trained in another notebook, it is only evaluated here.
Prepare Inference Wrapper
Model is available for download here:
https://storage.googleapis.com/issue_label_bot/model/multi_class_model/export.hdf
Can you write Python code for it?

import os
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"]="2"

from inference import InferenceWrapper, pass_through
iw = InferenceWrapper(model_path='/ds/multi_class_model/',
                      model_file_name='/ds/multi_class_model/export.hdf')

#because the pre-trained mdoel predicts more than just sig/ labels
pred_mask = [x in label_columns for x in iw.learn.data.classes]


I am working on the file "Issue_Embeddings/notebooks/08_Train_Repo_Specific_IssueLabeler.ipynb".
The first task is:
Part 3:  Using Text & Bag of Words Instead of Pre-Trained Embeddings (Classic ML)
Count Vectorizer w/ Nueral Net
Choose Nueral Net because supports multi-label classification natively
Can you write Python code for it?

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.pipeline import make_pipeline, FeatureUnion
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.neural_network import MLPClassifier
from sklearn.model_selection import GridSearchCV

train_mask = combined_sig_df.part != 6

parameters = {'hidden_layer_sizes': [(50,), (100,), (200,), (400, ), (50, 50), (100, 100)],
              'alpha': [0.0001, .001, .01, .1, 1, 10],
              'learning_rate': ['constant', 'adaptive'],
              'learning_rate_init': [.001, .01]}

mlp = MLPClassifier(early_stopping=True, validation_fraction=.2, n_iter_no_change=4, max_iter=500)

class ColumnSelector(BaseEstimator, TransformerMixin):
    def __init__(self, column):
        self.column = column

    def fit(self, X, y=None):
        return self

    def transform(self, X):
        assert isinstance(X, pd.DataFrame)

        try:
            return X[self.column].values
        except KeyError:
            col_error = list(set(self.column) - set(X.column))
            raise KeyError("The DataFrame does not include the columns: %s" % col_error)

model_pipeline = make_pipeline(
    FeatureUnion(transformer_list=[
        ("title", make_pipeline(
            ColumnSelector("title"),
            CountVectorizer(ngram_range=(1,3), min_df=3, strip_accents='unicode'),
        )),
        ("body", make_pipeline(
            ColumnSelector("body"),
            CountVectorizer(ngram_range=(1,3), min_df=3, strip_accents='unicode'),
        ))
    ]),
    GridSearchCV(mlp, parameters, cv=5, n_jobs=-1)
)

y_train = combined_sig_df[label_columns][train_mask].values
x_train_df = combined_sig_df[['body', 'title']][train_mask]


I am working on the file "Issue_Embeddings/notebooks/08_Train_Repo_Specific_IssueLabeler.ipynb".
The first task is:
Sklearn KNN
Can you write Python code for it?

from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(weights='distance', n_neighbors=10, n_jobs=-1)

knn.fit(X, y)

knn_preds = knn.predict_proba(X_holdout)

import numpy as np
knn_preds_stacked = np.stack([x[:, 1] for x in knn_preds], axis=0).T

knn_preds_stacked

knn_df, knn_auc = calculate_auc(knn_preds_stacked)


I am working on the file "Issue_Embeddings/notebooks/08_Train_Repo_Specific_IssueLabeler.ipynb".
The first task is:
Sklearn Random Forest
Can you write Python code for it?

from sklearn.ensemble import RandomForestClassifier
from sklearn.multiclass import OneVsRestClassifier
from sklearn.svm import NuSVC

rf = RandomForestClassifier(n_estimators=300,
                             random_state=1234, min_samples_leaf=3, class_weight='balanced')

clf = OneVsRestClassifier(rf, n_jobs=-1)

clf.fit(X, y)

rf_predictions = clf.predict_proba(X_holdout)

rf_df, rf_auc = calculate_auc(rf_predictions)


I am working on the file "Issue_Embeddings/notebooks/08_Train_Repo_Specific_IssueLabeler.ipynb".
The first task is:
There are some columns that the model was not trained on
Can you write Python code for it?

missing_cols = [x for x in label_columns if x not in lm_df.columns]

for col in missing_cols:
    lm_df[col] = 0.0

lm_df, lm_auc = calculate_auc(lm_df.values)

missing_cols


I am working on the file "Issue_Embeddings/notebooks/08_Train_Repo_Specific_IssueLabeler.ipynb".
The first task is:
Try Tuning the MLP
Can you write Python code for it?

params = {'hidden_layer_sizes': [(100,), (200,), (400, ), (50, 50), (100, 100), (200, 200)],
              'alpha': [.001, .01, .1, 1, 10],
              'learning_rate': ['constant', 'adaptive'],
              'learning_rate_init': [.001, .01, .1]}

mlp_clf = MLPClassifier(early_stopping=True, validation_fraction=.2, n_iter_no_change=4, max_iter=500)

gscvmlp = GridSearchCV(mlp_clf, params, cv=5, n_jobs=-1)

gscvmlp.fit(X, y)

print(f'The best model from grid search is:\n=====================================\n{gscvmlp.best_estimator_}')

mlp_tuned_predictions = gscvmlp.predict_proba(X_holdout)

mlp_tuned_df, mlp_tuned_auc = calculate_auc(mlp_tuned_predictions)


I am working on the file "Issue_Embeddings/notebooks/08_Train_Repo_Specific_IssueLabeler.ipynb".
The first task is:
wow that sucks, lets try something else
Sklearn Nueral Network
b/c multi-label is supported naturally
Can you write Python code for it?

from sklearn.neural_network import MLPClassifier
mlp = MLPClassifier(early_stopping=True, n_iter_no_change=5, max_iter=500, solver='adam',
                   random_state=1234)

mlp.fit(X, y)

mlp_predictions = mlp.predict_proba(X_holdout)

mlp_df, mlp_auc = calculate_auc(mlp_predictions)


I am working on the file "Issue_Embeddings/notebooks/09_LangModel_API_Demo.ipynb".
The first task is:
API Endpoints
Route 1: https://embeddings.gh-issue-labeler.com/text
Allows you to get embeddings for the raw text corresponding to a single GitHub issue.  The motivation for this endpoint is to use this at inference time, for example, when you need to perform computation on a new issue.
This endpoint listens to POST requests, with the payload illustrated below:
Can you write Python code for it?

data = {'title': 'Fix the issue',
        'body': 'I am encountering an error\n when trying to push the button.'}

# sending post request and saving response as response object
r = requests.post(url=API_ENDPOINT,
                  headers={'Token':pwd_context.hash(API_KEY)},
                  json=data)


I am working on the file "Issue_Embeddings/notebooks/09_LangModel_API_Demo.ipynb".
The first task is:
Convert byte stream sent over REST back to a numpy array.  The numpy array is a 2,400 dimensional embedding which are latent features of the GitHub Issue.
Can you write Python code for it?

embeddings = np.frombuffer(r.content, dtype='<f4')
embeddings.shape


I am working on the file "Issue_Embeddings/notebooks/Get-GitHub-Issues.ipynb".
The first task is:
Compute Embeddings

For each repo compute the embeddings and save to GCS
TODO(jlewi): Can we use the metadata storage to keep track of artifacts?
Can you write Python code for it?

input_data = latest_issues[["title", "body"]]

issue_embeddings = inference_wrapper.df_to_embedding(input_data)

issue_embeddings.shape


I am working on the file "Issue_Embeddings/notebooks/Get-GitHub-Issues.ipynb".
The first task is:
Load Model Artifacts (Download from GC if not on local)

We need to load the model used to compute embeddings
Can you write Python code for it?

from pathlib import Path
from urllib import request as request_url

def pass_through(x):
    return x

model_url = 'https://storage.googleapis.com/issue_label_bot/model/lang_model/models_22zkdqlr/trained_model_22zkdqlr.hdf'
inference_wrapper = embeddings.load_model_artifact(model_url)


I am working on the file "Issue_Embeddings/notebooks/Get-GitHub-Issues.ipynb".
The first task is:
Pull request comments also get included so we need to filter those out
Can you write Python code for it?

import re
pattern = re.compile(".*issues/[\d]+")
issues_index = issues_and_pulls["html_url"].apply(lambda x: pattern.match(x) is not None)
issues=issues_and_pulls[issues_index]


I am working on the file "Issue_Embeddings/notebooks/Get-GitHub-Issues.ipynb".
The first task is:
Sanity Check the embeddings

We want to make sure the embeddings are computed the same way as during inference time
During inference IssueLabelerPredict.predict_labels_for_issue calls embeddings.get_issue_text to fetch the body and title
We call embeddings.get_issue_text one of the issues to make sure it matches the data in the dataframe from which we compute the embeddings

This calls the /text on the embeddings microservice

TODO(https://github.com/kubeflow/code-intelligence/issues/126) The label bot microservice needs to be updated to actually
  use the GraphQL API to match this code. Hopefully, in the interim the model is robust to slight deviations caused
  by the differences in whitespace
Can you write Python code for it?

from code_intelligence import util as code_intelligence_util

issue_index = 1020
logging.info(f"Fetching issue {latest_issues.iloc[issue_index]['html_url']}")
issue_owner, issue_repo, issue_num = code_intelligence_util.parse_issue_url(latest_issues.iloc[issue_index]["html_url"].strip("\""))

some_issue_data = embeddings.get_issue(latest_issues.iloc[issue_index]["html_url"], gh_client)

some_issue_data

print(latest_issues.iloc[issue_index]["title"])
print(some_issue_data["title"])
print(latest_issues.iloc[issue_index]["body"])
print(some_issue_data["body"])
some_issue_data["title"] == latest_issues.iloc[issue_index]["title"]
some_issue_data["body"] == latest_issues.iloc[issue_index]["body"]


I am working on the file "Issue_Embeddings/notebooks/Get-GitHub-Issues.ipynb".
The first task is:
Save Embeddings to GCS
Can you write Python code for it?

embeddings_file = os.path.join(embeddings_dir, file_name)
if gcs_util.check_gcs_object(embeddings_file):
    logging.info(f"File {embeddings_file} exists")
else:
    logging.info(f"Copying {local_file} to {embeddings_file}")
    gcs_util.copy_to_gcs(local_file, embeddings_file)

embeddings_file


I am working on the file "Issue_Embeddings/notebooks/Get-GitHub-Issues.ipynb".
The first task is:
Save the issues and embeddings to an HDF5 file
Can you write Python code for it?

import h5py
import datetime

now = code_intelligence_util.now().isoformat()

git_tag = subprocess.check_output(["git", "describe", "--tags", "--always", "--dirty"]).decode().strip()
file_name = f"kubeflow_issue_embeddings_{now}.hdf5"
local_file = os.path.join(home, file_name)

latest_issues.to_hdf(local_file, "issues", mode="a")

h5_file = h5py.File(local_file, mode="a")

h5_file.create_dataset("issue_embeddings", data=issue_embeddings)

# store some metadata
h5_file.attrs["file"] = "Get-GitHub-Issues.ipynb"
h5_file.attrs["git-tag"] = git_tag

h5_file.close()


I am working on the file "Issue_Embeddings/notebooks/Get-GitHub-Issues.ipynb".
The first task is:
We need to deserialize the json strings to remove escaping
Can you write Python code for it?

for f in ["html_url", "title", "body"]:
    latest_issues[f] = latest_issues[f].apply(lambda x : json.loads(x))


I am working on the file "Issue_Embeddings/notebooks/Get-GitHub-Issues.ipynb".
The first task is:
We need to group the events by issue and then select the most recent event for each issue as that should have
  the most up to date labels for each issue
TODO(jlewi): We should look for the most recent event in the dataset and then have some alert if the age exceeds some
  limit as that indicates the data isn't up to date.
Can you write Python code for it?

latest_issues = issues.groupby("html_url", as_index=False).apply(lambda x: x.sort_values(["updated_at"]).iloc[-1])

# Example of fetching a specific issue
# This allows easy spot checking of the data
some_issue = "https://github.com/kubeflow/kubeflow/issues/4916"
test_issue = latest_issues.loc[latest_issues["html_url"]==f'"{some_issue}"']
test_issue


I am working on the file "Issue_Embeddings/notebooks/scratch/Kubeflow-FewShotLearning.ipynb".
The first task is:
2. Extract One Hot Encoded Labels
Can you write Python code for it?

from sklearn.preprocessing import MultiLabelBinarizer
mlb = MultiLabelBinarizer()

ohe_labels = mlb.fit_transform(h_labeled_df.final_labels.values.tolist())
display(ohe_labels)
print(ohe_labels.shape)

mlb.classes_


I am working on the file "Issue_Embeddings/notebooks/scratch/Kubeflow-FewShotLearning.ipynb".
The first task is:
Flatten list of labels
Can you write Python code for it?

# unpack the lists of labels and flatten
def unpack_list(x):
    "convert list as string into list."
    if x == '':
        return 'no_labels'
    else:
        return json.loads(x)

#flatten lists
labels = []
label_series = kfdf.labels.apply(lambda x: unpack_list(x))
for x in label_series:
    labels.extend(x)


I am working on the file "Issue_Embeddings/notebooks/scratch/Kubeflow-FewShotLearning.ipynb".
The first task is:
Inference
Can you write Python code for it?

learn.model.reset() # so the hidden states reset between predictions
_ = learn.model.eval() # turn off dropout, etc. only need to do this after loading model.


I am working on the file "Issue_Embeddings/notebooks/scratch/Kubeflow-FewShotLearning.ipynb".
The first task is:
Look at Data For Kubeflow/Kubeflow
Get Kubeflow Data
See the query in GCP BigQuery Console
Can you write Python code for it?

df = pd.read_csv(f'https://storage.googleapis.com/issue_label_bot/kubeflow_issues/000000000000.csv')
# filter for kubeflow/kubeflow
kfdf = df[df.repo.apply(lambda x: x.split('/')[1] =='kubeflow')]


I am working on the file "Issue_Embeddings/notebooks/scratch/Kubeflow-FewShotLearning.ipynb".
The first task is:
Notes

Some labels have a fairly high N.  Do we really need few shot for these?
Do you really want to maintain local models for each repo?  If you do should be a seperate service with API endpoint to keep dependencies clean.
First lets see if few shot can even work?
Looks like we might be able to get pretty far on keyword matching and BPE

Supervised Learning On Kubeflow/* (the whole org)
1. Get Data That has at least one label that occurs > 20 times.
Can you write Python code for it?

from collections import Counter, Set
label_counter = Counter()

df['labels_unpacked'] = df.labels.apply(lambda x: unpack_list(x))

for labels in df.labels_unpacked:
    label_counter.update(labels)

labels_to_keep = {x:label_counter[x] for x in label_counter if label_counter[x] >= 20}


I am working on the file "Issue_Embeddings/notebooks/scratch/Kubeflow-FewShotLearning.ipynb".
The first task is:
Notes
Fastai encoder produces a tuple of two lists raw_output and output.  see this reference  raw_output are the hidden states emitted for each element of the sequence without dropout.  Because you are turning off dropout during inference with .eval(), it really doesn't matter which one you get as they will both be the same (if they are not, this is a bug).
Can you write Python code for it?

ex = processed_issue_texts[0]['text']
print(ex)

ex_numericalized_x,  ex_numericalized_y = learn.data.one_item(ex)
ex_numericalized_x


I am working on the file "Issue_Embeddings/notebooks/scratch/Kubeflow-FewShotLearning.ipynb".
The first task is:
Only retain labels that occur at least 20 times
Can you write Python code for it?

h_labeled_df['final_labels'] = h_labeled_df.labels_unpacked.apply(lambda x: set(x).intersection(label_set))


I am working on the file "Issue_Embeddings/notebooks/scratch/Kubeflow-FewShotLearning.ipynb".
The first task is:
Read in Model
Notes: you can export a lightweight learner for inference per https://docs.fast.ai/tutorial.inference.html
Can you write Python code for it?

pd.read_csv('lang_model_onecycle_resume/history.csv')

pd.read_csv('lang_model_onecycle/history.csv')

from fastai.text.models import AWD_LSTM
from fastai.text import TextLMDataBunch as lmdb, load_data
from fastai.text.learner import language_model_learner
from fastai.basic_train import load_learner
path = Path('lang_model_onecycle/')

def pass_through(x):
    return x


I am working on the file "Issue_Embeddings/notebooks/scratch/Kubeflow-FewShotLearning.ipynb".
The first task is:
See the labels that occur > 20 times.
Can you write Python code for it?

display(labels_to_keep)
print(f' Number of labels: {len(labels_to_keep)}')

label_set = set(labels_to_keep.keys())

h_labeled_df = df[df.labels_unpacked.apply(lambda x: len(set(x).intersection(label_set)) > 0)]
h_labeled_df.shape


I am working on the file "Issue_Embeddings/notebooks/scratch/Kubeflow-FewShotLearning.ipynb".
The first task is:
The next two output tensors should be the same, this is testing that the model state is being reset correctly between predictions
Can you write Python code for it?

encoder = learn.model[0]
rep = encoder.forward(ex_numericalized_x)[-1][-1]
print(rep)
print(rep.shape)

learn.model.reset()
rep = encoder.forward(ex_numericalized_x)[-1][-1]
print(rep)
print(rep.shape)


I am working on the file "Issue_Triage/notebooks/metrics.ipynb".
The first task is:
Number of Issues Labeled Per Day

Make a plot of the number of issues labeled each day
Can you write Python code for it?

import numpy as np
issues_per_day = labeled[["timestamp","repo_owner", "repo_name", "issue_num"]]
# Deduplicate the issues by taking the first entry
issues_per_day =  issues_per_day.groupby(["repo_owner", "repo_name", "issue_num"], as_index=False).min()
# Compute the day
issues_per_day["day"] = issues_per_day["timestamp"].apply(lambda x: datetime.datetime(x.year, x.month, x.day))
issue_counts = issues_per_day[["day"]]
issue_counts["num_issues"] = 1
issue_counts = issue_counts.groupby(["day"], as_index=False).sum()

chart = alt.Chart(issue_counts)
line = chart.mark_line().encode(
  x=alt.X('day'),
  y=alt.Y('num_issues'),
)

point = line + line.mark_point()

point.interactive()


I am working on the file "Issue_Triage/notebooks/metrics.ipynb".
The first task is:
Setup Authorization
If you are using a service account run
%%bash
Activate Service Account provided by Kubeflow.
gcloud auth activate-service-account --key-file=${GOOGLE_APPLICATION_CREDENTIALS}
If you are running using user credentials
gcloud auth application-default login
Query Bigquery

We need to query bigquery to get the issues were we added predictions
Can you write Python code for it?

query = """
SELECT
    timestamp,
    jsonPayload.repo_owner,
    jsonPayload.repo_name,
    cast(jsonPayload.issue_num as numeric) as issue_num,
    jsonPayload.predictions
  FROM `issue-label-bot-dev.issue_label_bot_logs_dev.stderr_*`
  where jsonPayload.message = "Add labels to issue."
        and timestamp_diff(current_timestamp(), timestamp, day) <=28
"""

labeled=gbq.read_gbq(str(query), dialect='standard', project_id=PROJECT)

# Count how many times each label was added
label_counts = collections.defaultdict(lambda: 0)

# We need to compute the number of issues that got labeled with an area or kind label
results = pd.DataFrame(index=range(labeled.shape[0]), columns=["area", "kind"])
results = results.fillna(0)

for i in range(labeled.shape[0]):
    predictions = labeled["predictions"][i]

    if not predictions:
        continue

    # Loop over the predictions to see if one of them includes an area or kind label
    for l, p in predictions.items():
        label_counts[l] = label_counts[l] + 1

# Now for each issue count whether a particular label is added
issue_labels = pd.DataFrame(index=range(labeled.shape[0]), columns=label_counts.keys())
issue_labels = issue_labels.fillna(0)

for c in ["repo_owner", "repo_name", "issue_num"]:
    issue_labels[c] = labeled[c]

for i in range(labeled.shape[0]):
    predictions = labeled["predictions"][i]

    if not predictions:
        continue

    for l, p in predictions.items():
        if not p:
            continue

        issue_labels.at[i, l] = 1

# Deduplicate the rows
# We need to group by (repo_owner, repo_name, issue_num); we should take the max of each column
# as a way of dealing with duplicates
issue_labels = issue_labels.groupby(["repo_owner", "repo_name", "issue_num"], as_index=False).max()

# Create a mapping from label prefixes to all all the labels with that prefix
# e.g. area -> ["area_jupyter", "area_kfctl", ...]

label_prefixes = collections.defaultdict(lambda: [])

for l in...


I am working on the file "Issue_Triage/notebooks/triage.ipynb".
The first task is:
Compute a series containing the number of untriaged issues as a function of time
Can you write Python code for it?

opened = pd.Series([1]*issues_df.shape[0], index=issues_df["created_at"])
triaged_issues = issues_df.iloc[(issues_df["needs_triage"] == False).values]
triaged = pd.Series([-1]*triaged_issues.shape[0], index=triaged_issues["triaged_at"])

deltas = pd.concat([opened, triaged])
deltas = deltas.sort_index()
untriaged = deltas.cumsum()


I am working on the file "Issue_Triage/notebooks/triage.ipynb".
The first task is:
Triage a Single issue

This cell can be used to triage a single issue
Its useful if you want to find out specific reasons why an issue isn't considered triaged
Can you write Python code for it?

importlib.reload(triage)
triager = triage.IssueTriage()
url = "https://github.com/kubeflow/kubeflow/issues/1583"
issue_info = triager.triage_issue(url)


I am working on the file "Issues/algorithms/Arrays and Strings.ipynb".
The first task is:
Check Permutation
Check Permutation: Given two strings, write a method to decide if one is a permutation of the other.
Can you write Python code for it?

def str_shuffle(str):
    str_list = list(str)
    random.shuffle(str_list)
    return "".join(str_list)

str0 = gen_randstr()
str1 = gen_randstr()

str2 = gen_randstr()
str3 = str_shuffle(str2[:])
print(str2)
print(str3)

def check_permutation(str0, str1):
    str0_ = "".join(sorted(str0))
    str1_ = "".join(sorted(str1))
    print(str0_)
    print(str1_)
    if len(str0_) != len(str1_):
        return False

    for i in range(len(str0_)):
        if str0_[i] != str1_[i]:
            print(i)
            return False

    return True

print("Test#1: %s"%("Pass" if False == check_permutation(str0, str1) else "Fail"))
print("Test#2: %s"%("Pass" if True == check_permutation(str2, str3) else 'Fail'))


I am working on the file "Issues/algorithms/Arrays and Strings.ipynb".
The first task is:
One Away:
There are three types of edits that can be performed on strings: insert a character,remove a character, or replace a character. Given two strings, write a function to check if they are one edit (or zero edits) away.
Can you write Python code for it?

def check_same(str0, str1):
    if len(str0) != len(str1):
        return False
    for i in range(len(str0)):
        if str0[i] != str1[i]:
            return False
    return True

def check_oneaway(str0, str1):

    for i in range(len(str0)):
        if (i < len(str1) and str0[i] != str1[i]) or i > (len(str1) - 1):

            if check_same(str0[i + 1:], str1[i:]):
                # remove
                return True
            if check_same(str0[i:],str1[i + 1:]):
                # insert
                return True
            if check_same(str0[i+1:], str1[i+1:]):
                # replace
                return True
            return False
    return False

print("Test#1: %s"%("Pass" if True == check_oneaway("pale", "ple") else "Fail"))
print("Test#2: %s"%("Pass" if True == check_oneaway("pales", "pale") else "Fail"))
print("Test#3: %s"%("Pass" if True == check_oneaway("pale", "bale") else "Fail"))
print("Test#4: %s"%("Pass" if False == check_oneaway("pale", "bake") else "Fail"))
print("Test#4: %s"%("Pass" if False == check_oneaway("AAAAABBBBBBBBB", "AAAAABBBB") else "Fail"))


I am working on the file "Issues/algorithms/Arrays and Strings.ipynb".
The first task is:
Palindrome Permutation
Given a string, write a function to check if it is a permutation of a palindrome. A palindrome is a word or phrase that is the same forwards and backwards. A permutation is a rearrangement of letters. The palindrome does not need to be limited to just dictionary words.
Can you write Python code for it?

def check_palindrome_permutation(str0):

    str0 = str0.lower()
    histogram = {}
    for ch in str0:
        if ch != ' ':
            histogram[ch] = histogram.get(ch, 0) + 1

    # check one odd entries
    found_odd = False
    for ch, value in histogram.items():
        if value%2 == 1:
            if found_odd:
                return False
            found_odd = True

    return True

print("Test#1: %s"%("Pass" if True == check_palindrome_permutation('Tact Coa') else "Fail"))
print("Test#2: %s"%("Pass" if False == check_palindrome_permutation('AAABBBCCC') else "Fail"))


I am working on the file "Issues/algorithms/Arrays and Strings.ipynb".
The first task is:
Palindrome Permutation:
Given a string, write a function to check if it is a permutation of a palindrome. A palindrome is a word or phrase that is the same forwards and backwards. A permutation is a rearrangement of letters. The palindrome does not need to be limited to just dictionary words.
Can you write Python code for it?

def check_palindrome(str0):
    size = len(str0)
    for i in range(size):
        if str0[i] != str0[size - 1 - i]:
            return False
    return True

str0 = "AAAABBBBAAAA"
str1 = "AAAAAAAAABBB"

print("Test#1: %s"%("Pass" if True == check_palindrome(str0) else "Fail"))
print("Test#2: %s"%("Pass" if False == check_palindrome(str1) else "Fail"))


I am working on the file "Issues/algorithms/Arrays and Strings.ipynb".
The first task is:
String Compression:
Implement a method to perform basic string compression using the counts of repeated characters. For example, the string aabcccccaaa would become a2b1c5a3. If the "compressed" string would not become smaller than the original string, your method should return the original string. You can assume the string has only uppercase and lowercase letters (a - z).
Can you write Python code for it?

def _string_compression(str0):
    dest = ""
    cur_ch = str0[0]
    count = 0
    for i in range(len(str0)):
        if cur_ch == str0[i]:
            count += 1
        else:
            dest += cur_ch
            dest += str(count)
            cur_ch = str0[i]
            count = 1
    dest += cur_ch
    dest += str(count)
    cur_ch = str0[i]
    count = 0

    return dest

def string_compression(str0):
    str_ = _string_compression(str0)
    if len(str_) < len(str0):
        return str_
    else:
        return str0
print("Test#1: %s"%("Pass" if "A4B5C3" == string_compression("AAAABBBBBCCC") else "Fail"))
print("Test#2: %s"%("Pass" if "ABCDEF" == string_compression("ABCDEF") else "Fail"))
print("Test#3: %s"%("Pass" if "a2b1c5a3" == string_compression("aabcccccaaa") else "Fail"))


I am working on the file "Issues/algorithms/Linked Lists.ipynb".
The first task is:
2.1 Remove Dups:
Write code to remove duplicates from an unsorted linked list.
FOLLOW UP
How would you solve this problem if a temporary buffer is not allowed?
Can you write Python code for it?

List = Node(1, Node(2, Node(3, Node(4, Node(4, Node(4, Node(3, Node(2, Node(1)))))))))

def remove_dups(List):
    marks = {}
    cur = List
    prev = None
    while cur != None:
        if marks.get(cur.value, 0) == 0: # not duplicated
            marks[cur.value] = 1
        else: # duplicated
            prev.next = cur.next
            cur = prev

        prev = cur
        cur = cur.next

print('input:' + str(List))
remove_dups(List)
print('output:' + str(List))

def remove_dups_wo_buffer(List):
    cur0 = List
    while cur0 != None:
        prev = cur0
        cur1 = cur0.next
        while cur1 != None:
            if cur1.value == cur0.value:
                prev.next = cur1.next
                cur1 = prev
            prev = cur1
            cur1 = cur1.next

        cur0 = cur0.next

List = Node(1, Node(2, Node(3, Node(4, Node(4, Node(4, Node(3, Node(2, Node(1, Node(3, Node(2)))))))))))

print('input:' + str(List))
remove_dups_wo_buffer(List)
print('output:' + str(List))


I am working on the file "Issues/algorithms/Linked Lists.ipynb".
The first task is:
2.2 Return Kth to Last:
Implement an algorithm to find the kth to last element of a singly linked list.
Can you write Python code for it?

List = Node(1, Node(2, Node(3, Node(4, Node(4, Node(4, Node(3, Node(2, Node(1, Node(3, Node(2)))))))))))

def kth_to_last(List, k):
    cur = List
    size = 0
    while cur != None:
        size += 1
        cur = cur.next
    if size < k:
        return None

    cur = List
    for _ in range(size - k):
        cur = cur.next
    return cur.value

print(kth_to_last(List, 4))

def kth_to_last(head, k, i):

    if head == None:
        return None

    node = kth_to_last(head.next, k, i)
    i[0] = i[0] + 1
    if i[0] == k:
        return head
    else:
        return node

print(kth_to_last(List, 4, [0]))


I am working on the file "Iterable object.ipynb".
The first task is:
Creating iterable object
Iterable object can be defined using class.
Can you write Python code for it?

class Reverse:
    """Iterator for looping over a sequence backwards."""
    def __init__(self, data):
        self.data = data
        self.index = len(data)

    def __iter__(self):
        return self

    def __next__(self):
        if self.index == 0:
            raise StopIteration
        self.index = self.index - 1
        return self.data[self.index]

rev = Reverse('UCSY')
for c in rev:
    print(c)

print('---')

rev = Reverse([100, 200, 300])
for c in rev:
    print(c)


I am working on the file "Iterable object.ipynb".
The first task is:
Generator
Generators are a simple and powerful tool for creating iterators. They are written like regular functions but use the yield statement whenever they want to return data. Each time next() is called on it, the generator resumes where it left off (it remembers all the data values and which statement was last executed).
Can you write Python code for it?

def rev_gen(data):
#    for index in range(len(data)-1, -1, -1):
#        yield data[index]
    for elem in data[::-1]:
        yield elem

rev = rev_gen('ucsy')
for c in rev:
    print(c)

type(rev)


I am working on the file "IterateAndEvaluateYourClassifier.ipynb".
The first task is:
2. Gaussian Naive Bayes
Can you write Python code for it?

gnb = GaussianNB()
scores = cross_val_score(gnb, data, target, cv = 10, scoring="accuracy")

print("Percent accuracy within each fold:\n")
print(scores)
print("\nMean accuracy:\n")
print(scores.mean())

classifiers_compare = classifiers_compare.append([["GaussianNB", (scores.mean())]])


I am working on the file "IterateAndEvaluateYourClassifier.ipynb".
The first task is:
4. Decision Tree Classifier
Can you write Python code for it?

dtc = DecisionTreeClassifier(max_depth=5)
scores = cross_val_score(dtc, data, target, cv = 10, scoring="accuracy")
scores

print("Percent accuracy within each fold:\n")
print(scores)
print("\nMean accuracy:\n")
print(scores.mean())

classifiers_compare = classifiers_compare.append([["DecisionTree", (scores.mean())]])


I am working on the file "IterateAndEvaluateYourClassifier.ipynb".
The first task is:
Do any of your classifiers seem to overfit?
None of these classifiers seem to overfit. 10-fold cross validation was used in all cases, and the highest rate of accuracy was 77.3% on this run.
Which seem to perform the best? Why?
The Bernoulli Naive Bayes model returned the best accuracy when cross-validation was used. This is because the Bernoulli distribution is well suited to cases with two possible outcomes.
What features seemed to be most impactful to performance?
The model used had a large impact on performance. Specific keywords are also likely to be impactful to performance. Let's look at the difference in accuracy when we remove the five most-common keywords from the model.
Can you write Python code for it?

# Create a new Data Frame with the value counts for each key
mydf = pd.DataFrame(data.sum())
mydf.reset_index(inplace=True)
mydf.columns = ["key", "valueCounts"]

# Return the 5 most common keys found in the review set
myTopVals = sorted(list(mydf["valueCounts"]))[-5]
myTopVals

# Remove keys with more than threshold value of valuecounts.
newKeys = mydf[mydf.valueCounts < myTopVals].key
newKeys = list(newKeys)

data = amazon_raw[newKeys]
target = amazon_raw['satisfaction']
scores = cross_val_score(bnb, data, target, cv = 10, scoring="accuracy")

print("Percent accuracy within each fold:\n")
print(scores)
print("\nMean accuracy:\n")
print(scores.mean())


I am working on the file "IterateAndEvaluateYourClassifier.ipynb".
The first task is:
Looks like k=5 is the optimal k-value for this dataset.
Can you write Python code for it?

knn = KNeighborsClassifier(n_neighbors=5)

scores = cross_val_score(knn, data, target, cv = 10, scoring = "accuracy")

print("Percent accuracy within each fold:\n")
print(scores)
print("\nMean accuracy:\n")
print(scores.mean())

classifiers_compare = classifiers_compare.append([["KNN", (scores.mean())]])

classifiers_compare

plt.figure(figsize=(15,5))
sns.set(style="whitegrid")
sns.barplot(x=0, y=1, data=classifiers_compare, palette="BuGn_d")
plt.xlabel("Model and Distribution")
plt.ylim(0,1)
plt.ylabel("Percent Accuracy")
plt.show()


I am working on the file "Iterators and Generators Homework.ipynb".
The first task is:
Problem 2
Create a generator that yields "n" random numbers between a low and high number (that are inputs). Note: Use the random library. For example:
Can you write Python code for it?

import random

random.randint(1,10)

def rand_num(low,high,n):
    for i in range(n+1):
        yield random.randint(low, high)

for num in rand_num(1,10,12):
    print num


I am working on the file "Iterators and Generators.ipynb".
The first task is:
4. Pipelining Generators
Generators can be used to pipeline a series of operations.
If we are analysing a log file and if the log file has a 3rd column that keeps track of the ips every hour and we want to sum it to find unique ips in last 5 months.
Can you write Python code for it?

with open('sells.log') as file:
    ip_col = (line[3] for line in file)
    per_hr = (int(x) for x in ip_col if x != 'N/A')
    print("IPs =", sum(per_hr))


I am working on the file "Iterators and Generators.ipynb".
The first task is:
Generator Expressions
Generator Expressions are generator version of list comprehensions. They look like list comprehensions, but returns a generator back instead of a list.
Can you write Python code for it?


I am working on the file "Iterators and Generators.ipynb".
The first task is:
Generator expressions are more compact but less versatile than full generator definitions and tend to be more memory friendly than equivalent list comprehensions.
Can you write Python code for it?

xvec = [5,16,7]
yvec = [4,12,18]

sum(x * y for x,y in zip(xvec,yvec))

data = 'golf'
list(data[i] for i in range(len(data)-1, -1, -1))

# unique_words = set(word  for line in page  for word in line.split())

# valedictorian = max((student.gpa, student.name) for student in graduates)


I am working on the file "Iterators and Generators.ipynb".
The first task is:
Generators
Generators are a simple and powerful tool for creating iterators.
They are written like regular functions but use the yield statement whenever they want to return data. Each time next() is called on it, the generator resumes where it left off (it remembers all the data values and which statement was last executed). An example shows that generators can be trivially easy to create:
Can you write Python code for it?

def reverse(data):
    for index in range(len(data)-1, -1, -1):
        yield data[index]

for ch in reverse('shallow'):
    print(ch)


I am working on the file "Iterators and Generators.ipynb".
The first task is:
If use it with a dictionary, it loops over its keys
Can you write Python code for it?

for k in {1:'test1',2:'test'}:
    print(k)


I am working on the file "Iterators and Generators.ipynb".
The first task is:
Iteration Protocol
Can you write Python code for it?

x = iter([1,2,3])
print(x)
print(next(x))
print(next(x))
print(next(x))
print(next(x))  # <-- will create an error


I am working on the file "Iterators and Generators.ipynb".
The first task is:
Note that generators provide another way to deal with infinity, for example:
Can you write Python code for it?

from time import gmtime, strftime
def myGen():
    while True:
        yield strftime("%a, %d %b %Y %H:%M:%S +0000", gmtime())

myGeneratorInstance = myGen()
next(myGeneratorInstance)

next(myGeneratorInstance)


I am working on the file "Iterators and Generators.ipynb".
The first task is:
Since generators keep track of details automatically, it was concise and much cleaner in implementation.
2. Memory Efficient
A normal function to return a sequence will create the entire sequence in memory before returning the result. This is an overkill if the number of items in the sequence is very large.
3. Represent Infinite Stream
Generators are excellent medium to represent an infinite stream of data. Infinite streams cannot be stored in memory and since generators produce only one item at a time, it can represent infinite stream of data.
Can you write Python code for it?

def all_event():
    n = 0
    while True:
        yield n
        n += 2


I am working on the file "Iterators and Generators.ipynb".
The first task is:
This was lengthy. Now lets do the same using a generator function.
Can you write Python code for it?

def PowTwoGen(max = 0):
    n = 0
    while n < max:
        yield 2 ** n
        n += 1


I am working on the file "Iterators and Generators.ipynb".
The first task is:
Use of Generators
1. Easy to Implement
Generators can be implemented in a clear and concise way as compared to their iterator class counterpart.
Can you write Python code for it?

# Iterator Class
class PowTwo:
    def __init__(self, max = 0):
        self.max = max
    def __iter__(self):
        self.n = 0
        return self
    def __next__(self):
        if self.n > self.max:
            raise StopIteration

        result = 2 ** self.n
        self.n += 1
        return result


I am working on the file "JC_inferential_statistics_ex1.ipynb".
The first task is:
(1) The histogram and normal probability plot shows that the distribution of body temperatures approximately follows a normal distribution
Can you write Python code for it?

import numpy as np
import math
import pylab
import scipy.stats as stats
import matplotlib.pyplot as plt

plt.hist(df.temperature)
plt.show()

stats.probplot(df.temperature, dist="norm", plot=pylab)
pylab.show()


I am working on the file "JC_inferential_statistics_ex1.ipynb".
The first task is:
(5) We can use two-sample z test:
$H_0: T_M = T_F$
$H_A: T_M \neq T_F$
The p value is 0.02, which is smaller than 0.05. This indicates that there is a significant difference between males and females in normal temperature
Can you write Python code for it?

male_temp = df[df.gender=='M'].temperature
female_temp = df[df.gender=='F'].temperature

mean_diff = abs(np.mean(male_temp) - np.mean(female_temp))
se = math.sqrt(np.var(male_temp)/male_temp.count() +  np.var(female_temp)/female_temp.count() )
z = mean_diff/se
p_z = (1-stats.norm.cdf(z))*2
print('mean for male is ' + str(np.mean(male_temp)))
print('mean for female is ' + str(np.mean(female_temp)))
print('p value for z test is ' + str(p_z))


I am working on the file "JC_inferential_statistics_ex3.ipynb".
The first task is:
(1) Examine the distributions of excess readmission ratio for small and large hospitals using histograms and normal probability plots:
1. The distribution of excess readmission ratio for large hospitals follows a normal distribution, however, the distribution of excess readmission ratio for small hospitals deviates from a normal distribution with a long upper tail.
2. The distribution of the excess readmission ratio for large hospitals is more spread-out compared to that for small hospitals. The mean of excess readmission ratio for larger hospitals seem to be somewhat lower than that for smaller hospitals.
Can you write Python code for it?

plt.hist(small['Excess Readmission Ratio'], 50, normed=1, alpha = 0.7, label = 'Small Hospitals')
plt.hist(large['Excess Readmission Ratio'], 50, normed=1, alpha = 0.7, label = 'Large Hospitals')
plt.legend(loc='upper right')
plt.title('Normalized histograms of excess readmission ratio for small and large hospitals')
plt.show()

stats.probplot(small['Excess Readmission Ratio'], dist="norm", plot=pylab)
pylab.title('Probability plot for the excess readmission ratio of small hospitals')
pylab.show()

stats.probplot(large['Excess Readmission Ratio'], dist="norm", plot=pylab)
pylab.title('Probability plot for the excess readmission ratio of large hospitals')
pylab.show()

#data = [small['Excess Readmission Ratio'], large['Excess Readmission Ratio']]
#plt.boxplot(data)


I am working on the file "JC_inferential_statistics_ex3.ipynb".
The first task is:
(2) Since the distribution of the excess readmission ratio for small hospital deviates from normal distribution, a Mann–Whitney U test is performed here to test if there is a significant difference in excess readmission ratio $R$ between small (number of discharges < 100) and large (number of discharges > 1000) hospitals
$H_0: R_s = R_l$
$H_A: R_s \neq R_l$
In this case, the excess readmission ratio for small hospital is 0.044 higher than that for large hospital, with p value 3.50e-14 (<0.1). Therefore, we are able to reject the Hypothesis that the excess readmission ratio $R$ between small and large hospitals are the same. The 90% confidence interval of mean excess readmission ratio for small hospitals is [1.020, 1.026], and the 90% confidence interval of mean excess readmission ratio for large hospitals is [0.969, 0.987]. These confidence intervals were calculated using bootstrap.
Can you write Python code for it?

large = df1[df1['Number of Discharges']>=1000]
small = df1[df1['Number of Discharges']<100]

n_l = large['Number of Discharges'].count()
n_s = small['Number of Discharges'].count()
print('number of large hospitals: ' + str(n_l))
print('number of large hospitals: ' + str(n_s))

mean_l = np.mean(large['Excess Readmission Ratio'])
mean_s = np.mean(small['Excess Readmission Ratio'])

diff = mean_l - mean_s
md = stats.mannwhitneyu(small['Excess Readmission Ratio'],large['Excess Readmission Ratio'])
p_mw = md[1]

print('The excess readmission ratio of large hospitals is ' + str(abs(diff)) + ' lower compared to that of small hospitals with p value ' + str(abs(p_mw)) )

CI_s = bt.ci(data=small['Excess Readmission Ratio'], statfunction=scipy.mean, alpha=0.1)
CI_l = bt.ci(data=large['Excess Readmission Ratio'], statfunction=scipy.mean, alpha=0.1)
print('90% confidence interval of excess readmission ratio for small hospitals:')
print(CI_s)
print('90% confidence interval of excess readmission ratio for large hospitals:')
print(CI_l)


I am working on the file "JC_inferential_statistics_ex3.ipynb".
The first task is:
(3) Calculate the Correlation Coefficient

The distribution of the excess readmission ratio somewhat deviates from a normal distribution around its upper tail, and the distribution of the number of discharges is very right skew.
The Spearman Correlation Coefficient between these two variables shows that there is a weak correlation (r=-0.077) between these two variables, and this weak correlation is not due to sampling errors (p=0.22e-16).
Can you write Python code for it?

#Examine the distributions of 'Excess Readmission Ratio' and 'Number of Discharges'
excess_ratio = df1['Excess Readmission Ratio']
discharged = df1['Number of Discharges']

plt.hist(excess_ratio, 50)
plt.title('Histogram of excess readmission ratio')
plt.show()

stats.probplot(excess_ratio, dist="norm", plot=pylab)
pylab.title('Probability plot for the excess readmission ratio')
pylab.show()

plt.hist(discharged, 50)
plt.title('Histogram of number of discharges')
plt.show()

stats.probplot(discharged, dist="norm", plot=pylab)
pylab.title('Probability plot for number of discharges')
pylab.show()

#Since not both distributions are normal distributions, it is more appropriate to use Spearman rank correlation coefficient here.
sp = stats.spearmanr(discharged.as_matrix(), excess_ratio.as_matrix())
print('Spearman correlation coeffcient between number of discharges and excess readmission ratio: ')
print('correlation=' + str(sp[0]) + ' pvalue=' + str(sp[1]))


I am working on the file "JC_inferential_statistics_ex3.ipynb".
The first task is:
Preliminary Analysis
Can you write Python code for it?

# deal with missing and inconvenient portions of data
clean_hospital_read_df = hospital_read_df[hospital_read_df['Number of Discharges'] != 'Not Available']
clean_hospital_read_df.loc[:, 'Number of Discharges'] = clean_hospital_read_df['Number of Discharges'].astype(int)
clean_hospital_read_df = clean_hospital_read_df.sort_values('Number of Discharges')

# generate a scatterplot for number of discharges vs. excess rate of readmissions
# lists work better with matplotlib scatterplot function
x = [a for a in clean_hospital_read_df['Number of Discharges'][81:-3]]
y = list(clean_hospital_read_df['Excess Readmission Ratio'][81:-3])

fig, ax = plt.subplots(figsize=(8,5))
ax.scatter(x, y,alpha=0.2)

ax.fill_between([0,350], 1.15, 2, facecolor='red', alpha = .15, interpolate=True)
ax.fill_between([800,2500], .5, .95, facecolor='green', alpha = .15, interpolate=True)

ax.set_xlim([0, max(x)])
ax.set_xlabel('Number of discharges', fontsize=12)
ax.set_ylabel('Excess rate of readmissions', fontsize=12)
ax.set_title('Scatterplot of number of discharges vs. excess rate of readmissions', fontsize=14)

ax.grid(True)
fig.tight_layout()


I am working on the file "JC_inferential_statistics_ex3.ipynb".
The first task is:
Preliminary Report
Read the following results/report. While you are reading it, think about if the conclusions are correct, incorrect, misleading or unfounded. Think about what you would change or what additional analyses you would perform.
A. Initial observations based on the plot above
+ Overall, rate of readmissions is trending down with increasing number of discharges
+ With lower number of discharges, there is a greater incidence of excess rate of readmissions (area shaded red)
+ With higher number of discharges, there is a greater incidence of lower rates of readmissions (area shaded green)
B. Statistics
+ In hospitals/facilities with number of discharges < 100, mean excess readmission rate is 1.023 and 63% have excess readmission rate greater than 1
+ In hospitals/facilities with number of discharges > 1000, mean excess readmission rate is 0.978 and 44% have excess readmission rate greater than 1
C. Conclusions
+ There is a significant correlation between hospital capacity (number of discharges) and readmission rates.
+ Smaller hospitals/facilities may be lacking necessary resources to ensure quality care and prevent complications that lead to readmissions.
D. Regulatory policy recommendations
+ Hospitals/facilties with small capacity (< 300) should be required to demonstrate upgraded resource allocation for quality care to continue operation.
+ Directives and incentives should be provided for consolidation of hospitals and facilities to have a smaller number of them with higher capacity and number of discharges.
Exercise
Include your work on the following in this notebook and submit to your Github account.
A. Do you agree with the above analysis and recommendations? Why or why not?
B. Provide support for your arguments and your own recommendations with a statistically sound analysis:

Setup an appropriate hypothesis test.
Compute and report the observed significance value (or p-value).
Report statistical significance for $\alpha$ = .01.
Discuss statistical significance and practical...
Can you write Python code for it?

#Collect the entries with non-zero 'Number of Discharges' and not-null 'Excess Readmission Ratio'
df = clean_hospital_read_df.loc[clean_hospital_read_df ['Number of Discharges']!=0,]
df1 = df.loc[df['Excess Readmission Ratio'].notnull(),]


I am working on the file "JFreeChart Versions - Diving Differences.ipynb".
The first task is:

Analysis

Load Data

Analysis of Methods in Common (Stats)

Analysis of Comments

Methods with Same Comment
Take Away No. 1

Methods with Different Comment
Take Away No. 2

Analysis of Implementation

Methods with Same Comment
Take Away No. 3

Methods with Different Comment
Take Away No. 4

Analysis of Coherence

Methods with Same Comment and Code
Methods with Same Comment, Different Code
Take Away No. 5

Methods with Different Comment, Same Code
Take Away No. 6
Take Away No. 7

Methods with Different Comment and Code
Take Away No. 8
Take Away No. 9

Analysis of Methods NOT in Common

Coherence of Methods in JFreeChart 0.6.0
Coherence of Methods in JFreeChart 0.7.1
(Likely) Modified Methods from JFreeChart 0.6.0 to 0.7.1

Summary

Load Data
Can you write Python code for it?

from source_code_analysis.models import SoftwareProject

jfreechart_060 = SoftwareProject.objects.get(name__iexact='JFreeChart', version='0.6.0')
jfreechart_071 = SoftwareProject.objects.get(name__iexact='JFreeChart', version='0.7.1')

jf060_methods = gather_all_methods(jfreechart_060)
jf071_methods = gather_all_methods(jfreechart_071)


I am working on the file "JFreeChart Versions - Diving Differences.ipynb".
The first task is:
 Back to top
Analysis of the Comments
Can you write Python code for it?

# Set the Pipelines
comment_pipeline = Pipeline(strip_tags, strip_punctuations, normalise_lines)

# List to store the references to methods
# sharing (or not) the same comment between the two versions
same_comment = list()
different_comment = list()

for mkey in methods_in_common:
    comment_in_060 = comment_pipeline.process(jf060_methods[mkey].comment)
    comment_in_071 = comment_pipeline.process(jf071_methods[mkey].comment)

    if comment_in_060 == comment_in_071:
        same_comment.append(mkey)
    else:
        different_comment.append(mkey)


I am working on the file "JFreeChart Versions - Diving Differences.ipynb".
The first task is:
 Back to top
Common Methods
Can you write Python code for it?

# Set of all the Keys in common between the two considered versions
methods_in_common = set(jf060_methods.keys()).intersection(set(jf071_methods.keys()))
print('Total Methods in Common: ', len(methods_in_common))


I am working on the file "JFreeChart Versions - Diving Differences.ipynb".
The first task is:
 Back to top
JFreeChart 0.7.1
Can you write Python code for it?

coherent = list()
not_coherent = list()

for mkey in methods_in_071_not_in_060:
    method = jf071_methods[mkey]
    if is_coherent(method):
        coherent.append(mkey)
    else:
        not_coherent.append(mkey)

print('Total Number of Methods in JFreeChart 0.7.1 but NOT in 0.6.0:', len(methods_in_071_not_in_060))
print('\t Coherent: ', len(coherent))
print('\t Not Coherent', len(not_coherent))


I am working on the file "JFreeChart Versions - Diving Differences.ipynb".
The first task is:
 Back to top
(Likely) Modified Methods from JFreeChart 0.6.0 to 0.7.1
In this section, the main purpose of the analysis is to try to check some possible matches between methods not in JFreeChart 0.6.0 but present in JFreeChart 0.7.1.
The main idea is that we would like to find (or guess) all those methods whose signature has been changed/updated thus not appearing in the set of Common Methods.
Can you write Python code for it?

# Analyse the differences: Try to guess if there is some method that has been CHANGED between the two versions
from collections import defaultdict
associations_map = defaultdict(list)

def match(method_key, target_class, target_file, target_signature_stub):
    """We try to infer a possible matching between two methods if they share:
    - the same class name
    - the same source file name
    - their signature starts with the same _stub_
        In particular, the `target_signature_stub` corresponds to the
        first part of the signature till the first open paranthesis, i.e. "("
    """
    mth = jf060_methods[method_key]
    return (method_key not in jf071_methods and
            mth.code_class.src_filename == target_file
            and mth.code_class.class_name == target_class and
            signature(mth.code_fragment).startswith(target_signature_stub))

# Iterate over all methods in JFreeChart 0.7.1 and NOT in 0.6.0
# in order to guess some possible signature matchings
for mkey in methods_in_071_not_in_060:
    method = jf071_methods[mkey]
    signature_071 = signature(method.code_fragment)

    # get all the methods whose signature starts similarly to the target method
    signature_stub = signature_071[:signature_071.find('(')]
    class_name = method.code_class.class_name
    src_file = method.code_class.src_filename
    associations = list(filter(lambda k: match(k, class_name, src_file, signature_stub),
                               jf060_methods.keys()))
    associations_map[mkey] = associations

# Filter out all that had no matching in the first place
possible_mappings = {k:v for k, v in associations_map.items() if len(v)}

print('We inferred a total of {} matchings for {} methods NOT in common!'.format(len(possible_mappings),
                                                                                len(methods_in_071_not_in_060)))

# Print Matchings Guessed
for mkey, associations in possible_mappings.items():
    print('Target Signature: ',...


I am working on the file "JFreeChart Versions - Diving Differences.ipynb".
The first task is:
 Back to top
Methods in Common with the Same Comment
Can you write Python code for it?

# Test: get a random key and check that they actually share the same comments
# regardless the formattings (e.g. trailing spaces)
randomly_pick_a_method_from(same_comment)  # Test 1
randomly_pick_a_method_from(same_comment)  # Test 2
randomly_pick_a_method_from(same_comment)  # Test 3
randomly_pick_a_method_from(same_comment)  # Test 4


I am working on the file "JFreeChart Versions - Diving Differences.ipynb".
The first task is:
 Back to top
Take Away No. 1
From the total set of $283$ methods in common between JFreeChart 0.6.0 and 0.7.1,

$257$ ($\approx 91\%$) have no significant differences in comments (differences limited only to text formattings and punctuaction characters)
Hence this changes do not affect the coherence evaluation in any way.

 Back to top
Methods in Common with Different Comments
Can you write Python code for it?

# Qualitative Analysis (preliminary)

randomly_pick_a_method_from(different_comment)  # Test 1
randomly_pick_a_method_from(different_comment)  # Test 2
randomly_pick_a_method_from(different_comment)  # Test 3
randomly_pick_a_method_from(different_comment)  # Test 4


I am working on the file "JFreeChart Versions - Diving Differences.ipynb".
The first task is:
 Back to top
Take Away No. 2:
At a first glance, considering the comments of the methods included in the list of $26$ those with different comments, differences appear quite resonable (thus not only limited to layout and formattings).

 Back to top
Analysis of the Implementation
 Back to top
Analysis of the Implementation for Methods with the Same Comment
Can you write Python code for it?

methods_with_same_comments = same_comment  # code readability purposes

# Set the Pipeline
code_pipeline = Pipeline(extract_locs)

same_comment_and_code = list()
same_comment_different_code = list()

for mkey in methods_with_same_comments:
    code_in_060 = code_pipeline.process(jf060_methods[mkey].code_fragment)
    code_in_071 = code_pipeline.process(jf071_methods[mkey].code_fragment)

    if code_in_060 == code_in_071:
        same_comment_and_code.append(mkey)
    else:
        same_comment_different_code.append(mkey)

print('Total Number of Common Methods with the same comments: ', len(methods_with_same_comments))
print('\t No. of those Sharing the Same Code: ', len(same_comment_and_code))
print('\t No. of those With Differences in Code: ', len(same_comment_different_code))


I am working on the file "JFreeChart Versions - Diving Differences.ipynb".
The first task is:
 Back to top
Take Away No. 4:
Among the $26$ methods in common between the two version that have differences in their lead comments:

$16$ methods ($\approx 62\%$) have no difference in the implementation
$10$ methods have changes in code (as well as in comments)

 Back to top
Analysis of the Coherence
 Back to top
Coherence of  Methods with Same Comments and Code
Can you write Python code for it?

same_coherence = list()
coherence_changed = list()

for mkey in same_comment_and_code:
    mth_in_060 = jf060_methods[mkey]
    mth_in_071 = jf071_methods[mkey]

    if is_coherent(mth_in_060) == is_coherent(mth_in_071):
        same_coherence.append(mkey)
    else:
        coherence_changed.append(mkey)

print('Total Number of Methods Sharing the same Lead Comment and Code')
print('\t Same Coherence: ', len(same_coherence))
print('\t Different Coherence', len(coherence_changed))


I am working on the file "JFreeChart Versions - Diving Differences.ipynb".
The first task is:
 Back to top
Take Away No. 5:
Among those methods sharing the same lead comments but have differences in the implementations ($32$ in total),
None of them have differences in the coherence evaluation.
This means that differences in the implementation were limited to syntactic constructs and names of the variables.
 Back to top
Coherence of Methods with Different Comments but the Same Code
Can you write Python code for it?

same_coherence = list()
coherence_changed = list()

for mkey in different_comment_same_code:
    mth_in_060 = jf060_methods[mkey]
    mth_in_071 = jf071_methods[mkey]

    if is_coherent(mth_in_060) == is_coherent(mth_in_071):
        same_coherence.append(mkey)
    else:
        coherence_changed.append(mkey)

print('Total Number of Methods having different comment but the same code', len(different_comment_same_code))
print('\t Same Coherence: ', len(same_coherence))
print('\t Different Coherence', len(coherence_changed))

# Try to slpot some insights
randomly_pick_a_method_from(coherence_changed, with_code=True, with_coherence=True)  # Method 1
randomly_pick_a_method_from(coherence_changed, with_code=True, with_coherence=True)  # Method 2


I am working on the file "JFreeChart Versions - Diving Differences.ipynb".
The first task is:
 Back to top
Take Away No. 9:
This is interesting (and correct)!!.
In the only case (out of 10) where there is difference in the coherence evaluation between the two methods in common, the corresponding lead comments and implementations have been updated accordingly!
In fact, while there was no coherence for the method extracted from JFreeChart 0.6.0, there is for the one from JfreeChart 0.7.1.
 Back to top
Analysis of Methods NOT in Common
Can you write Python code for it?

methods_in_060 = set(jf060_methods.keys())
methods_in_071 = set(jf071_methods.keys())

# Set of Methods in 0.6.0 and not in 0.7.1
methods_in_060_not_in_071 = methods_in_060.difference(methods_in_071)
print('Total No. of Methods in JfreeChart 0.6.0 and NOT in 0.7.1: ', len(methods_in_060_not_in_071))

print('-'*80)

# Set of Methods in 0.7.1 and not in 0.6.0
methods_in_071_not_in_060 = methods_in_071.difference(methods_in_060)
print('Total No. of Methods in JfreeChart 0.7.1 and NOT in 0.6.0: ', len(methods_in_071_not_in_060))


I am working on the file "JFreeChart Versions - Diving Differences.ipynb".
The first task is:
Function to Analyse Coherence
Can you write Python code for it?

NOT_EVALUATED = -1
DONT_KNOW = 2
FURTHER_EVAL = 5
AGREEMENT = 3
STRONG_AGREEMENT = 4

def is_coherent(method):
    """Return wheter or not comment is coherent with its method implementation according to judges evaluations.

    Parameters
    ----------
    method : `source_code_analysis.models.CodeMethod`
        Instance of a `CodeMethod` model holding the reference to its corresponding evaluations.

    Returns
    -------
    bool :
        True if the evaluation (the first one retrieved from the db) corresponds to an
        AGREEMENT | STRONG_AGREEMENT value.
    """
    return (method.agreement_evaluations.last().agreement_vote in (AGREEMENT, STRONG_AGREEMENT))

def has_agreement_evaluations(method):
    """Check that input methods has agreement evaluations interesting for the current analysis
    (i.e. different from DONT_KNOW).

    Parameters
    ----------
    method : `source_code_analysis.models.CodeMethod`
        Instance of a `CodeMethod` model holding the reference to its corresponding evaluations.

    Returns
    -------
    bool :
        True if the evaluation (the first one retrieved from the db) does **not**
        correspond to a DONT_KNOW value.
    """
    return (method.agreement_evaluations.last().agreement_vote not in (NOT_EVALUATED,
                                                                       DONT_KNOW, FURTHER_EVAL))


I am working on the file "JFreeChart Versions - Diving Differences.ipynb".
The first task is:
Functions to gather Data from the DB
Can you write Python code for it?

def signature(code_fragment):
    """Returns the signature of a method extracted from the input code fragment.

    Parameters
    ----------
    code_fragment : str
        The implementation code of a method (i.e. method.code_fragment attribute)

    Returns
    -------
    str :
        The signature string of the method

    """
    first_line = code_fragment[:code_fragment.find('{')]
    return ' '.join([l.strip() for l in first_line.splitlines() if len(l.strip())])

def gather_all_methods(sw_project):
    """Gather all methods for the input software project

    Parameters
    ----------
    sw_project : `source_code_analysis.models.SoftwareProject`
        Target Software Project

    Returns
    -------
    dict :
        A dictionary mapping all methods with its unique key.
        This key is extremely important to correctly identify similarities among
        multiple versions of the same software.

        In more details, the key for a single *method* is defined by the following triple:

        * Name of the Source File
        * Name of the Class
        * Signature of the method
    """

    # gather all methods
    methods = filter(has_agreement_evaluations, sw_project.code_methods.all())

    # create the map
    methods_map = dict()
    for method in methods:
        key = '{}{}{}'.format(method.code_class.src_filename,
                              method.code_class.class_name,
                              signature(method.code_fragment))
        if not key in methods_map:
            methods_map[key] = method
        else:
            print('Key already present: ')
            print('Current method ID: ', method.id)
            print('Already present method: ', methods_map[key].id)

    # return map
    return methods_map


I am working on the file "JFreeChart Versions - Diving Differences.ipynb".
The first task is:
Percentage of Coherente / Not Coherent Methods
Analyse how many of the methods in the difference between the two versions are Coherent or Not Coherent
 Back to top
JFreeChart 0.6.0
Can you write Python code for it?

coherent = list()
not_coherent = list()

for mkey in methods_in_060_not_in_071:
    method = jf060_methods[mkey]
    if is_coherent(method):
        coherent.append(mkey)
    else:
        not_coherent.append(mkey)

print('Total Number of Methods in JFreeChart 0.6.0 but NOT in 0.7.1:', len(methods_in_060_not_in_071))
print('\t Coherent: ', len(coherent))
print('\t Not Coherent', len(not_coherent))


I am working on the file "JFreeChart Versions - Diving Differences.ipynb".
The first task is:
Pick some examples from those methods who share the same coherence evaluations
Can you write Python code for it?

# Try to spot some insights
randomly_pick_a_method_from(same_coherence, with_code=True, with_coherence=True)  # Method 1
randomly_pick_a_method_from(same_coherence, with_code=True, with_coherence=True)  # Method 2
randomly_pick_a_method_from(same_coherence, with_code=True, with_coherence=True)  # Method 3
randomly_pick_a_method_from(same_coherence, with_code=True, with_coherence=True)  # Method 4
randomly_pick_a_method_from(same_coherence, with_code=True, with_coherence=True)  # Method 5


I am working on the file "JFreeChart Versions - Diving Differences.ipynb".
The first task is:
Pipeline Process
Helper class that aids the creation and the application of multiple text processing functions
Can you write Python code for it?

class Pipeline(object):
    """Implements a simple linear pipeline process"""

    def __init__(self, *callables):
        """Creates a new Pipeline of processes (i.e. `callables`.)
        Each of this callable, must always return a value as it
        will represent the new `data` parameter passed through
        the pipeline till finally returned.

        Parameters
        ----------
        callables : list
            A list of callables (e.g. functions) of arity one.
            This list constitutes the set of filters of the pipeline.
        """
        self._filters = list(callables)

    @property
    def filters(self):
        return self._filters

    def __iadd__(self, process):
        if not callable(process):
            raise ValueError('The value should be a callable')

        from inspect import signature
        sig_process = signature(process)
        if len(sig_process.parameters) != 1:
            raise ValueError('The input function must have arity one!')

        self._filters.append(process)
        return self

    def process(self, data):
        """Execute the pipeline"""
        for callable in self._filters:
            data = callable(data)
        return data


I am working on the file "JFreeChart Versions - Diving Differences.ipynb".
The first task is:
Result
No differences occur in the evaluation of coherences for common methods sharing the same code and comments (as expected)
 Back to top
Coherence of Methods with Same Comments but Different Code
Can you write Python code for it?

same_coherence = list()
coherence_changed = list()

for mkey in same_comment_different_code:
    mth_in_060 = jf060_methods[mkey]
    mth_in_071 = jf071_methods[mkey]

    if is_coherent(mth_in_060) == is_coherent(mth_in_071):
        same_coherence.append(mkey)
    else:
        coherence_changed.append(mkey)

print('Total Number of Methods sharing the same lead comment but have different code', len(same_comment_different_code))
print('\t Same Coherence: ', len(same_coherence))
print('\t Different Coherence', len(coherence_changed))

# Try to spot some insights
randomly_pick_a_method_from(same_comment_different_code, with_code=True, with_coherence=True)  # Method 1
randomly_pick_a_method_from(same_comment_different_code, with_code=True, with_coherence=True)  # Method 2
randomly_pick_a_method_from(same_comment_different_code, with_code=True, with_coherence=True)  # Method 3
randomly_pick_a_method_from(same_comment_different_code, with_code=True, with_coherence=True)  # Method 4


I am working on the file "JFreeChart Versions - Diving Differences.ipynb".
The first task is:
Text Processing Functions (for code and comments)
Can you write Python code for it?

import re

def strip_tags(text):
    """Strips all HTML tags from text"""
    HTML_TAG_RE = re.compile(r'<[^>]+>')
    return HTML_TAG_RE.sub('', text)

from string import punctuation

def strip_punctuations(text, allowed='@'):
    """Strips all the punctuation character from the input comment.

    Parameters
    ----------
    text : str
        The input text to process

    allowed : str, optional.
        The list of punctuation characters to exclude from the processing
        (default is '@' as for JavaDoc Comments).

    """
    black_list = punctuation
    for c in allowed:
        black_list = black_list.replace(c, '')
    for p in black_list:
        text = text.replace(p, ' ')
    return '\n'.join(' '.join(w for w in line.split() if len(w)) for line in text.splitlines() if len(line))

def normalise_lines(text):
    """Removes additional (trailing) spaces from lines of the given text and
    returns it normalised (no extra spaces)"""
    return '\n'.join(l.strip() for l in text.splitlines() if len(l.strip()))

def extract_locs(code_fragment):
    """Normalise and returns the lines of code in the input fragment."""
    locs = ' '.join(l.strip() for st in code_fragment.split(';') for l in st.splitlines()
                    if len(l.strip()) and len(strip_punctuations(l.strip())))
    return locs


I am working on the file "JFreeChart Versions - Diving Differences.ipynb".
The first task is:
Utility function to test processing results and check what's going on
Can you write Python code for it?

def randomly_pick_a_method_from(list_of_methods_keys, with_code=False, with_coherence=False):
    """Randomly pick a method from the input collection of keys and print the
    corresponding lead comments. If `with_code` parameter is provided,
    the code_fragment is printed as well.
    Moreover, if the `with_coherence` parameter is True, the corresponding coherence evaluation is
    reported in the output, as well.
    """

    from random import choice
    random_key = choice(list_of_methods_keys)

    method_in_jf060 = jf060_methods[random_key]
    method_in_jf071 = jf071_methods[random_key]

    print('='*80)
    print('Method in JFreeChart 0.6.0', end=' ')
    if with_coherence:
        print('Is Coherent: ', is_coherent(method_in_jf060))
    else:
        print('')

    print(method_in_jf060.comment)
    if with_code:
        print('')
        print(method_in_jf060.code_fragment)

    print('\n\n')

    print('Method in JFreeChart 0.7.1', end=' ')
    if with_coherence:
        print('Is Coherent: ', is_coherent(method_in_jf071))
    else:
        print('')

    print(method_in_jf071.comment)
    if with_code:
        print('')
        print(method_in_jf071.code_fragment)

    print('='*80, end='\n\n')


I am working on the file "JHU Wifi.ipynb".
The first task is:
Now we have the data loaded nicely into a Pandas dataframe and we can look at some of the basics of the data.
Can you write Python code for it?

display('Number of rows: {}'.format(len(df)))
display('Unique SSIDs: {}'.format(len(df['SSID'].unique())))
display('Unique MACs: {}'.format(len(df['MAC'].unique())))
display('Number of Auth Mode types: {}'.format(len(df['AuthMode'].unique())))

def auth_filter(x):
    if 'WPA2' in x:
        return 'WPA2'
    elif 'WPA' in x:
        return 'WPA'
    elif 'WEP' in x:
        return 'WEP'
    else:
        return 'OPEN'

df['AuthMode'].apply(auth_filter).value_counts().plot(kind='barh')


I am working on the file "JHU Wifi.ipynb".
The first task is:
Over 500 networks use WPS! Using a tool like Reaver an attacker could easily breach those networks.
This is just some basic insights into the data. We could look further into the different forms of WPA/WPA2 authentication, but for an attacker these insights are enough. Using the above function for extracting WPS networks, an attacker could determine the locations of those networks and mount an attack on each of them.
Can you write Python code for it?

s = df['AuthMode'].apply(wps)
wps_entries = df.ix[s[s == 'WPS'].index]
wps_entries.head()


I am working on the file "JHU Wifi.ipynb".
The first task is:
So there are a significant number of open networks, but the overall majority use WPA2. That's good for the University but not so great for attackers. Of course, there could be a way around that via WPS. How many networks use that?
Can you write Python code for it?

def wps(x):
    if 'WPS' in x:
        return 'WPS'
    else:
        return 'Not WPS'
df['AuthMode'].apply(wps).value_counts().plot(kind='barh')


I am working on the file "JHU/information_retrieval/prog_4/prog_4.ipynb".
The first task is:
Bayes
Title Data
Can you write Python code for it?

vectorizer = CountVectorizer()
train_counts = vectorizer.fit_transform(train['title'])

#clf = MultinomialNB().fit(train_counts, train['assessment'])
clf = BernoulliNB(alpha=.001).fit(train_counts, train['assessment'])

preds = clf.predict(train_counts)

recall(preds, train['assessment'])
precision(preds, train['assessment'])
f1(preds, train['assessment'])

dev_text = vectorizer.transform(dev['title'])
preds = clf.predict(dev_text)

recall(preds, dev['assessment'])
precision(preds, dev['assessment'])
f1(preds, dev['assessment'])

clf.predict(vectorizer.transform([train.iloc[133]['abstract']]))


I am working on the file "JHU/information_retrieval/prog_4/prog_4.ipynb".
The first task is:
SVM
Can you write Python code for it?

vectorizer = CountVectorizer()
train_counts = vectorizer.fit_transform(train['title'])

#vectorizer = TfidfVectorizer(sublinear_tf=True, max_df=0.5, stop_words='english')
#all_abs = [item if isinstance(item, str) else '' for item in train['title']]
#train_counts = vectorizer.fit_transform(train['title'])

clf = svm.LinearSVC(C=1).fit(train_counts, train['assessment'])
print(clf)

preds = clf.predict(train_counts)

recall(preds, train['assessment'])
precision(preds, train['assessment'])
f1(preds, train['assessment'])

dev_text = vectorizer.transform(dev['title'])
preds = clf.predict(dev_text)

recall(preds, dev['assessment'])
precision(preds, dev['assessment'])
f1(preds, dev['assessment'])