I am working on the file "TCARER_TensorFlow.ipynb".
The first task is:
Set features' metadata dictionaries
Can you write Python code for it?

# Dictionary of features types, dtypes, & max-states
features_types = dict()
features_dtypes = dict()
features_states_values = dict()
features_names_group = dict()

for _, row in features_metadata.iterrows():
    if not pd.isnull(row["Variable_Max_States"]):
        states_values = str(row["Variable_Max_States"]).split(',')
        states_values = list(map(int, states_values))
    else:
        states_values = None

    if not pd.isnull(row["Variable_Aggregation"]):
        postfixes = row["Variable_Aggregation"].replace(' ', '').split(',')
        f_types = row["Variable_Type"].replace(' ', '').split(',')
        f_dtypes = row["Variable_dType"].replace(' ', '').split(',')
        for p in range(len(postfixes)):
            features_types[row["Variable_Name"] + "_" + postfixes[p]] = f_types[p]
            features_dtypes[row["Variable_Name"] + "_" + postfixes[p]] = pd.Series(dtype=f_dtypes[p])
            features_states_values[row["Variable_Name"] + "_" + postfixes[p]] = states_values
            features_names_group[row["Variable_Name"] + "_" + postfixes[p]] = row["Variable_Name"] + "_" + postfixes[p]
    else:
        features_types[row["Variable_Name"]] = row["Variable_Type"]
        features_dtypes[row["Variable_Name"]] = row["Variable_dType"]
        features_states_values[row["Variable_Name"]] = states_values
        features_names_group[row["Variable_Name"]] = row["Variable_Name"]
        if states_values is not None:
            for postfix in states_values:
                features_names_group[row["Variable_Name"] + "_" + str(postfix)] = row["Variable_Name"]

features_dtypes = pd.DataFrame(features_dtypes).dtypes

# Dictionary of features groups
features_types_group = OrderedDict()

f_types = set([f_type for f_type in features_types.values()])
features_types_group = OrderedDict(zip(list(f_types), [set() for _ in range(len(f_types))]))
for f_name, f_type in features_types.items():
    features_types_group[f_type].add(f_name)

print("Features types: " + ','.join(f_types))


I am working on the file "TCARER_TensorFlow.ipynb".
The first task is:
Set the lists of continous and discrete function
Can you write Python code for it?

continuous_features = list(features_types_group["CONTINUOUS"])
discrete_features = list(features_types_group["CATEGORICAL"])


I am working on the file "TCARER_TensorFlow.ipynb".
The first task is:
Set the output directory of the Tensorflow model
Can you write Python code for it?


I am working on the file "TCARER_TensorFlow.ipynb".
The first task is:
Set the validation monitor
Can you write Python code for it?

validation_mointor = tf.contrib.learn.monitors.ValidationMonitor(
    input_fn=lambda: input_fn(test_file_names_full, monitor_batch_size,
                              monitor_num_epochs, monitor_randomize_input),
    every_n_steps=monitor_steps)


I am working on the file "TCARER_TensorFlow.ipynb".
The first task is:
Test the saved file for corruption
Can you write Python code for it?

print("The model temp. directory to back up:")
print(model_dir)


I am working on the file "TCARER_TensorFlow.ipynb".
The first task is:
Visual verification
Can you write Python code for it?

display(pd.concat([features["train_id"].head(), features["train_target"].head(), features["train_indep"].head()], axis=1))
display(pd.concat([features["test_id"].head(), features["test_target"].head(), features["test_indep"].head()], axis=1))


I am working on the file "TCARER_summaryReports.ipynb".
The first task is:
1.1.  Initialise General Settings
Can you write Python code for it?

# init paths & directories
config_path = os.path.abspath("ConfigInputs/CONFIGURATIONS.ini")
io_path = os.path.abspath("../../tmp/TCARER/Basic_prototype")
schema = "parr_sample_prototype"
app_name = "T-CARER"

print("Output path:", io_path)

# init logs
if not os.path.exists(io_path):
    os.makedirs(io_path, exist_ok=True)

logger = Logger(path=io_path, app_name=app_name, ext="log")
logger = logging.getLogger(app_name)

# init constants
CONSTANTS.set(io_path, app_name)

# initialise other classes
readers_writers = ReadersWriters()
plots = Plots()

# other Constant variables
submodel_name = "hesIp"
submodel_input_name = "tcarer_model_features_ip"

# set print settings
pd.set_option('display.width', 1600, 'display.max_colwidth', 800)


I am working on the file "TCARER_summaryReports.ipynb".
The first task is:
3.3. Fit
Fit Model
Can you write Python code for it?

o_summaries = dict()
# Fit
model = charlson_training_method.train(features_extra["train"][charlson_features_names], features_extra["train"][target_feature], **kwargs)
charlson_training_method.save_model(path=CONSTANTS.io_path, title=file_name)

# load model
# charlson_training_method.load(path=CONSTANTS.io_path, title=file_name)

# short summary
o_summaries = charlson_training_method.train_summaries()


I am working on the file "TCARER_summaryReports.ipynb".
The first task is:
3.4. Predict
Can you write Python code for it?

o_summaries = dict()
model = charlson_training_method.predict(features_extra["test"][charlson_features_names], "test")

# short summary
o_summaries = charlson_training_method.predict_summaries(pd.Series(features_extra["test"][target_feature]), "test")
print("ROC AUC:", o_summaries['roc_auc_score_1'], "\n", o_summaries['classification_report'])
for k in o_summaries.keys():
    print(k,  o_summaries[k])


I am working on the file "TCARER_summaryReports.ipynb".
The first task is:
3.5. Cross-Validate
Can you write Python code for it?

o_summaries = dict()
score = charlson_training_method.cross_validate(features_extra["test"][charlson_features_names], features_extra["test"][target_feature],
                                             scoring="neg_mean_squared_error", cv=10)

# short summary
o_summaries = charlson_training_method.cross_validate_summaries()
print("Scores: ", o_summaries)


I am working on the file "TCARER_summaryReports.ipynb".
The first task is:
3.6. Save
Can you write Python code for it?


I am working on the file "TCARER_summaryReports.ipynb".
The first task is:
5.2.1. Most Prevalent Diagnoses Groups
Most prevalent diagnoses groups (30-day, 1-year readmission):
* Total, Admissions, Emergency Readmission Rate, Prior Spells, Male (%), Age (IQR), LoS (IQR), TP, FP, FN, TN
Can you write Python code for it?

diagnoses = ['diagCCS_0t30d_others_cnt', 'diagCCS_0t30d_prevalence_1_cnt', 'diagCCS_0t30d_prevalence_2_cnt', 'diagCCS_0t30d_prevalence_3_cnt', 'diagCCS_0t30d_prevalence_4_cnt', 'diagCCS_0t30d_prevalence_5_cnt', 'diagCCS_0t30d_prevalence_6_cnt', 'diagCCS_0t30d_prevalence_7_cnt', 'diagCCS_0t30d_prevalence_8_cnt', 'diagCCS_0t30d_prevalence_9_cnt', 'diagCCS_0t30d_prevalence_10_cnt', 'diagCCS_0t30d_prevalence_11_cnt', 'diagCCS_0t30d_prevalence_12_cnt', 'diagCCS_0t30d_prevalence_13_cnt', 'diagCCS_0t30d_prevalence_14_cnt', 'diagCCS_0t30d_prevalence_15_cnt', 'diagCCS_0t30d_prevalence_16_cnt', 'diagCCS_0t30d_prevalence_17_cnt', 'diagCCS_0t30d_prevalence_18_cnt', 'diagCCS_0t30d_prevalence_19_cnt', 'diagCCS_0t30d_prevalence_20_cnt', 'diagCCS_0t30d_prevalence_21_cnt', 'diagCCS_0t30d_prevalence_22_cnt', 'diagCCS_0t30d_prevalence_23_cnt', 'diagCCS_0t30d_prevalence_24_cnt', 'diagCCS_0t30d_prevalence_25_cnt', 'diagCCS_0t30d_prevalence_26_cnt', 'diagCCS_0t30d_prevalence_27_cnt', 'diagCCS_0t30d_prevalence_28_cnt', 'diagCCS_0t30d_prevalence_29_cnt', 'diagCCS_0t30d_prevalence_30_cnt'
            , 'diagCCS_30t90d_others_cnt', 'diagCCS_30t90d_prevalence_1_cnt', 'diagCCS_30t90d_prevalence_2_cnt', 'diagCCS_30t90d_prevalence_3_cnt', 'diagCCS_30t90d_prevalence_4_cnt', 'diagCCS_30t90d_prevalence_5_cnt', 'diagCCS_30t90d_prevalence_6_cnt', 'diagCCS_30t90d_prevalence_7_cnt', 'diagCCS_30t90d_prevalence_8_cnt', 'diagCCS_30t90d_prevalence_9_cnt', 'diagCCS_30t90d_prevalence_10_cnt', 'diagCCS_30t90d_prevalence_11_cnt', 'diagCCS_30t90d_prevalence_12_cnt', 'diagCCS_30t90d_prevalence_13_cnt', 'diagCCS_30t90d_prevalence_14_cnt', 'diagCCS_30t90d_prevalence_15_cnt', 'diagCCS_30t90d_prevalence_16_cnt', 'diagCCS_30t90d_prevalence_17_cnt', 'diagCCS_30t90d_prevalence_18_cnt', 'diagCCS_30t90d_prevalence_19_cnt', 'diagCCS_30t90d_prevalence_20_cnt', 'diagCCS_30t90d_prevalence_21_cnt', 'diagCCS_30t90d_prevalence_22_cnt', 'diagCCS_30t90d_prevalence_23_cnt', 'diagCCS_30t90d_prevalence_24_cnt', 'diagCCS_30t90d_prevalence_25_cnt',...


I am working on the file "TCARER_summaryReports.ipynb".
The first task is:
5.2.2. Major Comorbidity Groups
Comorbidity diagnoses groups (30-day, 1-year readmission):
* Total, Admissions, Emergency Readmission Rate, Prior Spells, Male (%), Age (IQR), LoS (IQR), TP, FP, FN, TN
Can you write Python code for it?

diagnoses = ['prior_admiOther', 'prior_admiAcute', 'prior_spells', 'prior_asthma', 'prior_copd', 'prior_depression', 'prior_diabetes', 'prior_hypertension', 'prior_cancer', 'prior_chd', 'prior_chf']
file_name = "report_population_comorbidity_diagnoses_" + method_name + "_" + target_feature + "_"

o_summaries = population_statistics(features_extra['train'], diagnoses)
readers_writers.save_csv(path=CONSTANTS.io_path, title=file_name + "train", data=o_summaries, append=False, extension="csv", header=o_summaries.columns)

o_summaries = population_statistics(features_extra['test'], diagnoses)
readers_writers.save_csv(path=CONSTANTS.io_path, title=file_name + "test", data=o_summaries, append=False, extension="csv", header=o_summaries.columns)


I am working on the file "TCARER_summaryReports.ipynb".
The first task is:
5.2.3. Charlson Comorbidity Groups
Charlson diagnoses groups (30-day, 1-year readmission):
* Total, Admissions, Emergency Readmission Rate, Prior Spells, Male (%), Age (IQR), LoS (IQR), TP, FP, FN, TN
Can you write Python code for it?

diagnoses = ['diagCci_01_myocardial_freq', 'diagCci_02_chf_freq', 'diagCci_03_pvd_freq', 'diagCci_04_cerebrovascular_freq', 'diagCci_05_dementia_freq', 'diagCci_06_cpd_freq', 'diagCci_07_rheumatic_freq', 'diagCci_08_ulcer_freq', 'diagCci_09_liverMild_freq', 'diagCci_10_diabetesNotChronic_freq', 'diagCci_11_diabetesChronic_freq', 'diagCci_12_hemiplegia_freq', 'diagCci_13_renal_freq', 'diagCci_14_malignancy_freq', 'diagCci_15_liverSevere_freq', 'diagCci_16_tumorSec_freq', 'diagCci_17_aids_freq', 'diagCci_18_depression_freq', 'diagCci_19_cardiac_freq', 'diagCci_20_valvular_freq', 'diagCci_21_pulmonary_freq', 'diagCci_22_vascular_freq', 'diagCci_23_hypertensionNotComplicated_freq', 'diagCci_24_hypertensionComplicated_freq', 'diagCci_25_paralysis_freq', 'diagCci_26_neuroOther_freq', 'diagCci_27_pulmonaryChronic_freq', 'diagCci_28_diabetesNotComplicated_freq', 'diagCci_29_diabetesComplicated_freq', 'diagCci_30_hypothyroidism_freq', 'diagCci_31_renal_freq', 'diagCci_32_liver_freq', 'diagCci_33_ulcerNotBleeding_freq', 'diagCci_34_psychoses_freq', 'diagCci_35_lymphoma_freq', 'diagCci_36_cancerSec_freq', 'diagCci_37_tumorNotSec_freq', 'diagCci_38_rheumatoid_freq', 'diagCci_39_coagulopathy_freq', 'diagCci_40_obesity_freq', 'diagCci_41_weightLoss_freq', 'diagCci_42_fluidDisorder_freq', 'diagCci_43_bloodLoss_freq', 'diagCci_44_anemia_freq', 'diagCci_45_alcohol_freq', 'diagCci_46_drug_freq']
file_name = "report_population_charlson_diagnoses_" + method_name + "_" + target_feature + "_"

o_summaries = population_statistics(features_extra['train'], diagnoses)
readers_writers.save_csv(path=CONSTANTS.io_path, title=file_name + "train", data=o_summaries, append=False, extension="csv", header=o_summaries.columns)

o_summaries = population_statistics(features_extra['test'], diagnoses)
readers_writers.save_csv(path=CONSTANTS.io_path, title=file_name + "test", data=o_summaries, append=False, extension="csv", header=o_summaries.columns)


I am working on the file "TCARER_summaryReports.ipynb".
The first task is:
5.2.4. Most Prevalent Main Speciality
Most prevalent operations variables (30-day, 1-year readmission):
* Total, Admissions, Emergency Readmission Rate, Prior Spells, Male (%), Age (IQR), LoS (IQR), TP, FP, FN, TN
Can you write Python code for it?

diagnoses = ['mainspef_0t30d_others_cnt', 'mainspef_0t30d_prevalence_1_cnt', 'mainspef_0t30d_prevalence_2_cnt', 'mainspef_0t30d_prevalence_3_cnt', 'mainspef_0t30d_prevalence_4_cnt', 'mainspef_0t30d_prevalence_5_cnt', 'mainspef_0t30d_prevalence_6_cnt', 'mainspef_0t30d_prevalence_7_cnt', 'mainspef_0t30d_prevalence_8_cnt', 'mainspef_0t30d_prevalence_9_cnt', 'mainspef_0t30d_prevalence_10_cnt'
            , 'mainspef_30t90d_others_cnt', 'mainspef_30t90d_prevalence_1_cnt', 'mainspef_30t90d_prevalence_2_cnt', 'mainspef_30t90d_prevalence_3_cnt', 'mainspef_30t90d_prevalence_4_cnt', 'mainspef_30t90d_prevalence_5_cnt', 'mainspef_30t90d_prevalence_6_cnt', 'mainspef_30t90d_prevalence_7_cnt', 'mainspef_30t90d_prevalence_8_cnt', 'mainspef_30t90d_prevalence_9_cnt', 'mainspef_30t90d_prevalence_10_cnt'
            , 'mainspef_90t180d_others_cnt', 'mainspef_90t180d_prevalence_1_cnt', 'mainspef_90t180d_prevalence_2_cnt', 'mainspef_90t180d_prevalence_3_cnt', 'mainspef_90t180d_prevalence_4_cnt', 'mainspef_90t180d_prevalence_5_cnt', 'mainspef_90t180d_prevalence_6_cnt', 'mainspef_90t180d_prevalence_7_cnt', 'mainspef_90t180d_prevalence_8_cnt', 'mainspef_90t180d_prevalence_9_cnt', 'mainspef_90t180d_prevalence_10_cnt'
            , 'mainspef_180t365d_others_cnt', 'mainspef_180t365d_prevalence_1_cnt', 'mainspef_180t365d_prevalence_2_cnt', 'mainspef_180t365d_prevalence_3_cnt', 'mainspef_180t365d_prevalence_4_cnt', 'mainspef_180t365d_prevalence_5_cnt', 'mainspef_180t365d_prevalence_6_cnt', 'mainspef_180t365d_prevalence_7_cnt', 'mainspef_180t365d_prevalence_8_cnt', 'mainspef_180t365d_prevalence_9_cnt', 'mainspef_180t365d_prevalence_10_cnt'
            , 'mainspef_365t730d_others_cnt', 'mainspef_365t730d_prevalence_1_cnt', 'mainspef_365t730d_prevalence_2_cnt', 'mainspef_365t730d_prevalence_3_cnt', 'mainspef_365t730d_prevalence_4_cnt', 'mainspef_365t730d_prevalence_5_cnt', 'mainspef_365t730d_prevalence_6_cnt', 'mainspef_365t730d_prevalence_7_cnt', 'mainspef_365t730d_prevalence_8_cnt', 'mainspef_365t730d_prevalence_9_cnt',...


I am working on the file "TCARER_summaryReports.ipynb".
The first task is:
5.2.4. Most Prevalent Operatons
Most prevalent operations variables (30-day, 1-year readmission):
* Total, Admissions, Emergency Readmission Rate, Prior Spells, Male (%), Age (IQR), LoS (IQR), TP, FP, FN, TN
Can you write Python code for it?

diagnoses = ['operOPCSL1_0t30d_others_cnt', 'operOPCSL1_0t30d_prevalence_1_cnt', 'operOPCSL1_0t30d_prevalence_2_cnt', 'operOPCSL1_0t30d_prevalence_3_cnt', 'operOPCSL1_0t30d_prevalence_4_cnt', 'operOPCSL1_0t30d_prevalence_5_cnt', 'operOPCSL1_0t30d_prevalence_6_cnt', 'operOPCSL1_0t30d_prevalence_7_cnt', 'operOPCSL1_0t30d_prevalence_8_cnt', 'operOPCSL1_0t30d_prevalence_9_cnt', 'operOPCSL1_0t30d_prevalence_10_cnt', 'operOPCSL1_0t30d_prevalence_11_cnt', 'operOPCSL1_0t30d_prevalence_12_cnt', 'operOPCSL1_0t30d_prevalence_13_cnt', 'operOPCSL1_0t30d_prevalence_14_cnt', 'operOPCSL1_0t30d_prevalence_15_cnt', 'operOPCSL1_0t30d_prevalence_16_cnt', 'operOPCSL1_0t30d_prevalence_17_cnt', 'operOPCSL1_0t30d_prevalence_18_cnt', 'operOPCSL1_0t30d_prevalence_19_cnt', 'operOPCSL1_0t30d_prevalence_20_cnt', 'operOPCSL1_0t30d_prevalence_21_cnt', 'operOPCSL1_0t30d_prevalence_22_cnt', 'operOPCSL1_0t30d_prevalence_23_cnt', 'operOPCSL1_0t30d_prevalence_24_cnt', 'operOPCSL1_0t30d_prevalence_25_cnt', 'operOPCSL1_0t30d_prevalence_26_cnt', 'operOPCSL1_0t30d_prevalence_27_cnt', 'operOPCSL1_0t30d_prevalence_28_cnt', 'operOPCSL1_0t30d_prevalence_29_cnt', 'operOPCSL1_0t30d_prevalence_30_cnt'
            , 'operOPCSL1_30t90d_others_cnt', 'operOPCSL1_30t90d_prevalence_1_cnt', 'operOPCSL1_30t90d_prevalence_2_cnt', 'operOPCSL1_30t90d_prevalence_3_cnt', 'operOPCSL1_30t90d_prevalence_4_cnt', 'operOPCSL1_30t90d_prevalence_5_cnt', 'operOPCSL1_30t90d_prevalence_6_cnt', 'operOPCSL1_30t90d_prevalence_7_cnt', 'operOPCSL1_30t90d_prevalence_8_cnt', 'operOPCSL1_30t90d_prevalence_9_cnt', 'operOPCSL1_30t90d_prevalence_10_cnt', 'operOPCSL1_30t90d_prevalence_11_cnt', 'operOPCSL1_30t90d_prevalence_12_cnt', 'operOPCSL1_30t90d_prevalence_13_cnt', 'operOPCSL1_30t90d_prevalence_14_cnt', 'operOPCSL1_30t90d_prevalence_15_cnt', 'operOPCSL1_30t90d_prevalence_16_cnt', 'operOPCSL1_30t90d_prevalence_17_cnt', 'operOPCSL1_30t90d_prevalence_18_cnt', 'operOPCSL1_30t90d_prevalence_19_cnt', 'operOPCSL1_30t90d_prevalence_20_cnt', 'operOPCSL1_30t90d_prevalence_21_cnt',...


I am working on the file "TCARER_summaryReports.ipynb".
The first task is:
5.2.5. Other Variables
Other variables (30-day, 1-year readmission):
* Total, Admissions, Emergency Readmission Rate, Prior Spells, Male (%), Age (IQR), LoS (IQR), TP, FP, FN, TN
Can you write Python code for it?

diagnoses = ['gapDays_0t30d_avg', 'gapDays_30t90d_avg', 'gapDays_90t180d_avg', 'gapDays_180t365d_avg', 'gapDays_365t730d_avg',
             'epidur_0t30d_avg', 'epidur_30t90d_avg', 'epidur_90t180d_avg', 'epidur_180t365d_avg', 'epidur_365t730d_avg',
             'preopdur_0t30d_avg', 'preopdur_30t90d_avg', 'preopdur_90t180d_avg', 'preopdur_180t365d_avg', 'preopdur_365t730d_avg',
             'posopdur_0t30d_avg', 'posopdur_30t90d_avg', 'posopdur_90t180d_avg', 'posopdur_180t365d_avg', 'posopdur_365t730d_avg']
file_name = "report_population_other_variables_" + method_name + "_" + target_feature + "_"

o_summaries = population_statistics(features_extra['train'], diagnoses)
readers_writers.save_csv(path=CONSTANTS.io_path, title=file_name + "train", data=o_summaries, append=False, extension="csv", header=o_summaries.columns)

o_summaries = population_statistics(features_extra['test'], diagnoses)
readers_writers.save_csv(path=CONSTANTS.io_path, title=file_name + "test", data=o_summaries, append=False, extension="csv", header=o_summaries.columns)


I am working on the file "TCARER_summaryReports.ipynb".
The first task is:
5.3.1. ROC
Can you write Python code for it?

fig, summaries = plots.roc(training_method.model_predict["test"], features_extra["test"][target_feature],
                           title="ROC Curve", lw=2)
display(fig)

# save
plt.savefig(os.path.join(CONSTANTS.io_path, file_name + "_roc" + ".pdf"),
            dpi=300, facecolor='w', edgecolor='w', orientation='portrait', papertype=None, format="pdf",
            transparent=False, bbox_inches=None, pad_inches=0.1, frameon=None)


I am working on the file "TCARER_summaryReports.ipynb".
The first task is:
5.3.3. Learning Curve
Can you write Python code for it?

fig, summaries = plots.learning_curve(training_method.model_train,
                                      features_extra["test"][features_names_selected],
                                      features_extra["test"][target_feature],
                                      title="Learning Curve", ylim=None, cv=None, n_jobs=-1, train_sizes=np.linspace(.1, 1.0, 5))
display(fig)

# save
plt.savefig(os.path.join(CONSTANTS.io_path, file_name + "_learning_curve" + ".pdf"),
            dpi=300, facecolor='w', edgecolor='w', orientation='portrait', papertype=None, format="pdf",
            transparent=False, bbox_inches=None, pad_inches=0.1, frameon=None)


I am working on the file "TCARER_summaryReports.ipynb".
The first task is:

2.1. Initialise
Can you write Python code for it?

# select the target variable
target_feature = "label365" # "label365", "label30"
method_name = "rfc" # "rfc", "gbrt", "randLogit", "wdnn"
rank_models = ["rfc"] # ["rfc", "gbrt", "randLogit"]


I am working on the file "TCARER_summaryReports.ipynb".
The first task is:

2.2. Load Features
Load pre-processed features
Can you write Python code for it?

file_name = "Step_07_Features"
features = readers_writers.load_serialised_compressed(path=CONSTANTS.io_path, title=file_name)

# print
print("File size: ", os.stat(os.path.join(CONSTANTS.io_path, file_name + ".bz2")).st_size)
print("Number of columns: ", len(features["train_indep"].columns))
print("features: {train: ", len(features["train_indep"]), ", test: ", len(features["test_indep"]), "}")


I am working on the file "TCARER_summaryReports.ipynb".
The first task is:

2.5. Load the Extra Features for Benchmarking
Read the extra features
Can you write Python code for it?

table = 'tcarer_featuresExtra'
features_extra_dtypes = {'patientID': 'U32', 'trigger_charlsonFoster': 'i4', 'trigger_los': 'i4', 'trigger_age': 'i4', 'prior_admiOther': 'i4', 'prior_admiAcute': 'i4',
                         'prior_spells': 'i4', 'prior_asthma': 'i4', 'prior_copd': 'i4', 'prior_depression': 'i4', 'prior_diabetes': 'i4', 'prior_hypertension': 'i4', 'prior_cancer': 'i4', 'prior_chd': 'i4', 'prior_chf': 'i4',
                         'diagCci_01_myocardial_freq': 'i4', 'diagCci_02_chf_freq': 'i4', 'diagCci_03_pvd_freq': 'i4', 'diagCci_04_cerebrovascular_freq': 'i4', 'diagCci_05_dementia_freq': 'i4', 'diagCci_06_cpd_freq': 'i4', 'diagCci_07_rheumatic_freq': 'i4', 'diagCci_08_ulcer_freq': 'i4', 'diagCci_09_liverMild_freq': 'i4', 'diagCci_10_diabetesNotChronic_freq': 'i4', 'diagCci_11_diabetesChronic_freq': 'i4', 'diagCci_12_hemiplegia_freq': 'i4', 'diagCci_13_renal_freq': 'i4', 'diagCci_14_malignancy_freq': 'i4', 'diagCci_15_liverSevere_freq': 'i4', 'diagCci_16_tumorSec_freq': 'i4', 'diagCci_17_aids_freq': 'i4', 'diagCci_18_depression_freq': 'i4', 'diagCci_19_cardiac_freq': 'i4', 'diagCci_20_valvular_freq': 'i4', 'diagCci_21_pulmonary_freq': 'i4', 'diagCci_22_vascular_freq': 'i4', 'diagCci_23_hypertensionNotComplicated_freq': 'i4', 'diagCci_24_hypertensionComplicated_freq': 'i4', 'diagCci_25_paralysis_freq': 'i4', 'diagCci_26_neuroOther_freq': 'i4', 'diagCci_27_pulmonaryChronic_freq': 'i4', 'diagCci_28_diabetesNotComplicated_freq': 'i4', 'diagCci_29_diabetesComplicated_freq': 'i4', 'diagCci_30_hypothyroidism_freq': 'i4', 'diagCci_31_renal_freq': 'i4', 'diagCci_32_liver_freq': 'i4', 'diagCci_33_ulcerNotBleeding_freq': 'i4', 'diagCci_34_psychoses_freq': 'i4', 'diagCci_35_lymphoma_freq': 'i4', 'diagCci_36_cancerSec_freq': 'i4', 'diagCci_37_tumorNotSec_freq': 'i4', 'diagCci_38_rheumatoid_freq': 'i4', 'diagCci_39_coagulopathy_freq': 'i4', 'diagCci_40_obesity_freq': 'i4', 'diagCci_41_weightLoss_freq': 'i4', 'diagCci_42_fluidDisorder_freq': 'i4', 'diagCci_43_bloodLoss_freq': 'i4',...


I am working on the file "TCARER_summaryReports.ipynb".
The first task is:

3.2. Initialise
Can you write Python code for it?

# set features
charlson_features_names = ['trigger_charlsonFoster']

# select the target variable
charlson_target_feature = "label30" # "label30",  "label365"

# file name
file_name = "report_Model_Charlson_" + charlson_method_name + "_" + charlson_target_feature

# initialise
charlson_training_method = TrainingMethod(charlson_method_name)


I am working on the file "TCARER_summaryReports.ipynb".
The first task is:

5. Model Performance
5.1. Performance Indicators
Can you write Python code for it?

measures = ["accuracy_score", "precision_score", "recall_score",
            "roc_auc_score_1", "f1_score", "fbeta_score", "average_precision_score",
            "log_loss", "zero_one_loss", "hamming_loss", "jaccard_similarity_score", "matthews_corrcoef"]

# train
o_summaries = training_method.predict_summaries(features_extra["train"][target_feature], "train")
o_summaries = np.array([(m, o_summaries[m]) for m in measures])
report_performance = pd.DataFrame({"Measure": o_summaries[:, 0],
                                   "Sample Train": o_summaries[:, 1],
                                   "Sample Test": [None] * len(measures)})

# test
o_summaries = training_method.predict_summaries(features_extra["test"][target_feature], "test")
o_summaries = np.array([(m, o_summaries[m]) for m in measures])
report_performance["Sample Test"] = o_summaries[:, 1]

# print
file_name = "report_performance_" + method_name + "_" + target_feature
display(report_performance)
readers_writers.save_csv(path=CONSTANTS.io_path, title=file_name, data=report_performance, append=False)


I am working on the file "TCARER_summaryReports.ipynb".
The first task is:

5.3. Plots
Can you write Python code for it?

file_name = "report_population_" + method_name + "_" + target_feature + "_"


I am working on the file "TCARER_summaryReports.ipynb".
The first task is:

Performance
Can you write Python code for it?

# train
o_summaries = training_method.predict_summaries(features["train_target"][target_feature], "train")
for k in o_summaries.keys():
    print(k,  o_summaries[k])

print("\n")
# test
o_summaries = training_method.predict_summaries(features["test_target"][target_feature], "test")
for k in o_summaries.keys():
    print(k,  o_summaries[k])


I am working on the file "TCARER_summaryReports.ipynb".
The first task is:
Algorithm 2: Logistic Regression
Can you write Python code for it?

charlson_method_name = "lr"
kwargs = {"penalty": 'l2', "dual": False, "tol": 0.0001, "C": 1, "fit_intercept": True, "intercept_scaling": 1,
          "class_weight": None, "random_state": None, "solver": 'liblinear', "max_iter": 100, "multi_class": 'ovr',
          "verbose": 0, "warm_start": False, "n_jobs": -1}


I am working on the file "TCARER_summaryReports.ipynb".
The first task is:
Clean-up
Can you write Python code for it?


I am working on the file "TCARER_summaryReports.ipynb".
The first task is:
Combine (join by PatientID)
Can you write Python code for it?

features_extra['train'] = features_extra['train'].merge(
    pd.concat([features['train_id'], features['train_target'],
               pd.DataFrame({'score': training_method.model_predict["train"]['score']}), features['train_indep']], axis=1),
    how="inner", on="patientID")
features_extra['test'] = features_extra['test'].merge(
    pd.concat([features['test_id'], features['test_target'],
               pd.DataFrame({'score': training_method.model_predict["test"]['score']}), features['test_indep']], axis=1),
    how="inner", on="patientID")


I am working on the file "TCARER_summaryReports.ipynb".
The first task is:
Common variables:
* Readmission
    * 'label30', 'label365'
* Admissions Methods:
    * 'admimeth_0t30d_prevalence_1_cnt', ...
* Prior Spells:
    * 'prior_spells'
* Male:
    * 'gender_1'
* LoS:
    * 'trigger_los'
* Age:
    * 'trigger_age'
* Charlson Score:
    * 'trigger_charlsonFoster'
* predictions score
    * score
* Most prevalent diagnoses groups (0-30-day, 0-730-day):
    * 0-30-day: 'diagCCS_0t30d_prevalence_1_cnt', ...
    * 0-730-day: 'diagCCS_0t30d_prevalence_1_cnt' + 'diagCCS_30t90d_prevalence_1_cnt' +
        'diagCCS_90t180d_prevalence_1_cnt' + 'diagCCS_180t365d_prevalence_1_cnt' +
        'diagCCS_365t730d_prevalence_1_cnt', ...
* Comorbidity diagnoses groups (0-730-day):
    * 'prior_admiOther', 'prior_admiAcute', 'prior_spells', 'prior_asthma', 'prior_copd', 'prior_depression', 'prior_diabetes', 'prior_hypertension', 'prior_cancer', 'prior_chd', 'prior_chf'
* Charlson diagnoses groups (trigger):
    * 'diagCci_01_myocardial_freq__trigger',...

2. Load the Saved Model Outputs
Note: Make sure the following files are located at the input path
* Step_05_Features.bz2
* Step_07_Top_Features_...
* Step_07_Model_Train_model_rank_summaries_...
* Step_09_Model_...
Note: Create features extra (Run only once)
Can you write Python code for it?

# settings
feature_table = 'tcarer_features'
featureExtra_table = 'tcarer_featuresExtra'

result = readers_writers.load_mysql_procedure("tcarer_set_featuresExtra", [feature_table, featureExtra_table], schema)


I am working on the file "TCARER_summaryReports.ipynb".
The first task is:
Fit Performance
Can you write Python code for it?

o_summaries = dict()
model = charlson_training_method.predict(features_extra["train"][charlson_features_names], "train")

# short summary
o_summaries = charlson_training_method.predict_summaries(pd.Series(features_extra["train"][target_feature]), "train")
print("ROC AUC:", o_summaries['roc_auc_score_1'], "\n", o_summaries['classification_report'])
for k in o_summaries.keys():
    print(k,  o_summaries[k])


I am working on the file "TCC 02/Resultados/Abril/Abril.ipynb".
The first task is:
Análise sobre o bairro Barra do Ceará
Can you write Python code for it?

barra_do_ceara = df_abril[df_abril['BAIRRO'] == 'BARRA DO CEARA']
crime_types = barra_do_ceara[['NATUREZA DA OCORRÊNCIA']]
crime_type_total = crime_types.groupby('NATUREZA DA OCORRÊNCIA').size()
crime_type_counts = barra_do_ceara[['NATUREZA DA OCORRÊNCIA']].groupby('NATUREZA DA OCORRÊNCIA').sum()
crime_type_counts['TOTAL'] = crime_type_total
all_crime_types = crime_type_counts.sort_values(by='TOTAL', ascending=False)

all_crime_tipos_5 = all_crime_types.head(5)
all_crime_tipos_5.plot(kind='barh', figsize=(15,4), color='#3f3fff')
plt.title('Top 5 crimes na Barra do Ceará')
plt.xlabel('Número de Crimes')
plt.ylabel('Crime')
plt.tight_layout()
ax = plt.gca()
ax.xaxis.set_major_formatter(ticker.StrMethodFormatter('{x:,.0f}'))
plt.show()


I am working on the file "TCC 02/Resultados/Abril/Abril.ipynb".
The first task is:
Filtro dos 10 bairros com mais ocorrências em abril
Can you write Python code for it?

crimes_mes = df_abril.BAIRRO.value_counts()
crimes_mes_top10 = crimes_mes.head(10)

crimes_mes_top10.plot(kind='barh', figsize=(11,4), color='#3f3fff')
plt.title('Top 10 Bairros com mais crimes (Abr 2017)')
plt.xlabel('Número de ocorrências')
plt.ylabel('Bairro')
plt.tight_layout()
ax = plt.gca()
ax.xaxis.set_major_formatter(ticker.StrMethodFormatter('{x:,.0f}'))
plt.show()


I am working on the file "TCC 02/Resultados/Abril/Abril.ipynb".
The first task is:
O Bairro com o maior número de ocorrências em abril foi o Jangurussú
Vamos agora ver de forma mais detalhadas quais foram estes crimes
Can you write Python code for it?

barra_do_ceara = df_abril[df_abril['BAIRRO'] == 'JANGURUSSU']
crime_types = barra_do_ceara[['NATUREZA DA OCORRÊNCIA']]
crime_type_total = crime_types.groupby('NATUREZA DA OCORRÊNCIA').size()
crime_type_counts = barra_do_ceara[['NATUREZA DA OCORRÊNCIA']].groupby('NATUREZA DA OCORRÊNCIA').sum()
crime_type_counts['TOTAL'] = crime_type_total
all_crime_types = crime_type_counts.sort_values(by='TOTAL', ascending=False)

all_crime_tipos_5 = all_crime_types.head(5)
all_crime_tipos_5.plot(kind='barh', figsize=(15,4), color='#3f3fff')
plt.title('Top 5 crimes no Jangurussú')
plt.xlabel('Número de Crimes')
plt.ylabel('Crime')
plt.tight_layout()
ax = plt.gca()
ax.xaxis.set_major_formatter(ticker.StrMethodFormatter('{x:,.0f}'))
plt.show()


I am working on the file "TCC 02/Resultados/Abril/Abril.ipynb".
The first task is:
Todas as ocorrências criminais de abril
Can you write Python code for it?

all_crime_tipos

group_df_abril = df_abril.groupby('CLUSTER')
crimes = group_df_abril['NATUREZA DA OCORRÊNCIA'].count()

crimes.plot(kind='barh', figsize=(10,7), color='#3f3fff')
plt.title('Número de crimes por região (Abr 2017)')
plt.xlabel('Número')
plt.ylabel('Região')
plt.tight_layout()
ax = plt.gca()
ax.xaxis.set_major_formatter(ticker.StrMethodFormatter('{x:,.0f}'))
plt.show()


I am working on the file "TCC 02/Resultados/Abril/Abril.ipynb".
The first task is:
Uma análise sobre as 5 ocorrências mais comuns
Can you write Python code for it?

crime_types = regiao_1_detalhe[['NATUREZA DA OCORRÊNCIA']]
crime_type_total = crime_types.groupby('NATUREZA DA OCORRÊNCIA').size()
crime_type_counts = regiao_1_detalhe[['NATUREZA DA OCORRÊNCIA']].groupby('NATUREZA DA OCORRÊNCIA').sum()
crime_type_counts['TOTAL'] = crime_type_total
all_crime_types = crime_type_counts.sort_values(by='TOTAL', ascending=False)

crimes_top_5 = all_crime_types.head(5)
crimes_top_5.plot(kind='barh', figsize=(11,3), color='#3f3fff')
plt.title('Top 5 crimes na região 1')
plt.xlabel('Número de crimes')
plt.ylabel('Crime')
plt.tight_layout()
ax = plt.gca()
ax.xaxis.set_major_formatter(ticker.StrMethodFormatter('{x:,.0f}'))
plt.show()


I am working on the file "TCC 02/Resultados/Fevereiro/Fevereiro.ipynb".
The first task is:
Acima podemos ver que a região 3 teve o maior número de ocorrências criminais
Podemos agora ver quais são essas ocorrências de forma mais detalhada
Can you write Python code for it?

regiao_3_detalhe = df_fevereiro[df_fevereiro['CLUSTER'] == 3]
regiao_3_detalhe


I am working on the file "TCC 02/Resultados/Fevereiro/Fevereiro.ipynb".
The first task is:
As 5 regiões com mais ocorrências
Can you write Python code for it?

regioes = df_fevereiro.groupby('CLUSTER').count()
grupo_de_regioes = regioes.sort_values('NATUREZA DA OCORRÊNCIA', ascending=False)

grupo_de_regioes['TOTAL'] = grupo_de_regioes.ID
top_5_regioes_qtd = grupo_de_regioes.TOTAL.head(6)

top_5_regioes_qtd.plot(kind='barh', figsize=(10,4), color='#3f3fff')
plt.title('Top 5 regiões com mais crimes')
plt.xlabel('Número de crimes')
plt.ylabel('Região')
plt.tight_layout()
ax = plt.gca()
ax.xaxis.set_major_formatter(ticker.StrMethodFormatter('{x:,.0f}'))
plt.show()


I am working on the file "TCC 02/Resultados/Fevereiro/Fevereiro.ipynb".
The first task is:
Filtro dos 10 horários com mais ocorrências em fevereiro
Can you write Python code for it?

horas_mes = df_fevereiro.HORA.value_counts()
horas_mes_top10 = horas_mes.head(10)

horas_mes_top10.plot(kind='barh', figsize=(11,4), color='#3f3fff')
plt.title('Crimes por hora (Fev 2017)')
plt.xlabel('Número de ocorrências')
plt.ylabel('Hora do dia')
plt.tight_layout()
ax = plt.gca()
ax.xaxis.set_major_formatter(ticker.StrMethodFormatter('{x:,.0f}'))
plt.show()


I am working on the file "TCC 02/Resultados/Fevereiro/Fevereiro.ipynb".
The first task is:
Filtro dos 5 horários com mais ocorrências na região 3 (região com mais ocorrências em fevereiro)
Can you write Python code for it?

crime_hours = regiao_3_detalhe[['HORA']]
crime_hours_total = crime_hours.groupby('HORA').size()
crime_hours_counts = regiao_3_detalhe[['HORA']].groupby('HORA').sum()
crime_hours_counts['TOTAL'] = crime_hours_total
all_hours_types = crime_hours_counts.sort_values(by='TOTAL', ascending=False)

all_hours_types.head(5)

all_hours_types_top5 = all_hours_types.head(5)
all_hours_types_top5.plot(kind='barh', figsize=(11,3), color='#3f3fff')
plt.title('Top 5 crimes por hora na região 3')
plt.xlabel('Número de ocorrências')
plt.ylabel('Hora do dia')
plt.tight_layout()
ax = plt.gca()
ax.xaxis.set_major_formatter(ticker.StrMethodFormatter('{x:,.0f}'))
plt.show()


I am working on the file "TCC 02/Resultados/Fevereiro/Fevereiro.ipynb".
The first task is:
Os 5 bairros mais comuns na região 3
Can you write Python code for it?

crime_types_bairro = regiao_3_detalhe[['BAIRRO']]
crime_type_total_bairro = crime_types_bairro.groupby('BAIRRO').size()
crime_type_counts_bairro = regiao_3_detalhe[['BAIRRO']].groupby('BAIRRO').sum()
crime_type_counts_bairro['TOTAL'] = crime_type_total_bairro
all_crime_types_bairro = crime_type_counts_bairro.sort_values(by='TOTAL', ascending=False)

crimes_top_5_bairro = all_crime_types_bairro.head(5)
crimes_top_5_bairro.plot(kind='barh', figsize=(11,3), color='#3f3fff')
plt.title('Top 5 bairros na região 3')
plt.xlabel('Quantidade')
plt.ylabel('Bairro')
plt.tight_layout()
ax = plt.gca()
ax.xaxis.set_major_formatter(ticker.StrMethodFormatter('{x:,.0f}'))
plt.show()


I am working on the file "TCC 02/Resultados/Fevereiro/Fevereiro.ipynb".
The first task is:
Todas as ocorrências criminais de fevereiro
Can you write Python code for it?


I am working on the file "TCC 02/Resultados/Geral/Análise geral sobre os dados criminais.ipynb".
The first task is:
Como podemos ver no gráfico abaixo, os crimes estão bem distribuídos entre os meses, ou seja, nenhum deles teve uma quantidade de ocorrências que o faça destacar dos demais
Can you write Python code for it?

img=mpimg.imread('crimes_por_mês_per.png')
imgplot = plt.imshow(img)
plt.show()


I am working on the file "TCC 02/Resultados/Geral/Análise geral sobre os dados criminais.ipynb".
The first task is:
O mês de Março teve a maior quantidade de ocorrências
Can you write Python code for it?

img=mpimg.imread('crimes_por_mês.png')
imgplot = plt.imshow(img)
plt.show()

labels = meses
values = qtd_ocorrencias

trace = go.Pie(labels=labels, values=values)

py.iplot([trace], filename='basic_pie_chart')


I am working on the file "TCC 02/Resultados/Janeiro/Janeiro.ipynb".
The first task is:
Acima podemos ver que a região 4 teve o maior número de ocorrências criminais
Podemos agora ver quais são essas ocorrências de forma mais detalhada
Can you write Python code for it?

regiao_4_detalhe = df_janeiro[df_janeiro['CLUSTER'] == 4]
regiao_4_detalhe


I am working on the file "TCC 02/Resultados/Janeiro/Janeiro.ipynb".
The first task is:
Análise sobre o bairo Henrique Jorge
Can you write Python code for it?

henrique_jorge = df_janeiro[df_janeiro['BAIRRO'] == 'HENRIQUE JORGE']
crime_types = henrique_jorge[['NATUREZA DA OCORRÊNCIA']]
crime_type_total = crime_types.groupby('NATUREZA DA OCORRÊNCIA').size()
crime_type_counts = henrique_jorge[['NATUREZA DA OCORRÊNCIA']].groupby('NATUREZA DA OCORRÊNCIA').sum()
crime_type_counts['TOTAL'] = crime_type_total
all_crime_types = crime_type_counts.sort_values(by='TOTAL', ascending=False)

all_crime_tipos_5 = all_crime_types.head(5)
all_crime_tipos_5.plot(kind='barh', figsize=(15,4), color='#3f3fff')
plt.title('Top 5 crimes no Henrique Jorge')
plt.xlabel('Número de Crimes')
plt.ylabel('Crime')
plt.tight_layout()
ax = plt.gca()
ax.xaxis.set_major_formatter(ticker.StrMethodFormatter('{x:,.0f}'))
plt.show()


I am working on the file "TCC 02/Resultados/Janeiro/Janeiro.ipynb".
The first task is:
As 5 regiões com mais ocorrências
Can you write Python code for it?

regioes = df_janeiro.groupby('CLUSTER').count()
grupo_de_regioes = regioes.sort_values('NATUREZA DA OCORRÊNCIA', ascending=False)

grupo_de_regioes['TOTAL'] = grupo_de_regioes.ID
top_5_regioes_qtd = grupo_de_regioes.TOTAL.head(6)

top_5_regioes_qtd.plot(kind='barh', figsize=(10,4), color='#3f3fff')
plt.title('Top 5 regiões com mais crimes')
plt.xlabel('Número de crimes')
plt.ylabel('Região')
plt.tight_layout()
ax = plt.gca()
ax.xaxis.set_major_formatter(ticker.StrMethodFormatter('{x:,.0f}'))
plt.show()


I am working on the file "TCC 02/Resultados/Janeiro/Janeiro.ipynb".
The first task is:
Filtro dos 10 crimes com mais ocorrências em janeiro
Can you write Python code for it?

all_crime_tipos.head(10)

all_crime_tipos_top10 = all_crime_tipos.head(10)
all_crime_tipos_top10.plot(kind='barh', figsize=(12,6), color='#3f3fff')
plt.title('Top 10 crimes por tipo (Jan 2017)')
plt.xlabel('Número de crimes')
plt.ylabel('Crime')
plt.tight_layout()
ax = plt.gca()
ax.xaxis.set_major_formatter(ticker.StrMethodFormatter('{x:,.0f}'))
plt.show()


I am working on the file "TCC 02/Resultados/Janeiro/Janeiro.ipynb".
The first task is:
Filtro dos 10 horários com mais ocorrências em janeiro
Can you write Python code for it?

horas_mes = df_janeiro.HORA.value_counts()
horas_mes_top10 = horas_mes.head(10)

horas_mes_top10.plot(kind='barh', figsize=(11,4), color='#3f3fff')
plt.title('Crimes por hora (Jan 2017)')
plt.xlabel('Número de ocorrências')
plt.ylabel('Hora do dia')
plt.tight_layout()
ax = plt.gca()
ax.xaxis.set_major_formatter(ticker.StrMethodFormatter('{x:,.0f}'))
plt.show()


I am working on the file "TCC 02/Resultados/Janeiro/Janeiro.ipynb".
The first task is:
O Bairro com o maior número de ocorrências em janeiro foi o Barra do Ceará
Vamos agora ver de forma mais detalhadas quais foram estes crimes
Can you write Python code for it?

barra_do_ceara = df_janeiro[df_janeiro['BAIRRO'] == 'BARRA DO CEARA']
crime_types = barra_do_ceara[['NATUREZA DA OCORRÊNCIA']]
crime_type_total = crime_types.groupby('NATUREZA DA OCORRÊNCIA').size()
crime_type_counts = barra_do_ceara[['NATUREZA DA OCORRÊNCIA']].groupby('NATUREZA DA OCORRÊNCIA').sum()
crime_type_counts['TOTAL'] = crime_type_total
all_crime_types = crime_type_counts.sort_values(by='TOTAL', ascending=False)

all_crime_tipos_5 = all_crime_types.head(5)
all_crime_tipos_5.plot(kind='barh', figsize=(15,4), color='#3f3fff')
plt.title('Top 5 crimes na Barra do Ceará')
plt.xlabel('Número de Crimes')
plt.ylabel('Crime')
plt.tight_layout()
ax = plt.gca()
ax.xaxis.set_major_formatter(ticker.StrMethodFormatter('{x:,.0f}'))
plt.show()


I am working on the file "TCC 02/Resultados/Janeiro/Janeiro.ipynb".
The first task is:
Quantidade de crimes por região
Can you write Python code for it?

group_df_janeiro = df_janeiro.groupby('CLUSTER')
crimes = group_df_janeiro['NATUREZA DA OCORRÊNCIA'].count()

crimes.plot(kind='barh', figsize=(10,7), color='#3f3fff')
plt.title('Número de crimes por região (Jan 2017)')
plt.xlabel('Número')
plt.ylabel('Região')
plt.tight_layout()
ax = plt.gca()
ax.xaxis.set_major_formatter(ticker.StrMethodFormatter('{x:,.0f}'))
plt.show()


I am working on the file "TCC 02/Resultados/Janeiro/Janeiro.ipynb".
The first task is:
Todas as ocorrências criminais de janeiro
Can you write Python code for it?


I am working on the file "TCC 02/Resultados/Janeiro/Janeiro.ipynb".
The first task is:
Uma análise sobre as 5 ocorrências mais comuns
Can you write Python code for it?

crime_types = regiao_4_detalhe[['NATUREZA DA OCORRÊNCIA']]
crime_type_total = crime_types.groupby('NATUREZA DA OCORRÊNCIA').size()
crime_type_counts = regiao_4_detalhe[['NATUREZA DA OCORRÊNCIA']].groupby('NATUREZA DA OCORRÊNCIA').sum()
crime_type_counts['TOTAL'] = crime_type_total
all_crime_types = crime_type_counts.sort_values(by='TOTAL', ascending=False)

crimes_top_5 = all_crime_types.head(5)

crimes_top_5.plot(kind='barh', figsize=(11,3), color='#3f3fff')
plt.title('Top 5 crimes na região 4')
plt.xlabel('Número de crimes')
plt.ylabel('Crime')
plt.tight_layout()
ax = plt.gca()
ax.xaxis.set_major_formatter(ticker.StrMethodFormatter('{x:,.0f}'))
plt.show()


I am working on the file "TCC 02/Resultados/Maio/Maio.ipynb".
The first task is:
Acima podemos ver que a região 4 teve o maior número de ocorrências criminais
Podemos agora ver quais são essas ocorrências de forma mais detalhada
Can you write Python code for it?

regiao_4_detalhe = df_maio[df_maio['CLUSTER'] == 4]
regiao_4_detalhe


I am working on the file "TCC 02/Resultados/Maio/Maio.ipynb".
The first task is:
Análise sobre o bairro Jangurussú
Can you write Python code for it?

passare = df_maio[df_maio['BAIRRO'] == 'JANGURUSSU']
crime_types = passare[['NATUREZA DA OCORRÊNCIA']]
crime_type_total = crime_types.groupby('NATUREZA DA OCORRÊNCIA').size()
crime_type_counts = passare[['NATUREZA DA OCORRÊNCIA']].groupby('NATUREZA DA OCORRÊNCIA').sum()
crime_type_counts['TOTAL'] = crime_type_total
all_crime_types = crime_type_counts.sort_values(by='TOTAL', ascending=False)

all_crime_tipos_5 = all_crime_types.head(5)
all_crime_tipos_5.plot(kind='barh', figsize=(15,4), color='#3f3fff')
plt.title('Top 5 crimes no Jangurussú')
plt.xlabel('Número de Crimes')
plt.ylabel('Crime')
plt.tight_layout()
ax = plt.gca()
ax.xaxis.set_major_formatter(ticker.StrMethodFormatter('{x:,.0f}'))
plt.show()


I am working on the file "TCC 02/Resultados/Maio/Maio.ipynb".
The first task is:
As 5 regiões com mais ocorrências
Can you write Python code for it?

regioes = df_maio.groupby('CLUSTER').count()
grupo_de_regioes = regioes.sort_values('NATUREZA DA OCORRÊNCIA', ascending=False)

grupo_de_regioes['TOTAL'] = grupo_de_regioes.ID
top_5_regioes_qtd = grupo_de_regioes.TOTAL.head(6)

top_5_regioes_qtd.plot(kind='barh', figsize=(10,4), color='#3f3fff')
plt.title('Top 5 regiões com mais crimes')
plt.xlabel('Número de crimes')
plt.ylabel('Região')
plt.tight_layout()
ax = plt.gca()
ax.xaxis.set_major_formatter(ticker.StrMethodFormatter('{x:,.0f}'))
plt.show()


I am working on the file "TCC 02/Resultados/Maio/Maio.ipynb".
The first task is:
Filtro dos 10 horários com mais ocorrências em maio
Can you write Python code for it?

horas_mes = df_maio.HORA.value_counts()
horas_mes_top10 = horas_mes.head(10)

horas_mes_top10.plot(kind='barh', figsize=(11,4), color='#3f3fff')
plt.title('Crimes por hora (Mai 2017)')
plt.xlabel('Número de ocorrências')
plt.ylabel('Hora do dia')
plt.tight_layout()
ax = plt.gca()
ax.xaxis.set_major_formatter(ticker.StrMethodFormatter('{x:,.0f}'))
plt.show()


I am working on the file "TCC 02/Resultados/Maio/Maio.ipynb".
The first task is:
Filtro dos 5 horários com mais ocorrências na região 4 (região com mais ocorrências em maio)
Can you write Python code for it?

crime_hours = regiao_4_detalhe[['HORA']]
crime_hours_total = crime_hours.groupby('HORA').size()
crime_hours_counts = regiao_4_detalhe[['HORA']].groupby('HORA').sum()
crime_hours_counts['TOTAL'] = crime_hours_total
all_hours_types = crime_hours_counts.sort_values(by='TOTAL', ascending=False)

all_hours_types.head(5)

all_hours_types_top5 = all_hours_types.head(5)
all_hours_types_top5.plot(kind='barh', figsize=(11,3), color='#3f3fff')
plt.title('Top 5 crimes por hora na região 4')
plt.xlabel('Número de ocorrências')
plt.ylabel('Hora do dia')
plt.tight_layout()
ax = plt.gca()
ax.xaxis.set_major_formatter(ticker.StrMethodFormatter('{x:,.0f}'))
plt.show()


I am working on the file "TCC 02/Resultados/Maio/Maio.ipynb".
The first task is:
O Bairro com o maior número de ocorrências em janeiro foi o Jungurussú
Vamos agora ver de forma mais detalhadas quais foram estes crimes
Can you write Python code for it?

barra_do_ceara = df_maio[df_maio['BAIRRO'] == 'JANGURUSSU']
crime_types = barra_do_ceara[['NATUREZA DA OCORRÊNCIA']]
crime_type_total = crime_types.groupby('NATUREZA DA OCORRÊNCIA').size()
crime_type_counts = barra_do_ceara[['NATUREZA DA OCORRÊNCIA']].groupby('NATUREZA DA OCORRÊNCIA').sum()
crime_type_counts['TOTAL'] = crime_type_total
all_crime_types = crime_type_counts.sort_values(by='TOTAL', ascending=False)

all_crime_tipos_5 = all_crime_types.head(5)
all_crime_tipos_5.plot(kind='barh', figsize=(15,4), color='#3f3fff')
plt.title('Top 5 crimes no Jangurussú')
plt.xlabel('Número de Crimes')
plt.ylabel('Crime')
plt.tight_layout()
ax = plt.gca()
ax.xaxis.set_major_formatter(ticker.StrMethodFormatter('{x:,.0f}'))
plt.show()


I am working on the file "TCC 02/Resultados/Maio/Maio.ipynb".
The first task is:
Os 5 bairros mais comuns na região 4
Can you write Python code for it?

crime_types_bairro = regiao_4_detalhe[['BAIRRO']]
crime_type_total_bairro = crime_types_bairro.groupby('BAIRRO').size()
crime_type_counts_bairro = regiao_4_detalhe[['BAIRRO']].groupby('BAIRRO').sum()
crime_type_counts_bairro['TOTAL'] = crime_type_total_bairro
all_crime_types_bairro = crime_type_counts_bairro.sort_values(by='TOTAL', ascending=False)

crimes_top_5_bairro = all_crime_types_bairro.head(5)
crimes_top_5_bairro.plot(kind='barh', figsize=(11,3), color='#3f3fff')
plt.title('Top 5 bairros na região 4')
plt.xlabel('Quantidade')
plt.ylabel('Bairro')
plt.tight_layout()
ax = plt.gca()
ax.xaxis.set_major_formatter(ticker.StrMethodFormatter('{x:,.0f}'))
plt.show()


I am working on the file "TCC 02/Resultados/Maio/Maio.ipynb".
The first task is:
Quantidade de crimes por região
Can you write Python code for it?

group_df_maio = df_maio.groupby('CLUSTER')
crimes = group_df_maio['NATUREZA DA OCORRÊNCIA'].count()

crimes.plot(kind='barh', figsize=(10,7), color='#3f3fff')
plt.title('Número de crimes por região (Mai 2017)')
plt.xlabel('Número')
plt.ylabel('Região')
plt.tight_layout()
ax = plt.gca()
ax.xaxis.set_major_formatter(ticker.StrMethodFormatter('{x:,.0f}'))
plt.show()


I am working on the file "TCC 02/Resultados/Maio/Maio.ipynb".
The first task is:
Uma análise sobre as 5 ocorrências mais comuns
Can you write Python code for it?

crime_types = regiao_4_detalhe[['NATUREZA DA OCORRÊNCIA']]
crime_type_total = crime_types.groupby('NATUREZA DA OCORRÊNCIA').size()
crime_type_counts = regiao_4_detalhe[['NATUREZA DA OCORRÊNCIA']].groupby('NATUREZA DA OCORRÊNCIA').sum()
crime_type_counts['TOTAL'] = crime_type_total
all_crime_types = crime_type_counts.sort_values(by='TOTAL', ascending=False)

crimes_top_5 = all_crime_types.head(5)
crimes_top_5.plot(kind='barh', figsize=(11,3), color='#3f3fff')
plt.title('Top 5 crimes na região 4')
plt.xlabel('Número de crimes')
plt.ylabel('Crime')
plt.tight_layout()
ax = plt.gca()
ax.xaxis.set_major_formatter(ticker.StrMethodFormatter('{x:,.0f}'))
plt.show()


I am working on the file "TCC 02/Resultados/Março/Março.ipynb".
The first task is:
Filtro dos 10 bairros com mais ocorrências em março
Can you write Python code for it?

crimes_mes = df_marco.BAIRRO.value_counts()
crimes_mes_top10 = crimes_mes.head(10)

crimes_mes_top10.plot(kind='barh', figsize=(11,4), color='#3f3fff')
plt.title('Top 10 Bairros com mais crimes (Mar 2017)')
plt.xlabel('Número de ocorrências')
plt.ylabel('Bairro')
plt.tight_layout()
ax = plt.gca()
ax.xaxis.set_major_formatter(ticker.StrMethodFormatter('{x:,.0f}'))
plt.show()


I am working on the file "TCC 02/Resultados/Março/Março.ipynb".
The first task is:
O Bairro com o maior número de ocorrências em março foi Jangurussu
Vamos agora ver de forma mais detalhadas quais foram estes crimes
Can you write Python code for it?

messejana = df_marco[df_marco['BAIRRO'] == 'JANGURUSSU']
crime_types = messejana[['NATUREZA DA OCORRÊNCIA']]
crime_type_total = crime_types.groupby('NATUREZA DA OCORRÊNCIA').size()
crime_type_counts = messejana[['NATUREZA DA OCORRÊNCIA']].groupby('NATUREZA DA OCORRÊNCIA').sum()
crime_type_counts['TOTAL'] = crime_type_total
all_crime_types = crime_type_counts.sort_values(by='TOTAL', ascending=False)

all_crime_tipos_5 = all_crime_types.head(5)
all_crime_tipos_5.plot(kind='barh', figsize=(15,4), color='#3f3fff')
plt.title('Top 5 crimes no Jangurussú')
plt.xlabel('Número de Crimes')
plt.ylabel('Crime')
plt.tight_layout()
ax = plt.gca()
ax.xaxis.set_major_formatter(ticker.StrMethodFormatter('{x:,.0f}'))
plt.show()


I am working on the file "TCC 02/Resultados/Março/Março.ipynb".
The first task is:
Quantidade de crimes por região
Can you write Python code for it?

group_df_marco = df_marco.groupby('CLUSTER')
crimes = group_df_marco['NATUREZA DA OCORRÊNCIA'].count()

crimes.plot(kind='barh', figsize=(10,7), color='#3f3fff')
plt.title('Número de crimes por região (Mar 2017)')
plt.xlabel('Número')
plt.ylabel('Região')
plt.tight_layout()
ax = plt.gca()
ax.xaxis.set_major_formatter(ticker.StrMethodFormatter('{x:,.0f}'))
plt.show()


I am working on the file "TCC 02/Resultados/Março/Março.ipynb".
The first task is:
Todas as ocorrências criminais de março
Can you write Python code for it?


I am working on the file "TCC 02/Resultados/Março/Março.ipynb".
The first task is:
Uma análise sobre as 5 ocorrências mais comuns
Can you write Python code for it?


I am working on the file "TCGA_OV_exp-cn_jointplot.ipynb".
The first task is:
Jointplot of all data points as scatter
Used under hex plot to to show variation
Can you write Python code for it?

Fig1C_dist_reg=sns.jointplot("CNV", "RNA", data=all_data_df, kind='scatter', color='k')

overplot_both_df=all_data_df[((all_data_df.CNV < -0.5) | (all_data_df.CNV > 0.5)) & ((all_data_df.RNA < -1.0) | (all_data_df.RNA > 1.0))]
overplot_both_df=sns.jointplot("CNV", "RNA", data=overplot_both_df, kind='hex', color='g')

overplot_normal_copy_df=all_data_df[((all_data_df.CNV > -0.5) & (all_data_df.CNV < 0.5)) & ((all_data_df.RNA < -2.0) | (all_data_df.RNA > 2.0))]
overplot_normal_copy_df=sns.jointplot("CNV", "RNA", data=overplot_normal_copy_df, kind='hex', color='g')


I am working on the file "TCGA_OV_exp-cn_jointplot.ipynb".
The first task is:
Using Rmagic to query Biomart in R and return dataframe with genomic coordinate data.
Can you write Python code for it?

%%R

library(biomaRt)
library(plyr)

ensembl = useEnsembl(biomart="ensembl", dataset="hsapiens_gene_ensembl")

#=======================================================================#

# define biomart object
mart <- useMart(biomart = "ensembl", dataset = "hsapiens_gene_ensembl")

# query biomart
get_chr_genes <- function(chr_num, mart_server){
  subset(
    unique(
      getBM(
        attributes=c(
          'ensembl_gene_id',
          'ensembl_transcript_id',
          'hgnc_symbol',
          'chromosome_name',
          'start_position',
          'end_position'
        ),
        filters = 'chromosome_name', values =chr_num, mart = mart_server
      )[-c(1:2)]
    ),
    hgnc_symbol != ""
  )
}

N <- 23
chr_L <- vector("list", N)

for( i in 1:21) {
  chr <- get_chr_genes(i, ensembl)
  chr_L[[i]] <- chr
}

chr_X <- get_chr_genes("X", ensembl)
chr_Y <- get_chr_genes("Y", ensembl)

chr_L[[22]] <- chr_X
chr_L[[23]] <- chr_Y

#=======================================================================#

chr_df <- ldply(chr_L, data.frame)

%Rpull chr_df

CNV_samples, CNV_probes = accessXenaData(hub, CNV_dataset)
RNA_samples, RNA_probes = accessXenaData(hub, RNA_dataset)

CNV_L_df = matchEnsemblData(chr_df, CNV_probes)
RNA_L_df = matchEnsemblData(chr_df, RNA_probes)

CNV_df = analysisMultiIndexDataFrame(CNV_L_df, 'CNV', CNV_samples, CNV_dataset, CNV_probes)
RNA_df = analysisMultiIndexDataFrame(RNA_L_df, 'RNA', RNA_samples, RNA_dataset, RNA_probes)

all_data_df = pd.merge(CNV_df, RNA_df, left_index=True, right_index=True)
all_data_df.index.names=['samples']
all_data_df.columns=all_data_df.columns.droplevel([u'chromosome_name', u'start_position', u'end_position'])
all_data_df=all_data_df.T.unstack([u'hgnc_symbol']).T
all_data_df=all_data_df.dropna()

sns.axes_style('white')


I am working on the file "TD_RNN.ipynb".
The first task is:
Above, the left image shows a typical distribution of noise marked by black rectangulas.  On the right, the three images have been concatenated. From the left to the right, the first image corresponds to noise shown on the left, and the next two images show a linear track, looking as a staircase consisting from small black rectangular boxes going down through the middle and the rightmost images.
For purposes of LSTM processing, in this work the images are processed in the sequences of ten. Below, I show only the first three to stress the 3D arrangement of data.
Can you write Python code for it?

fig = plt.figure(figsize=(30,20));
cmap = colors.ListedColormap(['white', 'black']);
#rect = l,b,w,h
rect1 = 0.2, 0.1, 0.1, 0.2
rect2 = 0.22, 0.11, 0.1, 0.2
rect3 = 0.25, 0.12, 0.1, 0.2

ax1= fig.add_axes(rect3);
im = test_X[start+2,:,:].copy()
ax1.imshow(im.transpose(),origin='lower', cmap=cmap, interpolation = 'none',aspect='auto');
ax2= fig.add_axes(rect2);
im = test_X[start+1,:,:].copy()
ax2.imshow(im.transpose(),origin='lower', cmap=cmap, interpolation = 'none',aspect='auto');
ax3= fig.add_axes(rect1);
im = test_X[start,:,:].copy()
ax3.imshow(im.transpose(),origin='lower', cmap=cmap, interpolation = 'none',aspect='auto');
ax3.set_xlabel('non-dim time',fontproperties=font1);
ax3.set_ylabel('non-dim range',fontproperties=font1);


I am working on the file "TD_RNN.ipynb".
The first task is:
Finally, I estimated the probability of detection (Pd) and the probability of false alarms (Pfa), applied to the test, and to the training data. This metrics is often used for receiver operating characterisitc estimation.
Can you write Python code for it?

utils.roc_dat(Y_estim_test, test_y_true, 0.5)

utils.roc_dat(Y_estim_train, train_y_true, 0.5)


I am working on the file "TD_RNN.ipynb".
The first task is:
Next, to show the structure of the data, we look into a few example images.
Examples
Can you write Python code for it?

font0 = FontProperties();
font1 = font0.copy();
font1.set_size('xx-large');
font1.set_weight('bold');

fig = plt.figure(figsize=(30,20));
cmap = colors.ListedColormap(['white', 'black']);
#rect = l,b,w,h
rect1 = 0.2, 0.1, 0.1, 0.2
rect2 = 0.4, 0.1, 0.3, 0.2
start = 2*3
ax1= fig.add_axes(rect1);
ax2 = fig.add_axes(rect2);
im = test_X[start,:,:].copy()
ax1.imshow(im.transpose(),origin='lower', cmap=cmap, interpolation = 'none',aspect='auto');
ax1.set_title('Example of noise image',fontproperties=font1);
ax1.set_xlabel('non-dim time',fontproperties=font1);
ax1.set_ylabel('non-dim range',fontproperties=font1);
ims = test_X[start:start+3,:,:].copy()
im = np.reshape(ims, (ims.shape[0]*ims.shape[1],ims.shape[2]));
ax2.imshow(im.transpose(),origin='lower', cmap=cmap, interpolation = 'none',aspect='auto');
ax2.set_title('Example of three stacked images: noise, noise+track, noise+track',fontproperties=font1);
ax2.set_xlabel('non-dim time',fontproperties=font1);
ax2.set_ylabel('non-dim range',fontproperties=font1);
ax2.set_xlim(0,30);
ax2.set_ylim(0.30);
for i in range(0,30,10):
    ax2.plot([i, i],[0, 30],'r-');


I am working on the file "TD_RNN.ipynb".
The first task is:
Now we use the trained model to predict labels of the input images used for training as well as for testing
Can you write Python code for it?

Y_estim_train = model_lin.predict(train_X, batch_size = batch_size)
Y_estim_test = model_lin.predict(test_X, batch_size = batch_size)
print(Y_estim_train.shape, Y_estim_test.shape)


I am working on the file "TD_RNN.ipynb".
The first task is:
Results
Below is shown a subset of the training and test results. On top of each concatenated image, the distribution of averaged labels is plotted. The blue and the red colors show respectively the true and the estimated labels. Note that both label types match perfectly for the training subset shown below.
Can you write Python code for it?

row1_train = 60
row2_train = 90
row1_test = 100
row2_test = 150
dxn = 10

utils.plot_results(test_y_true, Y_estim_test, train_X, train_y_true, Y_estim_train, test_X,
                 dxn, row1_train, row2_train, row1_test, row2_test, N_plots = 7)


I am working on the file "TD_RNN.ipynb".
The first task is:
Since the predicted output is real valued, using the threshold value of 0.5, I categorized the output into two categories: the ones corresponding to the track presence (y=1), and the ones corresponding to the track absence (y=0). Below y corresponds either to the Y_estim_test or to the Y_estim_train.
Can you write Python code for it?

Y_estim_test[Y_estim_test < 0.5]=0
Y_estim_test[Y_estim_test >= 0.5]=1
Y_estim_train[Y_estim_train < 0.5]=0
Y_estim_train[Y_estim_train >= 0.5]=1


I am working on the file "TD_RNN.ipynb".
The first task is:
To estimate one label per image,  we need to average over 10 labels that were generated for 10 time steps of  each image.
Can you write Python code for it?


I am working on the file "TEM-image-simple/TEM-Tutorial-Setup.ipynb".
The first task is:
Host and user information
Can you write Python code for it?

user='admin'
pswd='admin'
host='http://127.0.0.1:8000'
template_name='TEM-Tutorial'


I am working on the file "TEM-image-simple/TEM-Tutorial-Setup.ipynb".
The first task is:
List of file prefixes for micrograph images and XML metadata
Can you write Python code for it?


I am working on the file "TESS 2015 - SunPy.ipynb".
The first task is:
Compares well to the official max from the HEK
Can you write Python code for it?

str(tmax)

goes.peek()
plt.axhline(goes.data['xrsb'].max())
plt.axvline(goes.data['xrsb'].idxmax())


I am working on the file "TESS 2015 - SunPy.ipynb".
The first task is:
Composite Maps
Let's plot the two channels we downloaded from AIA together! Composite map is the way to do this. Let's check out the other map we got.
Can you write Python code for it?

aia131 = Map(f[0])
aia131

smap131 = aia131.submap(xrange, yrange)
smap131.peek()

norm = colors.Normalize(0, 4000)
smap131.plot(norm=norm)
plt.colorbar()

smap171 = smap

compmap = Map(smap171, smap131, composite=True)

levels = np.arange(0,100,5)
print(levels)

compmap.set_levels(1, levels, percent=True)
compmap.set_mpl_color_normalizer(0, norm)
compmap.set_colors(1, plt.cm.Reds)

compmap.plot(norm=norm)
plt.show()


I am working on the file "TESS 2015 - SunPy.ipynb".
The first task is:
For SunPy the top level name-space is kept clean. Importing SunPy does not give you access to much. You need to import specific names. SciPy is the same.
So, the place to start here is with the core SunPy data object. It is called Map.
Can you write Python code for it?

from sunpy.map import Map

aia = Map(f[1])

aia


I am working on the file "TESS 2015 - SunPy.ipynb".
The first task is:
It works, now lets see how many results we have!
Can you write Python code for it?


I am working on the file "TESS 2015 - SunPy.ipynb".
The first task is:
Let's also grab another wavelength for later.
Can you write Python code for it?

recs = client.query(vso.attrs.Time('2014/01/01 18:52:08', '2014/01/01 18:52:15'),
                    vso.attrs.Instrument('AIA'),
                    vso.attrs.Wave(94,171))

recs.num_records()


I am working on the file "TESS 2015 - SunPy.ipynb".
The first task is:
Lightcurves!
Let's look at the GOES curve for this event.
First some time manipulation!
Can you write Python code for it?

from sunpy.time import TimeRange, parse_time
from datetime import timedelta

tmax = parse_time(result[0].get('event_peaktime'))

tmax

tr = TimeRange(tmax - timedelta(minutes=30), tmax + timedelta(minutes=30))

tr

from sunpy.lightcurve import GOESLightCurve
goes = GOESLightCurve.create(tr)

goes.peek()


I am working on the file "TESS 2015 - SunPy.ipynb".
The first task is:
Maps contain both the image data and the metadata associated with the image, this metadata currently does not deviate much from the standard FITS WCS keywords, but presented in a instrument-independent manner.
Can you write Python code for it?


I am working on the file "TESS 2015 - SunPy.ipynb".
The first task is:
Meta data is also stored in a standard place
Can you write Python code for it?


I am working on the file "TESS 2015 - SunPy.ipynb".
The first task is:
Solar Images in SunPy
SunPy has a Map type that supports 2D images, it makes it simple to read data in from any filetype supported in sunpy.io which is currently FITS, JPEG2000 and ANA files. You can also create maps from any (data, metadata) pair.
Let's download an AIA image of this flare from the vso
Can you write Python code for it?

from sunpy.net import vso
client=vso.VSOClient()


I am working on the file "TESS 2015 - SunPy.ipynb".
The first task is:
Some other topics...
Solar Constants
Can you write Python code for it?

from sunpy.sun import constants as solar_constants

solar_constants.mass

print(solar_constants.mass)

(solar_constants.mass/solar_constants.volume).cgs

solar_constants.volume + solar_constants.density


I am working on the file "TESS 2015 - SunPy.ipynb".
The first task is:
SunPy version (stable) 0.5
Let's study a flare!
Searching for events in the HEK
Can you write Python code for it?

from sunpy.net import hek
client = hek.HEKClient()
tstart, tend = '2014/01/01 00:00:00', '2014/01/02 00:00:00'
result = client.query(hek.attrs.Time(tstart, tend),
                      hek.attrs.EventType('FL'),
                      hek.attrs.FRM.Name=='SSW Latest Events')
len(result)

result[0]

for res in result:
    print(res.get('fl_goescls'))

result = client.query(hek.attrs.Time(tstart, tend),
                      hek.attrs.EventType('FL'),
                      hek.attrs.FRM.Name=='SSW Latest Events',
                      hek.attrs.FL.GOESCls>'M')

len(result)

result


I am working on the file "TESS 2015 - SunPy.ipynb".
The first task is:
That's way too many!
So that is every image that SDO/AIA had on that day. Let us reduce that amount.
To do this, we will limit the time search for a specify a wavelength.
Can you write Python code for it?

recs = client.query(vso.attrs.Time('2014/01/01 18:52:08', '2014/01/01 18:52:15'),
                    vso.attrs.Instrument('AIA'),
                    vso.attrs.Wave(171,171))

recs.num_records()

recs.show()


I am working on the file "TESS 2015 - SunPy.ipynb".
The first task is:
The data (stored in a numpy array)
Can you write Python code for it?


I am working on the file "TESS 2015 - SunPy.ipynb".
The first task is:
The original metadata (stored in a dictionary)
Can you write Python code for it?

aia.meta

aia.meta.keys()

aia.meta.get('rsun_obs')


I am working on the file "TESS 2015 - SunPy.ipynb".
The first task is:
Then do a search.
Can you write Python code for it?

recs = client.query(vso.attrs.Time(tr), vso.attrs.Instrument('AIA'))


I am working on the file "TESS 2015 - SunPy.ipynb".
The first task is:
This is a pandas dataframe! Provides lots of additional functionality. For example
Can you write Python code for it?

print('The max flux is {flux:2.5f} at {time}'.format(flux=goes.data['xrsb'].max(), time=goes.data['xrsb'].idxmax()))


I am working on the file "TESS 2015 - SunPy.ipynb".
The first task is:
and where it occurred
Can you write Python code for it?


I am working on the file "TF more basic tutuorials.ipynb".
The first task is:
2.
Can you write Python code for it?

import numpy as np
x=np.random.rand(10)
y=tf.Variable(5*x**2,name='y')
model = tf.initialize_all_variables()

with tf.Session() as session:
    session.run(model)
    print(session.run(y))


I am working on the file "TF more basic tutuorials.ipynb".
The first task is:
Exercise
1.
Can you write Python code for it?

import tensorflow as tf
x = tf.constant([35, 40, 45], name='x')
y = tf.Variable(x + 5, name='y')

model = tf.initialize_all_variables()

with tf.Session() as session:
	session.run(model)
	print(session.run(y))


I am working on the file "TF more basic tutuorials.ipynb".
The first task is:
tensorboard
Can you write Python code for it?

import tensorflow as tf

x = tf.constant(35, name='x')
print(x)
y = tf.Variable(x + 5, name='y')

with tf.Session() as session:
    merged = tf.merge_all_summaries()
    writer = tf.train.SummaryWriter("", session.graph)
    model = tf.initialize_all_variables()
    session.run(model)
    print(session.run(y))

import matplotlib.image as mpimg

# First, load the image
filename = "MarshOrchid.jpg"
image = mpimg.imread(filename)

# Print out its shape
print(image.shape)

import matplotlib.pyplot as plt
plt.imshow(image)
plt.show()

import tensorflow as tf
import matplotlib.image as mpimg
import matplotlib.pyplot as plt

# First, load the image again
filename = "MarshOrchid.jpg"
image = mpimg.imread(filename)

# Create a TensorFlow Variable
x = tf.Variable(image, name='x')

model = tf.initialize_all_variables()

with tf.Session() as session:
    x = tf.transpose(x, perm=[1, 0, 2])
    session.run(model)
    result = session.run(x)

plt.imshow(result)
plt.show()


I am working on the file "TF-Coder_Colab.ipynb".
The first task is:

Usage Tips
General

If TF-Coder finds a solution, it is guaranteed that the solution produces
  the example output when run on the example inputs. However, it is not
  guaranteed that the solution generalizes in the way you intend! Please
  carefully review solutions produced by TF-Coder before using them in your real
  project.

TF-Coder will often produce a solution that uses hardcoded constants for
  shapes or lengths, e.g., tf.reshape(to_flatten, (6,)) in order to flatten an
  input tensor with shape (2, 3). You may need to manually change these
  constants to improve the generality of the solution, e.g., replacing 6 with
  -1 in this case. Use the shape attribute to obtain dimension lengths of
  input tensors, e.g., to_flatten.shape[0] would be 2.

If you want to play with TensorFlow in Colab (e.g., to understand how a
  TF-Coder solution works or to test your own solution):

The TF-Coder Colab already imports TensorFlow 2 and Numpy, for your
    convenience.

Use tf.constant to create a tensor from the list format:
    ```
    >>> tf.constant([[13, 22], [17, 5]])
    <tf.Tensor: id=1, shape=(2, 2), dtype=int32, numpy=
    array([[13, 22],
           [17,  5]], dtype=int32)>

tf.constant(12.3)
<tf.Tensor: id=2, shape=(), dtype=float32, numpy=12.3>
```
  * A Colab notebook can only have one cell running at a time. If you want to
experiment with TensorFlow code while TF-Coder is running, consider doing so
in a separate Python shell.

TF-Coder's running time is exponential in the complexity of the solution.
  Simplifying the problem, or breaking it down into multiple steps, can help
  TF-Coder find solutions quickly. For instance, if you know that a reshape,
  transpose, cast, or other similar operation should be applied to an input or
  as the last operation to produce the output, consider applying that operation
  manually to the input-output example, to help TF-Coder focus on the more
  difficult parts.

Input-Output Example
Creating a good input-output example is crucial for TF-Coder...
Can you write Python code for it?

# Real task encountered by a Googler.
inputs = {
    'tensor': [[0, 1, 0, 0],
               [0, 1, 1, 0],
               [1, 1, 1, 1]],
}
output = [[0.0, 1.0, 0.0, 0.0],
          [0.0, 0.5, 0.5, 0.0],
          [0.25, 0.25, 0.25, 0.25]]
constants = []
description = 'normalize the rows of a tensor'

# Real task encountered by a Googler.
inputs = {
    'elements': [0, 0, 0, 1, 3, 3],
}
output = [[0, 0], [0, 1], [0, 2], [1, 0], [3, 0], [3, 1]]
constants = []
description = 'pair each element with a counter'

# Real task encountered by a Googler.
inputs = {
    'sparse': tf.SparseTensor(
        indices=[[0, 0, 0], [0, 1, 1], [1, 1, 1], [1, 1, 2]],
        values=[1., 1., 1., 1.],
        dense_shape=[2, 2, 800]),
}
output = tf.SparseTensor(
    indices=[[0, 0, 0], [0, 1, 1]],
    values=[1., 1.],
    dense_shape=[1, 2, 800])
constants = []
description = 'slice index 0 of the first dimension of a SparseTensor'

# Real task encountered by a Googler.
inputs = {
    'lengths': [3, 4, 2, 1],
}
output = [[1, 1, 1, 0, 0],
          [1, 1, 1, 1, 0],
          [1, 1, 0, 0, 0],
          [1, 0, 0, 0, 0]]
constants = [5]
description = 'create a mask for sequences of the given lengths'

# Real task encountered by a Googler.
inputs = {
    'segments': [ 1,  1,  1,  0,  0,  2],
    'data':     [10, 20, 30, 14, 15, 26],
}
output = [14, 15, 10, 20, 30, 26]
constants = []
description = 'sort the segments'

# Adapted from https://stackoverflow.com/questions/53054668
inputs = {
    'values': [37, 42, 42, 37, 28, 15, 42, 15],
}
output = [0, 1, 1, 0, 2, 3, 1, 3]
constants = []
description = 'group items by value and get the group indices'

# Adapted from https://stackoverflow.com/questions/47816231
inputs = {
    'vector': [3, 5, 0, 2, 3, 3, 0],
}
output = [[1., 0., 0., 0., 1., 1., 0.],
          [0., 1., 0., 0., 0., 0., 0.],
          [0., 0., 1., 0., 0., 0., 1.],
          [0., 0., 0., 1., 0., 0., 0.],
          [1., 0., 0., 0., 1., 1., 0.],
          [1., 0., 0., 0., 1., 1., 0.],
          [0., 0., 1., 0., 0., 0., 1.]]
constants...


I am working on the file "TF-Coder_Colab.ipynb".
The first task is:
Supported Operations
Can you write Python code for it?

# Run this cell to print all supported operations.
colab_interface.print_supported_operations()


I am working on the file "TF/DummyNet.ipynb".
The first task is:
Inspect the protobuf containing the model's architecture and logic
Can you write Python code for it?

from tensorflow.core.framework import graph_pb2

# -- read in the graph
f = open("models/graph.pb", "rb")
graph_def = graph_pb2.GraphDef()
graph_def.ParseFromString(f.read())

import tensorflow as tf
# -- actually import the graph described by graph_def
tf.import_graph_def(graph_def, name = '')

for node in graph_def.node:
    print node.name


I am working on the file "TF/DummyNet.ipynb".
The first task is:
Pick an example to test if load.cc works
Can you write Python code for it?

# -- inputs
X_test[0]

# -- predicted output (using Keras)
yhat[0]


I am working on the file "TFIDF Cosine.ipynb".
The first task is:
To calculate the normalized Term Frequency – Inverse Document Frequency for every president
Can you write Python code for it?

import re
refs_pat = '^[a-z][a-z\'-]+[a-z]$'
refs_prog = re.compile(refs_pat)
NAMEWORDS = set()

# l2norm() function to divide a person's TF value by the l2norm of the term vector
def l2norm(vec):
    return float(math.sqrt(sum(map(lambda (term, c): c**2, vec))))

# get_terms() to perform all of the data cleaning
def get_terms(s):
    # Casing and short words
    # We can deal with these by lower-casing all of the terms and filtering out the short terms.
    s = s.lower()
    lines = filter(lambda line: not line.strip().startswith(">"), s.split('\n'))
    arr = '\n'.join(lines).split()
    terms = []
    for term in arr:
        if re.match(refs_pat, term) != None:
            terms.append(term)
    terms = map(lambda term: term.replace("'s",'').replace("'", '').replace(".", "").replace(",", ""), terms)
    terms = filter(lambda term: len(term) > 3, terms)
    # Stop Words
    # The text_util module defines a variable STOPWORDS that contains a list of common english
    # stop words in lower case. We can filter out terms that are found in in this list.
    from text_util import STOPWORDS
    terms = filter(lambda term: term not in STOPWORDS, terms)
    # Remove names from the terms
    terms = filter(lambda term: term not in NAMEWORDS, terms)
    terms = filter(stemmer.stem, terms)
    return terms

# To calculate term frequency
folder_tf = defaultdict(Counter)

for e in TextWalker('sotu'):
    terms_in_text = get_terms(e['text'])
    folder_tf[e['folder']].update(terms_in_text)

# To calculate inverse document frequency
terms_per_folder = defaultdict(set)
ntexts = 0
for e in TextWalker('sotu'):
    terms_in_text = get_terms(e['text'])
    # this collects all of the terms in each folder
    terms_per_folder[e['folder']].update(terms_in_text)

# Each iteration retrieves the terms for a given folder, and adds them all to the counter.
allterms = Counter()
for folder, terms in terms_per_folder.iteritems():
    # this will increment the counter value for each term in `terms`
    allterms.update(terms)

# To...


I am working on the file "TFIDF.ipynb".
The first task is:
Calcularing the TF-IDF
Can you write Python code for it?

# Calculating the sum of the power of tf-idf of tags in tag vector of each movie and store it in vectorLength tag
itemVectors = {}

for movie,tagVector in itemCounts.items():
    vectLen= 0.0
    for tag,count in tagVector.items():
            # TF-IDF = TF  * IDF. count = TF ,  docFreq[tag] = IDF
            tfidf= count * docFreq[tag]

            vectLen += math.pow(tfidf, 2)

    vectLen = math.sqrt(vectLen)

    iv ={}
    for tag,count in tagVector.items():
            # Calculating the normalized tf-idf
            iv[tag] = count * docFreq[tag] / vectLen

    itemVectors[movie] = iv
print(len(docFreq))

def getItemVector(docID):
    df=pd.DataFrame.from_dict(itemVectors, orient='index')

    d=df.loc['2231']
    d=d[d.notnull()]

    d=pd.DataFrame(d)
    d.columns=['value']

    d.sort_values('value', axis=0,ascending=[False], inplace=True)
    print(d)

#df


I am working on the file "TFIDF.ipynb".
The first task is:
Calculating the IDF
Can you write Python code for it?

# Calculating the final IDF values
docFreq ={}
for tag,count in docCount.items():
    docFreq[tag]= logN - math.log(count)

# Testing Item 4
print('item 4==========================')
print( docFreq['CLV'])
print( docFreq['characters'])
print( docFreq['chick flick'])
print( docFreq['revenge'])


I am working on the file "TFIDF.ipynb".
The first task is:
Calculating the TF for Movie Tags
Can you write Python code for it?

# item Vectors as a dictionary(like hashMaps in Java) : { movie1 :{ tag1 : count1 , tag2:count2 ,...} , movieID: ...}
itemCounts={}

# Document Vectro as a dictionary : { tag1:count1 , tag2:count2 ,...}
docCount= {}

# Calculating the document frequency
# Stroing the length of the vectors in the last field as vectorLength

for movie,tag in items:
    if not tag in docCount:
        docCount[tag]=0.0

    add=0
    if not movie in itemCounts:
        # tagVector = Tag Vector
        tagVector={}
        tagVector[tag]=1.0

        # Initial Length is 0. Will be computed later
        #  tagVector["vectorLength"] =0.0

        # if movie not exists in the dict before add a new one with
        itemCounts[movie]={}

        # It is new tag for this movie so its count in the docFreq should be incremented
        add=1

    else:
        tagVector=itemCounts[movie]

        if not tag in itemCounts[movie]:
            tagVector[tag]=1.0

            # It is new tag for this movie so its count in the docFreq should be incremented
            add=1

        else:
            tagVector[tag] = tagVector[tag]+1

    # List of tag vectors for this movie gets u_profilesdated
    itemCounts[movie] =tagVector

    # The document frequency for this tag in the current movie.
    docCount[tag] +=add

logN = math.log(len(itemCounts))

print (len(itemCounts))
print(logN)


I am working on the file "TFIDF.ipynb".
The first task is:
Test the Results
Can you write Python code for it?


I am working on the file "TFLearn_sentiment/TFLearn_Sentiment_Analysis.ipynb".
The first task is:
Counting word frequency
To start off we'll need to count how often each word appears in the data. We'll use this count to create a vocabulary we'll use to encode the review data. This resulting count is known as a bag of words. We'll use it to select our vocabulary and build the word vectors. You should have seen how to do this in Andrew's lesson. Try to implement it here using the Counter class.

Exercise: Create the bag of words from the reviews data and assign it to total_counts. The reviews are stores in the reviews Pandas DataFrame. If you want the reviews as a Numpy array, use reviews.values. You can iterate through the rows in the DataFrame with for idx, row in reviews.iterrows(): (documentation). When you break up the reviews into words, use .split(' ') instead of .split() so your results match ours.
Can you write Python code for it?

from collections import Counter

total_counts = Counter()

for _, row in reviews.iterrows():
    total_counts.update(row[0].split(' '))

print("Total words in data set: ", len(total_counts))
total_counts.most_common()


I am working on the file "TFLearn_sentiment/TFLearn_Sentiment_Analysis.ipynb".
The first task is:
If you do this right, the following code should return
```
text_to_vector('The tea is for a party to celebrate '
               'the movie so she has no time for a cake')[:65]
array([0, 1, 0, 0, 2, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 1, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0])
```
Can you write Python code for it?


I am working on the file "TFLearn_sentiment/TFLearn_Sentiment_Analysis.ipynb".
The first task is:
Intializing the model
Next we need to call the build_model() function to actually build the model. In my solution I haven't included any arguments to the function, but you can add arguments so you can change parameters in the model if you want.

Note: You might get a bunch of warnings here. TFLearn uses a lot of deprecated code in TensorFlow. Hopefully it gets updated to the new TensorFlow version soon.
Can you write Python code for it?


I am working on the file "TFLearn_sentiment/TFLearn_Sentiment_Analysis.ipynb".
The first task is:
Let's keep the first 10000 most frequent words. As Andrew noted, most of the words in the vocabulary are rarely used so they will have little effect on our predictions. Below, we'll sort vocab by the count value and keep the 10000 most frequent words.
Can you write Python code for it?

vocab = sorted(total_counts, key=total_counts.get, reverse=True)[:10000]
#print(vocab[:60])
#total_counts.most_common()

def pretty_print_review_and_label(i):
    print(labels[0][i] + "\t:\t" + reviews[0][i][:80] + "...")
pretty_print_review_and_label(12816)


I am working on the file "TFLearn_sentiment/TFLearn_Sentiment_Analysis.ipynb".
The first task is:
Preparing the data
Following along with Andrew, our goal here is to convert our reviews into word vectors. The word vectors will have elements representing words in the total vocabulary. If the second position represents the word 'the', for each review we'll count up the number of times 'the' appears in the text and set the second position to that count. I'll show you examples as we build the input data from the reviews data. Check out Andrew's notebook and video for more about this.
Read the data
Use the pandas library to read the reviews and postive/negative labels from comma-separated files. The data we're using has already been preprocessed a bit and we know it uses only lower case characters. If we were working from raw data, where we didn't know it was all lower case, we would want to add a step here to convert it. That's so we treat different variations of the same word, like The, the, and THE, all the same way.
Can you write Python code for it?

reviews = pd.read_csv('reviews.txt', header=None)
labels = pd.read_csv('labels.txt', header=None)


I am working on the file "TFLearn_sentiment/TFLearn_Sentiment_Analysis.ipynb".
The first task is:
Testing
After you're satisified with your hyperparameters, you can run the network on the test set to measure its performance. Remember, only do this after finalizing the hyperparameters.
Can you write Python code for it?

predictions = (np.array(model.predict(testX))[:,0] >= 0.5).astype(np.int_)
test_accuracy = np.mean(predictions == testY[:,0], axis=0)
print("Test accuracy: ", test_accuracy)


I am working on the file "TFLearn_sentiment/TFLearn_Sentiment_Analysis.ipynb".
The first task is:
The last word in our vocabulary shows up in 30 reviews out of 25000. I think it's fair to say this is a tiny proportion of reviews. We are probably fine with this number of words.
Note: When you run, you may see a different word from the one shown above, but it will also have the value 30. That's because there are many words tied for that number of counts, and the Counter class does not guarantee which one will be returned in the case of a tie.
Now for each review in the data, we'll make a word vector. First we need to make a mapping of word to index, pretty easy to do with a dictionary comprehension.

Exercise: Create a dictionary called word2idx that maps each word in the vocabulary to an index. The first word in vocab has index 0, the second word has index 1, and so on.
Can you write Python code for it?

word2idx = {word : i for i, word in enumerate(vocab)}


I am working on the file "TFLearn_sentiment/TFLearn_Sentiment_Analysis.ipynb".
The first task is:
Train, Validation, Test sets
Now that we have the word_vectors, we're ready to split our data into train, validation, and test sets. Remember that we train on the train data, use the validation data to set the hyperparameters, and at the very end measure the network performance on the test data. Here we're using the function to_categorical from TFLearn to reshape the target data so that we'll have two output units and can classify with a softmax activation function. We actually won't be creating the validation set here, TFLearn will do that for us later.
Can you write Python code for it?

Y = (labels=='positive').astype(np.int_)
records = len(labels)

shuffle = np.arange(records)
np.random.shuffle(shuffle)
test_fraction = 0.9

train_split, test_split = shuffle[:int(records*test_fraction)], shuffle[int(records*test_fraction):]
trainX = word_vectors[train_split,:]
trainY = to_categorical(Y[0][train_split], 2)
testX, testY = word_vectors[test_split,:], to_categorical(Y[0][test_split], 2)

print(trainX)
print(trainY)
print(testX)
print(testY)


I am working on the file "TFLearn_sentiment/TFLearn_Sentiment_Analysis.ipynb".
The first task is:
Training the network
Now that we've constructed the network, saved as the variable model, we can fit it to the data. Here we use the model.fit method. You pass in the training features trainX and the training targets trainY. Below I set validation_set=0.1 which reserves 10% of the data set as the validation set. You can also set the batch size and number of epochs with the batch_size and n_epoch keywords, respectively. Below is the code to fit our the network to our word vectors.
You can rerun model.fit to train the network further if you think you can increase the validation accuracy. Remember, all hyperparameter adjustments must be done using the validation set. Only use the test set after you're completely done training the network.
Can you write Python code for it?

# Training
model.fit(trainX, trainY, validation_set=0.2, show_metric=True, batch_size=128, n_epoch= 100)


I am working on the file "TFLearn_sentiment/TFLearn_Sentiment_Analysis.ipynb".
The first task is:
What's the last word in our vocabulary? We can use this to judge if 10000 is too few. If the last word is pretty common, we probably need to keep more words.
Can you write Python code for it?

print(vocab[-1], ': ', total_counts[vocab[-1]])


I am working on the file "TFT_LCD.ipynb".
The first task is:
Using the PIL Drawing class
The Python Imaging library makes it easy to draw simple shapes.
Can you write Python code for it?

draw.rectangle((10, 90, 110, 160), outline=(255,255,0), fill=(0,0,255))
# Write buffer to the screen
disp.display()


I am working on the file "TFT_LCD.ipynb".
The first task is:
Using the TFT Class
The TFT LCD userspace library can run without requiring a kernel module. Using it in this mode will be somewhat slower, suitable for drawing images one-at-a-time.
Can you write Python code for it?

# Create TFT LCD display class.
disp = TFT.ILI9341(DC, rst=RST, spi=SPI.SpiDev(SPI_PORT, SPI_DEVICE, max_speed_hz=1000000))

# Initialize display.
disp.begin()

# Clear the display to a red background.
# Can pass any tuple of red, green, blue values (from 0 to 255 each).
disp.clear((255, 0, 0))

# Get a PIL Draw object to start drawing on the display buffer.
draw = disp.draw()


I am working on the file "TF_1_x/SelfNormalizingNetworks_MLP_MNIST.ipynb".
The first task is:
(1) Definition of scaled exponential linear units (SELUs)
Can you write Python code for it?


I am working on the file "TF_1_x/SelfNormalizingNetworks_MLP_MNIST.ipynb".
The first task is:
(2) Definition of dropout variant for SNNs
Can you write Python code for it?

def dropout_selu(x, rate, alpha= -1.7580993408473766, fixedPointMean=0.0, fixedPointVar=1.0,
                 noise_shape=None, seed=None, name=None, training=False):
    """Dropout to a value with rescaling."""

    def dropout_selu_impl(x, rate, alpha, noise_shape, seed, name):
        keep_prob = 1.0 - rate
        x = ops.convert_to_tensor(x, name="x")
        if isinstance(keep_prob, numbers.Real) and not 0 < keep_prob <= 1:
            raise ValueError("keep_prob must be a scalar tensor or a float in the "
                                             "range (0, 1], got %g" % keep_prob)
        keep_prob = ops.convert_to_tensor(keep_prob, dtype=x.dtype, name="keep_prob")
        keep_prob.get_shape().assert_is_compatible_with(tensor_shape.scalar())

        alpha = ops.convert_to_tensor(alpha, dtype=x.dtype, name="alpha")
        keep_prob.get_shape().assert_is_compatible_with(tensor_shape.scalar())

        if tensor_util.constant_value(keep_prob) == 1:
            return x

        noise_shape = noise_shape if noise_shape is not None else array_ops.shape(x)
        random_tensor = keep_prob
        random_tensor += random_ops.random_uniform(noise_shape, seed=seed, dtype=x.dtype)
        binary_tensor = math_ops.floor(random_tensor)
        ret = x * binary_tensor + alpha * (1-binary_tensor)

        a = tf.sqrt(fixedPointVar / (keep_prob *((1-keep_prob) * tf.pow(alpha-fixedPointMean,2) + fixedPointVar)))

        b = fixedPointMean - a * (keep_prob * fixedPointMean + (1 - keep_prob) * alpha)
        ret = a * ret + b
        ret.set_shape(x.get_shape())
        return ret

    with ops.name_scope(name, "dropout", [x]) as name:
        return utils.smart_cond(training,
            lambda: dropout_selu_impl(x, rate, alpha, noise_shape, seed, name),
            lambda: array_ops.identity(x))


I am working on the file "TF_1_x/SelfNormalizingNetworks_MLP_MNIST.ipynb".
The first task is:
(4) Initialization with STDDEV of sqrt(1/n)
Can you write Python code for it?

# Store layers weight & bias
weights = {
    'h1': tf.Variable(tf.random_normal([n_input, n_hidden_1],stddev=np.sqrt(1/n_input))),
    'h2': tf.Variable(tf.random_normal([n_hidden_1, n_hidden_2],stddev=np.sqrt(1/n_hidden_1))),
    'out': tf.Variable(tf.random_normal([n_hidden_2, n_classes],stddev=np.sqrt(1/n_hidden_2)))
}
biases = {
    'b1': tf.Variable(tf.random_normal([n_hidden_1],stddev=0)),
    'b2': tf.Variable(tf.random_normal([n_hidden_2],stddev=0)),
    'out': tf.Variable(tf.random_normal([n_classes],stddev=0))
}
# Construct model
pred = multilayer_perceptron(x, weights, biases, rate=dropoutRate, is_training=is_training)

# Define loss and optimizer
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=pred, labels=y))
optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate).minimize(cost)

 # Test model
correct_prediction = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1))
# Calculate accuracy
accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))

# Initializing the variables
init = tf.global_variables_initializer()

# Create a histogramm for weights
tf.summary.histogram("weights2", weights['h2'])
tf.summary.histogram("weights1", weights['h1'])

# Create a summary to monitor cost tensor
tf.summary.scalar("loss", cost)
# Create a summary to monitor accuracy tensor
tf.summary.scalar("accuracy", accuracy)
# Merge all summaries into a single op
merged_summary_op = tf.summary.merge_all()

# Launch the graph
gpu_options = tf.GPUOptions(allow_growth=True)
with tf.Session(config=tf.ConfigProto(gpu_options=gpu_options)) as sess:
    sess.run(init)

    summary_writer = tf.summary.FileWriter(logs_path, graph=tf.get_default_graph())

    # Training cycle
    for epoch in range(training_epochs):
        avg_cost = 0.
        total_batch = int(mnist.train.num_examples/batch_size)
        # Loop over all batches
        for i in range(total_batch):
            batch_x, batch_y = mnist.train.next_batch(batch_size)
            batch_x = scaler.transform(batch_x)
     ...


I am working on the file "TF_1_x/getSELUparameters.ipynb".
The first task is:
For completeness: These are the correct expressions for mean zero and unit variance
Can you write Python code for it?

myAlpha = -np.sqrt(2/np.pi) / (np.exp(0.5) * erfc(1/np.sqrt(2))-1 )
myLambda = (1-np.sqrt(np.exp(1))*erfc(1/np.sqrt(2)))  *  \
            np.sqrt( 2*np.pi/ (2 + np.pi -2*np.sqrt(np.exp(1))*(2+np.pi)*erfc(1/np.sqrt(2)) + \
            np.exp(1)*np.pi*erfc(1/np.sqrt(2))**2 + 2*np.exp(2)*erfc(np.sqrt(2))))

print("Alpha parameter of the SELU: ", myAlpha)
print("Lambda parameter of the SELU: ", myLambda)


I am working on the file "TF_1_x/getSELUparameters.ipynb".
The first task is:
Function to obtain the parameters for the SELU with arbitrary fixed point (mean variance)
Can you write Python code for it?

def getSeluParameters(fixedpointMean=0,fixedpointVar=1):
    """ Finding the parameters of the SELU activation function. The function returns alpha and lambda for the desired fixed point. """

    import sympy
    from sympy import Symbol, solve, nsolve

    aa = Symbol('aa')
    ll = Symbol('ll')
    nu = fixedpointMean
    tau = fixedpointVar

    mean =  0.5*ll*(nu + np.exp(-nu**2/(2*tau))*np.sqrt(2/np.pi)*np.sqrt(tau) + \
                        nu*erf(nu/(np.sqrt(2*tau))) - aa*erfc(nu/(np.sqrt(2*tau))) + \
                        np.exp(nu+tau/2)*aa*erfc((nu+tau)/(np.sqrt(2*tau))))

    var = 0.5*ll**2*(np.exp(-nu**2/(2*tau))*np.sqrt(2/np.pi*tau)*nu + (nu**2+tau)* \
                          (1+erf(nu/(np.sqrt(2*tau)))) + aa**2 *erfc(nu/(np.sqrt(2*tau))) \
                          - aa**2 * 2 *np.exp(nu+tau/2)*erfc((nu+tau)/(np.sqrt(2*tau)))+ \
                          aa**2*np.exp(2*(nu+tau))*erfc((nu+2*tau)/(np.sqrt(2*tau))) ) - mean**2

    eq1 = mean - nu
    eq2 = var - tau

    res = nsolve( (eq2, eq1), (aa,ll), (1.67,1.05))
    return float(res[0]),float(res[1])

### To recover the parameters of the SELU with mean zero and unit variance
getSeluParameters(0,1)

### To obtain new parameters for mean zero and variance 2
myFixedPointMean = -0.1
myFixedPointVar = 2.0
myAlpha, myLambda = getSeluParameters(myFixedPointMean,myFixedPointVar)
getSeluParameters(myFixedPointMean,myFixedPointVar)


I am working on the file "TGA_Testing.ipynb".
The first task is:
And have a look at this!
Can you write Python code for it?

plt.ylim([-20,20])
plt.xlim([-20,20])
plt.scatter(*pts)
tga_line = np.dot(v[0].reshape((2,1)), np.array([-20,20]).reshape((1,2)))
plt.plot(*tga_line)
#plt.scatter(*L, c='red')


I am working on the file "TGA_Testing.ipynb".
The first task is:
Compute SVD on both the clean data and the outliery data
Can you write Python code for it?

U,s,Vt = np.linalg.svd(rotated)
U_n,s_n,Vt_n = np.linalg.svd(pts)


I am working on the file "TGA_Testing.ipynb".
The first task is:
Make Some Toy Data
Can you write Python code for it?

import matplotlib.pyplot as plt
import numpy as np

def mk_rot_mat(rad=np.pi / 4):
    rot = np.array([[np.cos(rad),-np.sin(rad)], [np.sin(rad), np.cos(rad)]])
    return rot

rot_mat = mk_rot_mat( np.pi / 4)
x = np.random.randn(100) * 5
y = np.random.randn(100)
points = np.vstack([y,x])

rotated = np.dot(points.T, rot_mat).T


I am working on the file "TIARA/GaussianProcessModelling/NOTEBOOKS/GPMCarbonDioxide.ipynb".
The first task is:
First grab the data:
Can you write Python code for it?

# For some reason the Mauna Loa dataset seems to have disappeared
# from the statsmodel package for the moment...

#data = sm.datasets.get_rdataset("co2").data

## time data values:
#t = np.array(data.time)

## CO_2 data values:
#y = np.array(data.co2)

# this is a function to read the Mauna Loa data from file
def read_co2(filename):

    co2file = open(filename,'r')

    time=[];co2=[]
    while True:
        line = co2file.readline()
        if not line: break

        items = line.split()

        if (items[0]!='#') and (items[3]!='-99.99'):

            time.append(float(items[2]))
            co2.append(float(items[3]))

    time=np.array(time)
    co2=np.array(co2)

    return time,co2

t,y = read_co2("mauna_loa.dat")


I am working on the file "TIARA/GaussianProcessModelling/NOTEBOOKS/GPMCarbonDioxide.ipynb".
The first task is:
Now let's plot the optimised  prediction...
Can you write Python code for it?

ax = pl.subplot(111)

# plot the original values
pl.plot(t_to_2003,y_to_2003,ls=':')

# shade in the area inside a one standard deviation bound:
ax.fill_between(x,mu-std,mu+std,facecolor='lightgrey', lw=0, interpolate=True)
pl.title("Predicted")

pl.ylabel(r"CO$_2$ concentration, ppm")
pl.xlabel("year")
pl.title(r"Mauna Loa CO$_2$ data - Optimised Prediction")
pl.axis([1957.,2025.,310.,420.])
pl.grid()

# display the figure:
pl.show()


I am working on the file "TIARA/GaussianProcessModelling/NOTEBOOKS/GPMCarbonDioxide.ipynb".
The first task is:
Set up the various covariance kernels. These are given by Eq. 5.15 - 5.18 in RW. Note that we've given each of these kernels arguments. These will act as our initial guess at parameter values.
Can you write Python code for it?

k1 = 66.0**2 * kernels.ExpSquaredKernel(67.0**2)
k2 = 2.4**2 * kernels.ExpSquaredKernel(90**2) * kernels.ExpSine2Kernel(2.0 / 1.3**2, 1.0)
k3 = 0.66**2 * kernels.RationalQuadraticKernel(0.78, 1.2**2)
k4 = 0.18**2 * kernels.ExpSquaredKernel(1.6**2) + kernels.WhiteKernel(0.19)


I am working on the file "TIARA/GaussianProcessModelling/NOTEBOOKS/GPMCarbonDioxide.ipynb".
The first task is:
So how did we do? We can compare our prediction to the actual measurements from 2005 until now...
Can you write Python code for it?

ax = pl.subplot(111)

# plot the training values
pl.plot(t_to_2003,y_to_2003,ls=':',c='b')

# shade in the area inside a one standard deviation bound:
ax.fill_between(x,mu-std,mu+std,facecolor='lightgrey', lw=0, interpolate=True)

# plot the full dataset
pl.plot(t[np.where(t>2003)],y[np.where(t>2003)],ls='-',c='r')

pl.ylabel(r"CO$_2$ concentration, ppm")
pl.xlabel("year")
pl.title(r"Mauna Loa CO$_2$ data - Comparison")
pl.axis([1958.,2025.,310.,420.])
pl.grid()

# display the figure:
pl.show()


I am working on the file "TIARA/GaussianProcessModelling/NOTEBOOKS/GPMCarbonDioxide.ipynb".
The first task is:
Then let's import all the libraries we need...
Can you write Python code for it?

import numpy as np
import george
from george import kernels
import statsmodels.api as sm
import scipy.optimize as op
import pylab as pl


I am working on the file "TIARA/GaussianProcessModelling/NOTEBOOKS/GPMCarbonDioxide.ipynb".
The first task is:
Then run the optimization routine:
Can you write Python code for it?

# initial guess at parameters (see above):
p0 = gp.kernel.vector

# if you want to view your initial guess values uncomment the line below
# print p0

# run optimization:
results = op.minimize(nll, p0, jac=grad_nll)

print np.exp(results.x)


I am working on the file "TIARA/GaussianProcessModelling/NOTEBOOKS/GPMCarbonDioxide.ipynb".
The first task is:
Then we sum them to make the combined kernel (Eq. 5.19 in RW).
Can you write Python code for it?


I am working on the file "TIARA/GaussianProcessModelling/NOTEBOOKS/GPMCarbonDioxide.ipynb".
The first task is:
Update the kernel with the results of the optimization:
Can you write Python code for it?


I am working on the file "TIARA/GaussianProcessModelling/NOTEBOOKS/GPMCarbonDioxide.ipynb".
The first task is:
We can immediately compute the covariance matrix because we've entered some initial guess parameters as the arguments in our kernels.
Can you write Python code for it?


I am working on the file "TIARA/GaussianProcessModelling/NOTEBOOKS/GPMCarbonDioxide.ipynb".
The first task is:
We then feed this kernel to the george library.
Can you write Python code for it?

#gp = george.GP(kernel, mean=np.mean(y_to_2005))
gp = george.GP(kernel, mean=0.0)


I am working on the file "TIARA/GaussianProcessModelling/NOTEBOOKS/GPMCarbonDioxide.ipynb".
The first task is:
You need to compute the GP once before starting the optimization. We've actually already done this above, but it doesn't hurt to run it again.
Can you write Python code for it?