I am working on the file "Preetinder Kalsi - Udacity Project 4 - Investigate a Dataset.ipynb".
The first task is:
Note PatientId column have exponential values in it.
Note No-show displays No if the patient visited and Yes if the Patient did not visited.
Data Cleanup
From the data description and questions to answer, I've determined that some of the dataset columns are not necessary for the analysis process and will therefore be removed. This will help to process the Data Analysis Faster.

PatientId
ScheduledDay
Sms_received
AppointmentID
AppointmentDay

i'll take a 3 step approach to data cleanup

Identify and remove duplicate entries
Remove unnecessary columns
Fix missing and data format issues

Step 1 - Removing Duplicate entries
Concluded that no duplicates entries exists, based on the tests below
Can you write Python code for it?

# Identify and remove duplicate entries
appointment_data_duplicates = appointment_data.duplicated()
print 'Number of duplicate entries is/are {}'.format(appointment_data_duplicates.sum())

# Lets make sure that this is working
duplication_test = appointment_data.duplicated('Age').head()
print 'Number of entries with duplicate age in top entries are {}'.format(duplication_test.sum())
appointment_data.head()


I am working on the file "Preetinder Kalsi - Udacity Project 4 - Investigate a Dataset.ipynb".
The first task is:
Question 1
How many Female and male of different Age Group in the Dataset missed the Appointments ?
Can you write Python code for it?

# Returns the percentage of male and female who visited the
# hospital on their appointment day with their Age
def people_visited(age_group, gender):
    grouped_by_total = clean_appointment_data.groupby(['age_group', 'Gender']).size()[age_group,gender].astype('float')
    grouped_by_visiting_gender = \
        clean_appointment_data.groupby(['age_group', 'people_showed_up', 'Gender']).size()[age_group,1,gender].astype('float')
    visited_gender_pct = (grouped_by_visiting_gender / grouped_by_total * 100).round(2)

    return visited_gender_pct

# Get the actual numbers grouped by Age, No-show, Gender
groupedby_visitors = clean_appointment_data.groupby(['age_group','people_showed_up','Gender']).size()

# Print - Grouped by Age Group, Patients showing up on thier appointments and Gender
print groupedby_visitors
print '0-5 - Female Appointment Attendence: {}%'.format(people_visited('0-5','F'))
print '0-5 - Male Appointment Attendence: {}%'.format(people_visited('0-5','M'))
print '5-15 - Female Appointment Attendence: {}%'.format(people_visited('5-15','F'))
print '5-15 - Male Appointment Attendence: {}%'.format(people_visited('5-15','M'))
print '15-25 - Female Appointment Attendence: {}%'.format(people_visited('15-25','F'))
print '15-25 - Male Appointment Attendence: {}%'.format(people_visited('15-25','M'))
print '25-35 - Female Appointment Attendence: {}%'.format(people_visited('25-35','F'))
print '25-35 - Male Appointment Attendence: {}%'.format(people_visited('25-35','M'))
print '35-45 - Female Appointment Attendence: {}%'.format(people_visited('35-45','F'))
print '35-45 - Male Appointment Attendence: {}%'.format(people_visited('35-45','M'))
print '45-55 - Female Appointment Attendence: {}%'.format(people_visited('45-55','F'))
print '45-55 - Male Appointment Attendence: {}%'.format(people_visited('45-55','M'))
print '55-65 - Female Appointment Attendence: {}%'.format(people_visited('55-65','F'))
print '55-65 - Male Appointment Attendence: {}%'.format(people_visited('55-65','M'))
print '65-75 - Female...


I am working on the file "Preetinder Kalsi - Udacity Project 4 - Investigate a Dataset.ipynb".
The first task is:
The Fixed_Age column is created in order to replace the negative value available in the Age column. The newly created Fixed_Age column will help in the proper calculation in the further questions and the results will be perfect and clear too. The negative value (-1) is changed in to a positive value (1) by using the .abd() function.
Can you write Python code for it?

# Create AgeGroups for further Analysis
'''bins = [0, 25, 50, 75, 100, 120]
group_names = ['0-25', '25-50', '50-75', '75-100', '100-120']
clean_appointment_data['age-group'] = pd.cut(clean_appointment_data['Fixed_Age'], bins, labels=group_names)
clean_appointment_data.head()'''
clean_appointment_data['Age_rounded'] = np.round(clean_appointment_data['Fixed_Age'], -1)

categories_dict = {0: '0-5',
                   10: '5-15',
                   20: '15-25',
                    30 : '25-35',
                    40 : '35-45',
                    50 : '45-55',
                   60: '55-65',
                    70 : '65-75',
                    80 : '75-85',
                  90: '85-95',
                  100: '95-105',
                  120: '105-115'}

clean_appointment_data['age_group'] = clean_appointment_data['Age_rounded'].map(categories_dict)

clean_appointment_data['age_group']


I am working on the file "Preetinder Kalsi - Udacity Project 4 - Investigate a Dataset.ipynb".
The first task is:
The graph above shows the number of people who attended their appointment and those who did not attended their appointments acccording to the Gender of the people having the appointment in the hospital.
- According to the graph above, women are more concious about their health regardless of the age group.
Can you write Python code for it?

# Graph - Actual count of passengers by survival, group and sex
g = sns.factorplot('people_showed_up', col='Gender', hue='age_group', data=clean_appointment_data, kind='count', size=15, aspect=.6)

# Fix up the labels
(g.set_axis_labels('People Who Attended', 'No. of Appointment')
    .set_xticklabels(["False", "True"], fontsize=20)
    .set_titles('{col_name}')
)

titles = ['Men', 'Women']
for ax, title in zip(g.axes.flat, titles):
    ax.set_title(title)


I am working on the file "Preparation_TzW.ipynb".
The first task is:
<h3>#prepare the training dataset for export</h3>
Can you write Python code for it?


I am working on the file "Preparation_TzW.ipynb".
The first task is:
take 80% as train dataframe, and leave 20% as a validator/to check the accuracy of our prediction before applying the model to the real test data.
Can you write Python code for it?

59400*0.8

dftrain = pd.read_csv('preparation.csv',delimiter=',',nrows=47520)

dfpayment = pd.read_csv('trainingset.csv',delimiter=',',nrows=47520)

dftrain.tail()

dftrain['terrain'] = 'dataran tinggi'

dftrain['terrain'][dftrain['gps_height']<=500] = 'dataran rendah'

dftrain['terrain'][(dftrain['gps_height']>500) & (dftrain['gps_height']<=1000)] = 'dataran sedang'

dftrain['terrain'].replace('dataran rendah',0,inplace=True)
dftrain['terrain'].replace('dataran sedang',1,inplace=True)
dftrain['terrain'].replace('dataran tinggi',2,inplace=True)

dftrain.head()

print dftrain.columns

dftrain['water_availability'] = dftrain.apply(lambda row: 0 if row['amount_tsh']==0.0 else 1,axis=1)

dftrain['payment'] = dfpayment['payment']

dftrain['payment'].unique()

dftrain['payment'].replace('pay annually',1,inplace=True)
dftrain['payment'].replace('never pay',2,inplace=True)
dftrain['payment'].replace('pay per bucket',3,inplace=True)
dftrain['payment'].replace('unknown',0,inplace=True)
dftrain['payment'].replace('pay when scheme fails',4,inplace=True)
dftrain['payment'].replace('other',5,inplace=True)
dftrain['payment'].replace('pay monthly',6,inplace=True)


I am working on the file "Prepare analysis data.ipynb".
The first task is:
Add location states
df_merged_meta_cats.u_location.value_counts().to_csv("USER_LOCATIONS.txt", sep="\t", encoding='utf-8')
! head USER_LOCATIONS.txt
! python process_user_locations.py ## RUN using python3 from command line
Can you write Python code for it?

df_places = pd.read_csv("PARSED_STATES.final.txt", sep="\t")
df_places = df_places.rename(columns={
        "location": "u_location", "parse_manual": "u_state"
                                     })[["u_location", "u_state"]]
df_places.head()

df_merged_meta_cats = df_merged_meta_cats.merge(df_places, how="left", on="u_location")

df_merged_meta_cats.u_state.head()

df_merged_meta_cats.t_id.value_counts().head()

df_merged_meta_cats.to_hdf("FINAL_ANALYSIS_DATA.h5", "final_data")


I am working on the file "Prepare analysis data.ipynb".
The first task is:
Same tweet belongs to multiple datasets
Can you write Python code for it?

df_merged_meta.t_id.value_counts().head()

df_merged_meta[df_merged_meta.t_id == 700042121877835776][["topic_name"]]

df_merged_meta.t_id.value_counts()[df_merged_meta.t_id.value_counts() > 1]

df_merged_meta[df_merged_meta.t_id == 792354716521009152].T

df_merged_meta["is_controversial"] = df_merged_meta.topic_name.isin(CONTROVERSIAL_TOPICS)
df_merged_meta.is_controversial.value_counts()


I am working on the file "PrepareData.ipynb".
The first task is:
AdaBoost
Can you write Python code for it?

clf_adaboost = AdaBoostClassifier(DecisionTreeClassifier())
clf_adaboost = clf_adaboost.fit(X_train, y_train)
accuracy = clf_adaboost.score(X_test, y_test)
print("Predictions:\n{}".format(clf_adaboost.predict(X_test)))
print("Actual:\n{}".format(y_test[:10]))
print("Score:\n{}".format(accuracy))


I am working on the file "PrepareData.ipynb".
The first task is:
Compare raw data random forest classification (Optional)
Can you write Python code for it?

# Classify test data using random forest on raw data (optional)
clf = RandomForestClassifier(n_estimators=10)
clf = clf.fit(X_train_raw, y_train_raw)
accuracy = clf.score(X_test_raw,y_test_raw)

print("Predictions:\n{}".format(clf.predict(X_test_raw)))
print("Actual:\n{}".format(y_test_raw[:10]))
print("Score:\n{}".format(accuracy))

#FIXME
random_search = RandomizedSearchCV(clf, param_distributions=param_dist,
                                   n_iter=n_iter_search)

random_search.fit(X_raw.values, y_raw.values)
print("RandomizedSearchCV evaluated %d candidates"
      " parameter settings." % (n_iter_search))
report(random_search.cv_results_)


I am working on the file "PrepareData.ipynb".
The first task is:
Extra Trees
Can you write Python code for it?

clf_extra_tree = ExtraTreesClassifier()
clf_extra_tree = clf_extra_tree.fit(X_train, y_train)
print(clf_extra_tree.score(X_test, y_test))


I am working on the file "PrepareData.ipynb".
The first task is:
Feature Engineering
Find features that increase the sample classification.
WIP - Histogram of Gradients
Can you write Python code for it?

np.asarray(data_dict['slide-horizontally']).shape

# FIXME: Complete HoG feature selection
import matplotlib.mlab as mlab
%matplotlib inline
data_slide_v = np.asarray(data_dict['slide-vertically'])
x_values = data_slide_v[:,4,...,0].flatten()
y_values = data_slide_v[:,4,...,1].flatten()
data_slide_h = np.asarray(data_dict['slide-horizontally'])
x_values_h = data_slide_h[:,4,...,0].flatten()
y_values_h = data_slide_h[:4,...,1].flatten()

fig, axarr = plt.subplots(2,2)
axarr[0,0].set_title('Horizontal motion in slide-vertical')
axarr[0,0].hist(x_values, bins=50, normed=True)
axarr[0,0].set_autoscaley_on(False)
axarr[0,0].set_ylim([0,1])

axarr[0,1].set_title('Vertical motion in slide-vertical')
axarr[0,1].hist(y_values, bins=50, normed=True,orientation='horizontal')
axarr[0,1].set_autoscalex_on(False)
axarr[0,1].set_xlim([0,.3])

axarr[1,0].set_title('Horizontal motion in slide-horizontal')
axarr[1,0].hist(x_values_h,bins=50,normed=True)
axarr[1,0].set_autoscaley_on(False)
axarr[1,0].set_ylim([0,1])

axarr[1,1].set_title('Vertical motion in slide-horizontal')
axarr[1,1].hist(y_values_h, bins=50,normed=True, orientation='horizontal')
axarr[1,1].set_autoscalex_on(False)
axarr[1,1].set_xlim([0,.3])

plt.tight_layout()
plt.show()


I am working on the file "PrepareData.ipynb".
The first task is:
Gradient Boosting
Can you write Python code for it?

clf_gradient_boosting = GradientBoostingClassifier(n_estimators=100)
clf_gradient_boosting = clf_gradient_boosting.fit(X_train, y_train)
print(clf_gradient_boosting.score(X_test,y_test))
# print("Predictions:\n{}".format(clf_bagging.predict(X_test)))
# print("Actual:\n{}".format(y_test))


I am working on the file "PrepareData.ipynb".
The first task is:
Hyper-parameter Optimization with Random Search (Optional)
Initialize random search module.
Can you write Python code for it?

from scipy.stats import randint as sp_randint
from sklearn.model_selection import RandomizedSearchCV

# specify parameters and distributions to sample from
param_dist = {"max_depth": [None],
              "max_features": sp_randint(5, 25),
              "min_samples_split": sp_randint(2, 11),
              "min_samples_leaf": sp_randint(1, 11),
              "bootstrap": [True, False],
              "criterion": ["gini", "entropy"]}

# run randomized search
n_iter_search = 60


I am working on the file "PrepareData.ipynb".
The first task is:
Multiclass Random Forest Classification
Can you write Python code for it?

# Classify test data using random forest
clf = RandomForestClassifier(n_estimators=10)
clf = clf.fit(X_train, y_train)
accuracy = clf.score(X_test,y_test)

print("Predictions:\n{}".format(clf.predict(X_test)))
print("Actual:\n{}".format(y_test[:10]))
print("Score:\n{}".format(accuracy))

#FIXME
random_search = RandomizedSearchCV(clf, param_distributions=param_dist,
                                   n_iter=n_iter_search)

random_search.fit(X.values, y.values)
print("RandomizedSearchCV evaluated %d candidates"
      " parameter settings." % (n_iter_search))
report(random_search.cv_results_)


I am working on the file "PrepareData.ipynb".
The first task is:
Multilayer Perceptron
Can you write Python code for it?

clf_mlpc = mlpc(hidden_layer_sizes=800,verbose=True)
clf_mlpc = clf_mlpc.fit(X_train, y_train)
print(clf_mlpc.score(X_test,y_test))


I am working on the file "PrepareData.ipynb".
The first task is:
Perform random search over hyper-parameter space
Can you write Python code for it?

data_list = get_data_list(divs=[4,10,20])
divs=[4,10,20]
combis = get_combis(divs)

for index in range(len(models)):
        # Train models
        random_search = RandomizedSearchCV(models[index], param_distributions=param_dist[index],
                                   n_iter=n_iter_search[index])
        random_search.fit(X_train, y_train)
        #print("RandomizedSearchCV evaluated %d candidates"
        #      " parameter settings." % (n_iter_search))

        scores = random_search.score(X_test, y_test)
        # Create a title for each column and the console by using str() and
        # slicing away useless parts of the string
        model_title = str(type(models[index])).split(".")[-1][:-2][:-len("Classifier")]
        print("best score for {} is {}, \n    with the parameters {}\n".format(model_title, random_search.best_score_,random_search.best_params_))


I am working on the file "PrepareData.ipynb".
The first task is:
SVM
Can you write Python code for it?

clf_svm = svm.SVC(kernel='poly', C=1.0)
clf_svm = clf_svm.fit(X_train, y_train)
print(clf_svm.score(X_test,y_test))


I am working on the file "PrepareData.ipynb".
The first task is:
Save weights for best ensemble method for offline use (Optional)
Can you write Python code for it?


I am working on the file "PrepareData.ipynb".
The first task is:
Show horizontal motion across frames (Optional)
Can you write Python code for it?

sample = data_dict['open-close'][5] # Fifth sample
anim = display_frames(sample)


I am working on the file "PrepareData.ipynb".
The first task is:
Visualize optical flow (Optional)
Computation of optical flow can be obtained for pixel $I(x,y,t)$ by
$$ f_x{u} + f_y{v} + f_t = 0 $$
where:
$$ f_x = \frac{\partial f}{\partial x} \; ; \; f_y = \frac{\partial f}{\partial y} $$
$$ u = \frac{dx}{dt} \; ; \; v = \frac{dy}{dt} $$
Show individual frame (Optional)
Can you write Python code for it?

gestures = list(data_dict)
print(gestures) # List gestures
sample = data_dict['open-close'][3] # 3rd Open-close sample
image = sample[4] # 5th frame of sample
plt.imshow(image[...,0]) # x-coordinates slice
plt.show()


I am working on the file "PrepareData.ipynb".
The first task is:
test averaging over frames and features
Can you write Python code for it?

avg_frame = np.zeros((len(data_open_close),len(data_open_close[1][1]),len(data_open_close[1][1][1]),len(data_open_close[1][1][1][1])))
for i in range(len(data_open_close)):
    sum_frame = np.zeros((40,40,2))
    for j in range(len(data_open_close[i])):
        sum_frame+= data_open_close[i][j]
    avg_frame[i] = sum_frame/len(data_open_close[i])

avg_feature = np.zeros((len(data_open_close[1]),len(data_open_close[1][1]),len(data_open_close[1][1][1]),len(data_open_close[1][1][1][1])))
sum_feature = np.zeros((10,40,40,2))
for i in range(len(data_open_close)):
    sum_feature+= data_open_close[i]
avg_feature = sum_feature/len(data_open_close)


I am working on the file "PrepareData.ipynb".
The first task is:
分词不是很成功，我们转向直接用汉字字符来代替分段，我们保留标点符号
Can you write Python code for it?

print('{} START'.format(time.strftime(TIME_FORMAT)))

DEFAULT_FOUT = os.path.join(DATA_FOLDER, 'poem.txt')
DEFAULT_charSegment = os.path.join(DATA_FOLDER, 'Charactersegment.txt')

fd = codecs.open(DEFAULT_FOUT, 'r', 'utf-8')
fw = codecs.open(DEFAULT_charSegment, 'w', 'utf-8')

start_flag = False
for line in fd:
    if len(line) > 0:
        for c in line:
            if c != '\n':
                fw.write(c)
                fw.write(' ')
    fw.write('\n')

fd.close()
fw.close()

print('{} STOP'.format(time.strftime(TIME_FORMAT)))

GetFirstNline(DEFAULT_charSegment, 3)

print('{} START'.format(time.strftime(TIME_FORMAT)))
from gensim.models import word2vec

#DEFAULT_Segment = os.path.join(DATA_FOLDER, 'wordsegment.txt')
DEFAULT_Char2Vec = os.path.join(DATA_FOLDER, 'Char2Vec100.bin')

fd = codecs.open(DEFAULT_charSegment, 'r', 'utf-8')

sentences = fd.readlines()

fd.close

model = word2vec.Word2Vec(sentences, size=100)

#DEFAULT_Segment = os.path.join(DATA_FOLDER, 'wordsegment.txt')
model.save(DEFAULT_Char2Vec)

print('{} STOP'.format(time.strftime(TIME_FORMAT)))

model[u'男']

print('{} START'.format(time.strftime(TIME_FORMAT)))
from gensim.models import word2vec

DEFAULT_charSegment = os.path.join(DATA_FOLDER, 'Charactersegment.txt')
DEFAULT_Char2Vec50 = os.path.join(DATA_FOLDER, 'Char2Vec50.bin')

fd = codecs.open(DEFAULT_charSegment, 'r', 'utf-8')

sentences = fd.readlines()

fd.close

model = word2vec.Word2Vec(sentences, size=50)

#DEFAULT_Segment = os.path.join(DATA_FOLDER, 'wordsegment.txt')
model.save(DEFAULT_Char2Vec50)

print('{} STOP'.format(time.strftime(TIME_FORMAT)))

model.wv.most_similar([u'好'])


I am working on the file "PrepareData.ipynb".
The first task is:
分词实验
DEFAULT_FOUT = os.path.join(DATA_FOLDER, 'poem.txt')
thu1 = thulac.thulac(seg_only=True)   #只进行分词，不进行词性标注
text = thu1.cut("我爱北京天安门", text=True)  #进行一句话分词
print(text)
thu1 = thulac.thulac(seg_only=True)  #只进行分词，不进行词性标注
thu1.cut_f(DEFAULT_FOUT, outp)  #对input.txt文件内容进行分词，输出到output.txt
Can you write Python code for it?

print('{} START'.format(time.strftime(TIME_FORMAT)))

import thulac
DEFAULT_Segment = os.path.join(DATA_FOLDER, 'wordsegment.txt')

fd = codecs.open(DEFAULT_FOUT, 'r', 'utf-8')
fw = codecs.open(DEFAULT_Segment, 'w', 'utf-8')

thu1 = thulac.thulac(seg_only=True)   #只进行分词，不进行词性标注

for line in fd:
    #print(line)
    fw.write(thu1.cut(line, text=True))
    fw.write('\n')

fd.close()
fw.close()

print('{} STOP'.format(time.strftime(TIME_FORMAT)))

print('{} START'.format(time.strftime(TIME_FORMAT)))
from gensim.models import word2vec

#DEFAULT_Segment = os.path.join(DATA_FOLDER, 'wordsegment.txt')
DEFAULT_Word2Vec = os.path.join(DATA_FOLDER, 'Word2Vec150.bin')

sentences = word2vec.Text8Corpus(DEFAULT_Segment)

model = word2vec.Word2Vec(sentences, size=150)

#DEFAULT_Segment = os.path.join(DATA_FOLDER, 'wordsegment.txt')
model.save(DEFAULT_Word2Vec)

print('{} STOP'.format(time.strftime(TIME_FORMAT)))

model[u'男']

DEFAULT_FIN = os.path.join(DATA_FOLDER, '唐诗语料库.txt')
DEFAULT_FOUT = os.path.join(DATA_FOLDER, 'poem.txt')
DEFAULT_Segment = os.path.join(DATA_FOLDER, 'wordsegment.txt')
def GetFirstNline(filePath, linesNumber):
    fd = codecs.open(filePath, 'r', 'utf-8')
    for i in range(1,linesNumber):
        print(fd.readline())
    fd.close()

GetFirstNline(DEFAULT_Segment, 3)
GetFirstNline(DEFAULT_FOUT, 3)


I am working on the file "PrepareData.ipynb".
The first task is:
把汉字转成拼音
Can you write Python code for it?


I am working on the file "PrepareData.ipynb".
The first task is:
读取数据，去掉不用的数据
Can you write Python code for it?

if __name__ == '__main__':
  #  parser = set_arguments()
  #  cmd_args = parser.parse_args()

    print('{} START'.format(time.strftime(TIME_FORMAT)))

    fd = codecs.open(DEFAULT_FIN, 'r', 'utf-8')
    fw = codecs.open( DEFAULT_FOUT, 'w', 'utf-8')
    reg = re.compile('〖(.*)〗')
    start_flag = False
    for line in fd:
        line = line.strip()
        if not line or '《全唐诗》' in line or '<http'  in line or '□' in line:
            continue
        elif '〖' in line and '〗' in line:
            if start_flag:
                fw.write('\n')
            start_flag = True
            g = reg.search(line)
            if g:
                fw.write(g.group(1))
                fw.write('\n')
            else:a
                # noisy data
                print(line)
        else:
            line = reg_noisy.sub('', line)
            line = reg_note.sub('', line)
            line = line.replace(' .', '')
            fw.write(line)

    fd.close()
    fw.close()

    print('{} STOP'.format(time.strftime(TIME_FORMAT)))


I am working on the file "Preprocessing and Pipelines.ipynb".
The first task is:
Cross-validation with a pipeline
Can you write Python code for it?

from sklearn.cross_validation import cross_val_score
cross_val_score(pipeline, X_train, y_train)


I am working on the file "Preprocessing and Pipelines.ipynb".
The first task is:
Exercise
Make a pipeline out of the StandardScaler and KNeighborsClassifier and search over the number of neighbors.
Can you write Python code for it?


I am working on the file "Preprocessing train dataset.ipynb".
The first task is:
Create symbolic link of images
Can you write Python code for it?

def remove_and_create_class(dirname):
    if os.path.exists(dirname):
        shutil.rmtree(dirname)
    os.mkdir(dirname)
    os.mkdir(dirname+'/cat')
    os.mkdir(dirname+'/dog')

remove_and_create_class('mytrain')
remove_and_create_class('myvalid')

for filename in mytrain_cat:
    os.symlink('../../train/'+filename, 'mytrain/cat/'+filename)

for filename in mytrain_dog:
    os.symlink('../../train/'+filename, 'mytrain/dog/'+filename)

for filename in myvalid_cat:
    os.symlink('../../train/'+filename, 'myvalid/cat/'+filename)

for filename in myvalid_dog:
    os.symlink('../../train/'+filename, 'myvalid/dog/'+filename)


I am working on the file "Preprocessing train dataset.ipynb".
The first task is:
Shuffle and split the train filenames
Can you write Python code for it?

mytrain, myvalid = train_test_split(train_filenames, test_size=0.1)
print len(mytrain), len(myvalid)


I am working on the file "Preprocessing train dataset.ipynb".
The first task is:
Visualize the size of the original train dataset.
Can you write Python code for it?

train_filenames = os.listdir('train')
train_cat = filter(lambda x:x[:3] == 'cat', train_filenames)
train_dog = filter(lambda x:x[:3] == 'dog', train_filenames)
x = ['train_cat', 'train_dog', 'test']
y = [len(train_cat), len(train_dog), len(os.listdir('test'))]
ax = sns.barplot(x=x, y=y)


I am working on the file "Preprocessing train dataset.ipynb".
The first task is:
Visualize the size of the processed train dataset
Can you write Python code for it?

mytrain_cat = filter(lambda x:x[:3] == 'cat', mytrain)
mytrain_dog = filter(lambda x:x[:3] == 'dog', mytrain)
myvalid_cat = filter(lambda x:x[:3] == 'cat', myvalid)
myvalid_dog = filter(lambda x:x[:3] == 'dog', myvalid)
x = ['mytrain_cat', 'mytrain_dog', 'myvalid_cat', 'myvalid_dog']
y = [len(mytrain_cat), len(mytrain_dog), len(myvalid_cat), len(myvalid_dog)]

ax = sns.barplot(x=x, y=y)


I am working on the file "PresentableNotebook.ipynb".
The first task is:
Phase 1: Construction
Step 1:
Concatenate time series from all subjects into a 264 x Time matrix
where times 1-720 are from person 1, times 721 - 1441 are from person
2, etc. through person N.
Can you write Python code for it?

# Gather all the files.
files = os.listdir('timeseries/')

# Concatenate (or stack) all the files.
# Approx 12.454981 seconds
i = 0
for f in files:
    if i == 0:
        ts_matrix = np.loadtxt('timeseries/' + f).T
        i += 1
    else:
        new_ts = np.loadtxt('timeseries/' + f).T
        ts_matrix = np.hstack((ts_matrix, new_ts))


I am working on the file "PresentableNotebook.ipynb".
The first task is:
Phase 2: Validation
Retrieve the vector containing the list of assigned clusters at each time (0 - 46208).
Each number in the vector of cluster assignments tells you which time
point and which subject is assigned to which cluster. So now you need
to go back and time the original data for that time point. Remember
that the original data you had is a vector of 264 activities for each
time point.
Can you write Python code for it?

def index_list(num, ind_list, ts_matrix):
    i = 0
    for z in zip(ind_list, ts_matrix):
        if z[0] == num and i == 0:
            output = np.array([z[1]])
            i += 1
        elif z[0] == num and i != 0:
            output = np.append(output, [z[1]], axis=0)
    return output

louvain_ind = read_csv('mem.csv').values.T

for f in files:
    ts_matrix = np.loadtxt('timeseries/' + f).T

    for i in range(1, 65):
        subject = louvain_ind[:722 * i][0]
        for j in range(4):
            i_list = index_list(j, subject, ts_matrix)
            avg = np.average(i_list, axis=1)
            Series(avg).to_csv("module_matrices/subject" + str(i)
                                + "mod" + str(j), index=False, sep="\t")


I am working on the file "PresentableNotebook.ipynb".
The first task is:
Step 2:
Compute a Time x Time correlation matrix using Pearson correlation
coefficients.
Can you write Python code for it?

"""
Compute the correlation matrix
"""
corr_mat = np.corrcoef(ts_matrix.T)

# Save in .npz file
# np.savez_compressed('corr_mat.npz', corr_mat=corr_mat)


I am working on the file "PresentableNotebook.ipynb".
The first task is:
Step 3:
Threshold the Time x Time correlation matrix to retain x% of the
strongest connections.
Can you write Python code for it?

# X = np.load('corr_mat.npz')
# X = X['corr_mat']

# a flatten function optimized by numba
@jit
def fast_flatten(X):
    k = 0
    length = X.shape[0] * X.shape[1]
    X_flat = np.empty(length)
    for i in xrange(X.shape[0]):
        for j in xrange(X.shape[1]):
            X_flat[k] = X[i, j]
            k += 1
    return X_flat

# helper function that returns the min of the number of
# unique values depending on the threshold
def min_thresh_val(X, threshold):
    X_flat = fast_flatten(X)
    index = int(len(X_flat) * threshold)
    return np.unique(sort(X_flat))[::-1][:index].min()

# Computes the threshold matrix without killing the python kernel
@jit
def thresh_mat(X, threshold):
    min_val = min_thresh_val(X, threshold)
    print("Done with min_thresh_val")
    # M = zeros((X.shape[0], X.shape[1]))
    for i in xrange(X.shape[0]):
        for j in xrange(X.shape[1]):
            # if X[i, j] >= min_val:
                # M[i, j] = X[i, j]
            if X[i, j] < min_val:
                X[i, j] = 0

thresh_mat(X, .01)
print("Finished Threshold Matrix")
# savez_compressed('threshold_mat.npz', threshold_mat=X)


I am working on the file "PresentableNotebook.ipynb".
The first task is:
Step 4:
Feed the thresholded Time x Time correlation matrix into igraph to
maximize modularity (a community detection technique) which will
provide us with an association of time points to brain states (a.k.a.
modules, communities, or clusters).
Can you write Python code for it?

# from: http://stackoverflow.com/questions/29655111/igraph-graph-from-numpy-or-pandas-adjacency-matrix

# get the row, col indices of the non-zero elements in your adjacency matrix
conn_indices = np.where(X)

# get the weights corresponding to these indices
weights = X[conn_indices]

# a sequence of (i, j) tuples, each corresponding to an edge from i -> j
edges = zip(*conn_indices)

# initialize the graph from the edge sequence
G = Graph(edges=edges, directed=False)

# assign node names and weights to be attributes of the vertices and edges
# respectively
G.vs['label'] = np.arange(X.shape[0])
G.es['weight'] = weights

# get the vertex clustering corresponding to the best modularity
cm = G.community_multilevel()

# save the cluster membership of each node in a csv file
Series(cm.membership).to_csv('mem.csv', index=False)


I am working on the file "Presentaciones/Notas/01_NBJupyter_basico.ipynb".
The first task is:
El bloque anterior se ejecuta en python y mantiene una conexión a una instancia de interprete interactivo de python, de manera que podemos usar celdas de markdown y otras celdas de código, y luego hacer uso de las definiciones de estas celdas. Como a continuación:
Can you write Python code for it?


I am working on the file "Presentaciones/Notas/01_NBJupyter_basico.ipynb".
The first task is:
En caso de usar los comandos bash con ! es posible asociar sus salidas a una variable python, y obtener una integración natural entre python y bash.
Can you write Python code for it?

listaarchivos = !ls
print(listaarchivos)


I am working on the file "Presentaciones/Notas/01_NBJupyter_basico.ipynb".
The first task is:
Magics
Existen una gran cantidad de magics disponibles. Particularmente nos interesarán dos casos, bash y LaTeX.
La ejecución de celdas LaTeX es solo para los ambientes matematicos (con soporte por MathJax) y no un soporte completo de una distribución LaTeX.
Can you write Python code for it?


I am working on the file "Presentaciones/Notas/01_NBJupyter_basico.ipynb".
The first task is:
Otros magics disponibles son:
Can you write Python code for it?


I am working on the file "Presentaciones/Notas/01_NBJupyter_basico.ipynb".
The first task is:
Para la ejecución de bash se puede anteceder un signo ! antes del comando a usar (de una sola linea) o usar el magic correspondiente para múltiples lineas. Esto solo es posible en sistemas UNIX.
Can you write Python code for it?

!ls -al

%%bash
touch archivoprueba.txt
ls *.txt
echo "Ya se verifico creación, ahora se eliminará."
rm archivoprueba.txt
ls archivoprueba.txt


I am working on the file "Presentaciones/Notas/02_NBJupyter_intermedio.ipynb".
The first task is:
El método display cumple la función de renderización de código html en el navegador, donde dicho código es generado por otro método, en este caso por el método markdown que convierte markdown a html y si almacena en un objeto de ipython para dicho fin.
De esta misma forma se pueden generar elementos de documento más complejos de forma automatica, como tablas.
Can you write Python code for it?

fila1 = "|columna 1|columna 2|"
filaalineacion = "|---:|:---:|"
fila2 = "|der|cen|"
display(Markdown(fila1+"\n"+filaalineacion+"\n"+fila2))


I am working on the file "Presentaciones/Notas/02_NBJupyter_intermedio.ipynb".
The first task is:
En este ejemplo las variables fueron asignadas de manera estatica, pero usted puede generar un código que cree el contenido de cada celda, y uniendo su representación en string a las cadenas necesarias para formar el patron markdown de una tabla.
De igual manera es posible generar renderizado de ecuaciones de forma sistematica (elemento aprovechado por paquetes de algebra como Sympy).
Can you write Python code for it?

latexexp = "$$\\frac{mc^2}{2}$$"
display(Latex(latexexp))


I am working on the file "Presentaciones/Notas/02_NBJupyter_intermedio.ipynb".
The first task is:
Otras opciones de visualización soportadas son:
Can you write Python code for it?

import IPython.display as Display
dir(Display)


I am working on the file "Presentaciones/Notas/03_Linux_basico.ipynb".
The first task is:
Ayuda
Si desconocemos las opciones y descripción de una instrucción podemos usar el argumento --help que la mayoria de instrucciones soportan o la instrucción man seguida de la instrucción que deseamos.
Can you write Python code for it?


I am working on the file "Presentaciones/Notas/03_Linux_basico.ipynb".
The first task is:
Búsqueda
Podemos hacer busqueda en archivos y en directorios de texto o archivos respectivamente. Para ello podemos usar las instrucciones grep, find y locate.
Can you write Python code for it?

%%bash
echo "Buscar texto en un archivo"
grep Jupyter prueba2.txt
echo "Buscar texto en salida de consola"
ls -l | grep prueba
echo "Buscar archivo en un directorio"
find . -iname "Jupyter*"


I am working on the file "Presentaciones/Notas/03_Linux_basico.ipynb".
The first task is:
Gestión de procesos
Podemos gestionar los procesos del sistema con las utilidades ps, top y kill. Las dos primeras permiten realizar observación de los procesos en ejecución en el sistema y ciertos argumentos para definir el detalle de la información de estos. Mediante kill y el identificador del proceso podemos elminar este.
Can you write Python code for it?


I am working on the file "Presentaciones/Notas/03_Linux_basico.ipynb".
The first task is:
Internet
ping seguido de la url valida el estado de recepción y envio de paquetes al sitio indicado de manera continua hasta que se realice interrupción de la consola.
dig realiza detección de la IP del sitio.
Can you write Python code for it?


I am working on the file "Presentaciones/Notas/03_Linux_basico.ipynb".
The first task is:
Las primeras columnas de los listados con propiedades se corresponden a los permisos de los archivos. La primera columna corresponde a si es un directorio d o un archivo regular -. Las siguientes columnas en grupos de 3 asocian a los permisos del usuario propietario, usuarios del grupo y resto de usuarios, siendo la primera columna del grupo para indicar lectura r, la segunda escritura w y la última ejecución x. La carencia de dichas propiedades se designa con -.
Para crear archivos o actualizar la huella de tiempo se puede usar la instrucción touch seguida del nombre del archivo. Si se usa para la creación de un archivo, este será por defecto un archivo de texto plano.
El contenido de un archivo de texto plano se puede poblar con ayuda de los simbolos > y >> que se encargar de direccionar las salidas de la consola hacia el archivo deseado. Si desea sobreescribir los archivos use la primera opción, pero si los desea continuar use la segunda opción.
Can you write Python code for it?

%%bash
touch prueba.txt
ls -lh *txt
ls > prueba.txt # Se redirige la salida de ls al archivo prueba.txt, el cual se sobre escribe
ls -lh *txt


I am working on the file "Presentaciones/Notas/03_Linux_basico.ipynb".
The first task is:
Permisos
Los permisos permiten no solo el acceso a los archivos por parte de los usuarios, sino la posibilidad de restringir el tipo de operaciones sobre los archivos.
|Permiso|Valor|Letra|
|---|---|---|
|Lectura|4|r|
|Escritura|2|w|
|Ejecución|1|x|
Los permisos se pueden habilitar mediante + seguido de la letra asociada al permiso y el nombre del archivo, o mediante la indicación de la suma de valores de los permisos y en orden acorde a usuario propietario, usuarios del grupo y resto de usuarios.
Can you write Python code for it?

%%bash
rm -f ejecutar_prueba.sh
echo "echo hola" > ejecutar_prueba.sh # Creamos un archivo para ejecución
echo "Propiedades del archivo"
ls -oh *sh # Validamos permisos de ejecución. Por defecto los permisos son solo lectura.
echo "Se ejecuta el archivo"
./ejecutar_prueba.sh # Ejecutamos y se genera error de permisos.
echo "Se puede obviar el error por parte del propietario mediante sh (bash/dash/zsh/..)"
bash ejecutar_prueba.sh
echo "Se cambian permisos y ejecuta nuevamente"
chmod 755 ejecutar_prueba.sh # Se otorga permiso de ejecución
./ejecutar_prueba.sh # Al ejecutar nuevamente se genera la salida esperada


I am working on the file "Presentaciones/Notas/03_Linux_basico.ipynb".
The first task is:
Podemos hacer consulta de los archivos y directorios existentes por medio de la utilidad ls, la cual posee argumentos que permiten controlar el nivel de información que obtenemos sobre estos.
Can you write Python code for it?

%%bash
echo "listar archivos"
ls
echo "listar archivos, incluyendo archivos ocultos"
ls -a
echo "listar archivos, incluyendo archivos ocultos e indicando propiedades"
ls -al
echo "listar archivos, incluyendo archivos ocultos e indicando propiedades con indicación de peso con prefijos"
ls -alh


I am working on the file "Presentaciones/Notas/03_Linux_basico.ipynb".
The first task is:
Si deseamos realizar una copia del archivo, usamos la instrucción cp. Para cambiar su ubicación o renombrar se usa la instrucción mv (renombrar es lo mismo que cambiar la locación del archivo).
Can you write Python code for it?

%%bash
cp prueba.txt prueba2.txt # Se hace una copia de prueba.txt en prueba2.txt
mv prueba.txt prueba3.txt # Se cambia el nombre de prueba.txt a prueba3.txt
ls prueba*.txt # Se valida la existencia de los archivos con patron 'prueba' al inicio y extensión '.txt'.


I am working on the file "Presentaciones/Notas/03_Linux_basico.ipynb".
The first task is:
Tambien es posible realizar descargas por medio de la consola con la instrucción wget.
Can you write Python code for it?

!wget -O comandoslinux.jpg https://tanaceto.files.wordpress.com/2007/11/fwunixref_es.jpg

!ls *jpg


I am working on the file "Presentaciones/Notas/03_Linux_basico.ipynb".
The first task is:
man genera una nueva instancia en la visualización de la consola y muestra información más detallada que --help normalmente.
Instalación
Partiendo de una distribución Debian/Ubuntu tenemos las siguientes utilidades: apt y dpkg.
La instrucción apt (cuya descripción y opciones se muestra en la celda superior) soporta los argumentos -y para confirmación de instalación, -q para proceso silencioso y -f para forzado, entre otros.
Algunas funciones de gestión de paquetes aún no se encuentran en apt por lo que aún debe usarse su predecesor apt-get y en combinación con apt-cache.
Can you write Python code for it?


I am working on the file "Presentaciones/Notas/04_Linux_bash.ipynb".
The first task is:
Ciclos
Can you write Python code for it?

%%bash
for i in $( ls ); do
    echo item: $i
done

%%bash
for i in `seq 1 10`; do
    echo $i
done

%%bash
for ((I=0; I<=10; I++)); do
    printf "$I\n"
done

%%bash
COUNTER=0
while [  $COUNTER -lt 10 ]; do
    echo The counter is $COUNTER
    let COUNTER=COUNTER+1
done

%%bash
COUNTER=20
until [  $COUNTER -lt 10 ]; do
    echo COUNTER $COUNTER
    let COUNTER-=1
done


I am working on the file "Presentaciones/Notas/04_Linux_bash.ipynb".
The first task is:
El paso de variables a las funciones se realiza con ayuda de las variables globales.
Es conveniente que las funciones de ayuda se creen al inicio tras la indicación de las variables de inicio. La idea de las funciones de ayuda son funciones que se ejecutan bajo condiciones como directamente un termino clave para mostrar la ayuda o cuando el ingreso de los argumentos es erroneo.
Can you write Python code for it?

%%bash
function imprimir_saludo(){
echo "Hola $0 $1"
}
imprimir_saludo Edward


I am working on the file "Presentaciones/Notas/04_Linux_bash.ipynb".
The first task is:
Estructuras de control
Condicionales
La estructura condicional tiene el esquema
if [condicion]; then
    sentencia
elif [condicion]; then
    sentencia
else
    sentencia
fi

Los corchetes pueden ser simples o dobles según el efecto de evaluación de la condición. Con corchetes simples se requiere indicar la variable entre comillas mientras que con corchetes dobles se puede omitir las comillas.
Can you write Python code for it?

%%bash
T1="foo"
T2="bar"
if [ "$T1" = "$T2" ]; then
    echo expression evaluated as true
else
    echo expression evaluated as false
fi


I am working on the file "Presentaciones/Notas/04_Linux_bash.ipynb".
The first task is:
La obtención del valor no sirve solo para su impresión o manipulación por otras expresiones sino tambien para la ejecución de la instrucción guardada como cadena de caracteres.
Can you write Python code for it?


I am working on the file "Presentaciones/Notas/04_Linux_bash.ipynb".
The first task is:
Para la lectura de variables recurrimos a la instrucción read seguido de la variable deseada.
Tambien es posible usar vectores mediante la siguiente descripción:
Can you write Python code for it?

%%bash
vector=( 4 2 8 9 )
echo ${vector[0]}
echo ${vector[3]}
echo ${vector[@]}
echo ${#vector[@]}
echo ${vector[*]}
echo ${vector[@]:2}


I am working on the file "Presentaciones/Notas/04_Linux_bash.ipynb".
The first task is:
Scripts
Nuestros scripts en bash como en cualquier otro lenguaje poseen estructuras definidas claramente. En bash, encontramos:

Cabecera: Sirve para indicar el interprete a utilizar y la codificación.
Definición de variables globales: Asignamos valores a variables que serán de acceso en todo el script.
Ayuda: Define la función de ayuda del script. Útil para la indicación de uso.
Funciones: Permite un diseño modular de nuestro script.
Cuerpo principal.

Debemos recordar que antes de ejecutar el script debemos asignar los permisos adecuados, ejemplo: chmod 755 script.sh.
Cabecera
Nuestra primera linea de cabecera corresponde a la indicación del interprete de comandos a usar, el cual nos interesa que sea bash. En este caso la linea debe lucir como sigue:
#!/bin/bash

Esta linea puede usarse por ejemplo, para indicar que un script debe ejecutarse con python sin necesidad de ingresar a la consola de python.
Una vez se definio el interprete, ya es posible ejecutar nuestro script como ./script.sh.
La segunda linea obedece a la codificación, lo cual nos permitirá hacer uso de codificación UTF-8 para soporte de caracteres latinos, los cuales por defecto en la codificación ASCII no son posibles (es la versión de 7 bits y no el extendido de 8 bits).
# -*- ENCODING: UTF-8 -*-

Variables
La asignación de variables en bash se hace de forma directa sin requerir de la declaración de tipo de variable. Las variables en bash soportan números, caracteres y cadenas de caracteres, y se deben asignar sin dejar espacios alrededor del signo igual. Las variables pueden ser de caracter local antecediendo la instrucción local.
Para obtener el valor almacenado en una variable se requiere el uso del signo $ antes de su nombre.
Can you write Python code for it?


I am working on the file "Presentaciones/Notas/04_Linux_bash.ipynb".
The first task is:
Si el signo $ es omitido, se interpreta el nombre de la variable como un comando.
Can you write Python code for it?


I am working on the file "Presentaciones/Notas/06_Versionamiento_git.ipynb".
The first task is:
Ahora agregamos al registro los archivos de nuestro repositorio local.
Can you write Python code for it?

%%bash
cd herramientas
git add . # Con la indicación "." agrega todo.
git commit -m "Agregados archivos de prueba 1 y 2."


I am working on the file "Presentaciones/Notas/06_Versionamiento_git.ipynb".
The first task is:
Aunque en este caso es claro que nuestros archivos no se encuentran agregados al repositorio, consultaremos por el estado de los archivos frente a su registro en el repositorio.
Can you write Python code for it?


I am working on the file "Presentaciones/Notas/06_Versionamiento_git.ipynb".
The first task is:
Configuración
Para una adecuada identificación de nuestros aportes en los repositorios, es necesario configurar nuestro nombre y correo en git.
git config --global user.name "YOUR NAME"
git config --global user.email "YOUR EMAIL ADDRESS"

Creación de repositorios
Una vez creada la cuenta github, debe crear un repositorio git en ella y copiar la dirección de clonado, la cual se usa con el fin de obtener una copia del repositorio localmente.
Can you write Python code for it?

%%bash
git clone https://github.com/cosmoscalibur/herramientas_computacionales.git herramientas


I am working on the file "Presentaciones/Notas/06_Versionamiento_git.ipynb".
The first task is:
De esta forma crearemos una copia del repositorio git herramientas_computacionales en el directorio ./herramientas. Si no se indica el directorio, se crea un directorio nuevo en la posición actual.
Tras el clonado, tenemos a disposición todo el directorio del repositorio, incluido el directorio .git que almacena la información del historial y repositorios (para este momento, heredada del repositorio clonado).
Can you write Python code for it?


I am working on the file "Presentaciones/Notas/06_Versionamiento_git.ipynb".
The first task is:
Durante la ejecución del push se nos solicitará el usuario y contraseña de nuestra cuenta en el servicio git. Al usar el protocolo https por defecto tendremos 3600 segundos tras cada autenticación, o si lo preferimos podemos usar ssh y crear la llave de autenticación con el fin de tener acceso permanente en nuestro equipo privado sin requerir de la autenticación continua.
Para la actualización de nuestro repositorio local con el repositorio remoto reaizamos una acción pull, tal cual se indico en la subsección anterior para actualizar el repositorio local con los archivos que posiblemente se crearon con el asistente del servicio git.
Otras acciones
Se recomienda la lectura de otras acciones de git importantes, las cuales son: branch y merge para experimentar con modificaciones de un código base y su posterior unión al original, o log, show para la visualización del historial de modificaciones.
Si particularmente queremos ver el estado del repositorio en un commit especifico, usamos checkout seguido del commit. Esta acción no genera ninguna modificación del repositorio y para continuar en el último estadosolo debemos volver a nuestro estado actual realizando el checkout al master. Si deseamos conservar este estado consultado realizamos una reversión al commit dado con reverse.
Si la acción anterior se realiza sobre un archivo, los cambios deben deshacerse con checkout a HEAD seguido del archivo, de lo contrario el estado del archivo consultado estará como una modificación.
Igualmente existen acciones de interes en caso de usar los servicios git como la acción del fork que hace uso de la solicitud de pull (acciones no incluidas en git pero que pueden ser usadas con clientes basados en las API de los servicios git).
Es posible tener submodules para lo cual git ofrece la utilidad submodule para su gestión.
Can you write Python code for it?

%%bash
cd herramientas
git log --oneline

%%bash
cd herramientas
git show d193ae6

%%bash
cd herramientas
echo "Actual"
ls
git checkout e4a4be5
echo "Anterior"
ls
git checkout master

%%bash
cd herramientas
echo "Actual"
cat probar_1
git checkout d193ae6 probar_1
echo "Anterior"
cat probar_1
git checkout HEAD probar_1


I am working on the file "Presentaciones/Notas/06_Versionamiento_git.ipynb".
The first task is:
En caso de ser un repositorio nuevo o desear reinicializar (aceptar nuevas plantillas o mover el directorio), debemos crear el directorio .git y agregar los elementos que teniamos en nuestro directorio. Siempre que se agreguen (o se realiza cualquier actualización de los archivos y directorios, se sugiere realizar un commit para la indicación de la acción en nuestro historial).
Can you write Python code for it?


I am working on the file "Presentaciones/Notas/06_Versionamiento_git.ipynb".
The first task is:
Es posible indicar directamente a git la eliminación de archivos mediante
git rm archivo

o tambien al eliminar el archivo de manera directa, se incluye en git
git add -A

con el fin de actualizar el arbol de archivos y rastrear incluso los archivos eliminados. Sin embargo, es recomendable que la adición y remoción de archivos sea lo más especifica posible con el fin de tener un registro descriptivo que ayude a la adecuada solución de problemas o la descripción de cambios de una versión a otro de los códigos. Tengase en cuenta que en el caso de directorios, debe indicarse el argumento recursivo para el borrado.
Si se desea eliminar un archivo del repositorio pero no del disco, debe eliminarse de la cache del repositorio.
git rm --cached archivo

Posterior a esto, realizamos el commit asociado.
Actualizar de/desde repositorios remotos
Ahora deseamos actualizar nuestro repositorio remoto con nuestro repositorio local, para lo cual debemos hacer un push con el nombre del repositorio remoto y el nombre del branch que por defecto si solo hay uno es master. Es posible configurar el push para que su acción sea la de enviar todos los branchs del repositorio pero es preferible un control más fino con el fin de reducir el trafico de datos en la red.
Can you write Python code for it?

%%bash
cd herramientas
git push pruebas master


I am working on the file "Presentaciones/Notas/06_Versionamiento_git.ipynb".
The first task is:
Objetos y referencias
A continuación, y para evitar posibles conflictos de los archivos, es recomendable iniciar con un fetch o pull para obtener los objetos y referencias del repositorio remoto (esto si creamos el repositorio con los archivos de licencia y README.md que nos invita a crear el servicio git), y hacer la unión del contenido existente en el remoto con el local justo en el local.
git pull pruebas master

El paso anterior se evita si solo usamos el servicio para la creación del repositorio remoto vacio en lugar de poblarlo con los asistentes de archivo de licencia y archivo de descripción. Si en el paso anterior los archivos del repositorio remoto y el local poseen los mismos nombres pero son diferentes, pueden existir conflictos que deban resolverse de forma manual.
Crearemos archivos de prueba la adición al repositorio.
Can you write Python code for it?

%%bash
cd herramientas
echo "linea 1" > probar_1
printf "s\n 5" > probar_2


I am working on the file "Presentaciones/Notas/06_Versionamiento_git.ipynb".
The first task is:
Realizaremos una modificación en uno de los archivos y agregaremos directamente a este.
Can you write Python code for it?

%%bash
cd herramientas
echo "linea 2" >> probar_1
printf "otro\narchivo\npara probar." > probar_3
git add probar_1 probar_3
git commit -m "Actualizado prueba 1 y agregado prueba 3."


I am working on the file "Presentaciones/Notas/06_Versionamiento_git.ipynb".
The first task is:
Repositorios remotos
Una vez se inicializa o se clona, podemos agregar nuestro reposotorio remoto, que previamente fue creado en github o bitbucket. Si solo se desea un uso local o si es justo el repositorio clonado no será necesario.
Can you write Python code for it?

%%bash
cd herramientas
git remote add pruebas https://github.com/cosmoscalibur/pruebas.git
git remote -v


I am working on the file "Presentaciones/Notas/06_Versionamiento_git.ipynb".
The first task is:
Se evidencia que tras el clonado, tenemos configurados los repositorios remotos por defecto.
Can you write Python code for it?


I am working on the file "Presentaciones/Notas/06_Versionamiento_git.ipynb".
The first task is:
Tambien es posible eliminar el acceso al servidor remoto, con git remote rm.
Can you write Python code for it?

%%bash
cd herramientas
git remote rm origin
git remote -v


I am working on the file "Presentaciones/Notas/08_Procesamiento_texto.ipynb".
The first task is:
Cortar
Para extraer secciones especificas de lineas de los archivos usamos la instrucción cut.
Can you write Python code for it?

%%bash
ls -oh
ls -oh | grep -Ev '[[:alpha:]]+ [[:alnum:]]+\.ipynb|total' | grep -Ev '[a-z]{3}  ' > cortar
echo ":: archivo cortar seleccionando lineas ::"
cat cortar
echo ":: Horas enteras de los archivos ::"
cut -d " " -f 7 cortar | cut -c 1-2 > horas
cut -d " " -f 7 cortar | cut -c 4-5 > minutos
echo ":: Horas ::"
cat horas
echo ":: Minutos ::"
cat minutos


I am working on the file "Presentaciones/Notas/08_Procesamiento_texto.ipynb".
The first task is:
Edición
Es posible hacer "traducciones" con la consola con la instrucción tr. Esto corresponde a substituciones basadas en expresiones regulares.
Can you write Python code for it?

%%bash
echo "EsCrItUrA TeRrIbLe" | tr [:upper:] [:lower:]


I am working on the file "Presentaciones/Notas/08_Procesamiento_texto.ipynb".
The first task is:
Ordenamiento
Es posible ordenar lineas de archivos con los valores definidos en ASCII con la instrucción sort.
Can you write Python code for it?


I am working on the file "Presentaciones/Notas/08_Procesamiento_texto.ipynb".
The first task is:
Otras utilidades para habilitar sus expresiones regulares requieren indicación explicita incluso para el soporte POSIX. Ejemplo locate --regex '<regex>'. En less y vim una vez visualizamos el archivo, habilitamos la búsqueda de expresiones regulares con /.
Manipulación de texto
Concatenación de archivos
Para la concatenación de archivos en linux podemos usar la utilidad cat. Permite la unión de archivos y redirige su unión a la salida estandar.
Can you write Python code for it?

%%bash
printf "hola\nesto\nes\nuna\nprueba.\n" > prueba1
a=1
echo $a > prueba2
while [ $a -lt 5 ]; do
    let a=a+1
    echo $a >> prueba2
done
cat prueba1 prueba2 > prueba3
more prueba3


I am working on the file "Presentaciones/Notas/08_Procesamiento_texto.ipynb".
The first task is:
Para los metacaracteres ERE se tiene el siguiente comportamiento.
| Metacaracter | Descripción |
|:---:|:---:|
|sec|sec| Alternación Coincidencia con alguno de los patrones |
|<sec>?| Elemento precedente opcional (cero o una vez) |
|<sec>*| Elemento precedente opcional (cero o multiples veces) |
|<sec>+| Elemento precedente con repetición opcional (una o multiples veces) |
| <sec>{n,m} | Coincidencia del precedente mínimo $n$ veces y máximo $m$ veces. Si se usa uno de los dos con la presencia de la , aplica como solo mínimo o máximo según corresponda, y si es uno solo sin la , es las coincidencias exactas |
Para multiples aplicaciones el soporte de ERE debe indicarse de forma explicita, así por ejemplo para la utilidad con la cual hemos ejemplificado, debe usarse grep -E, siendo el argumento -E la indicación del soporte extendido. Igualmente hay utilidades con el soporte incluido, en este caso la alternativa es egrep.
Can you write Python code for it?

%%bash
echo ":: meta <sec>|<sec> ::"
grep -E '^2|.zip$' test_regexp
echo ":: meta <sec>? ::"
grep -E '^[a-z]?zip.' test_regexp
echo ":: meta <sec>* ::"
grep -E '^[a-z]*zip.' test_regexp
echo ":: meta <sec>+ ::"
grep -E '^[a-z]+zip.' test_regexp
echo ":: meta <sec>{n,m} ::"
grep -E '^[a-z]{1,3}zip.' test_regexp


I am working on the file "Presentaciones/Notas/08_Procesamiento_texto.ipynb".
The first task is:
Pegado
Para pegar columnas usamos la instrucción paste.
Can you write Python code for it?


I am working on the file "Presentaciones/Notas/08_Procesamiento_texto.ipynb".
The first task is:
Una forma más avanzada es posible con sed.
Can you write Python code for it?

%%bash
echo "escritura terrible" | sed 'y/esib/3516/'
echo "escritura terrible" | sed 's/terrible/hermosa/'


I am working on the file "Presentaciones/Notas/08_Procesamiento_texto.ipynb".
The first task is:
Unicidad
Es posible remover los elementos repetidos de un conjunto con utilidades de linux. Para esto, primero debe ordenarse la lista de elemento (sort) y aplicar uniq.
Can you write Python code for it?

%%bash
cat prueba3 prueba1 prueba2 prueba3 > repetido
cat repetido
echo ":: uniq sin sort ::"
uniq repetido
echo ":: uniq con sort ::"
sort repetido | uniq


I am working on the file "Presentaciones/Notas/09_Extraccion_web.ipynb".
The first task is:
Ante un fallo en el proceso de obtención del código con la función get, es posible generar una notificación del motivo de fallo.
Can you write Python code for it?

res = requests.get('http://github.com/yomeinventoesto')
res.raise_for_status()


I am working on the file "Presentaciones/Notas/09_Extraccion_web.ipynb".
The first task is:
Cuando obtenemos un elemento de una dirección, este se encuentra como binario y no como texto plano. Esto nos facilita algunas cosas. Nos permite descargar contenido que no se solo texto plano (archivos de texto o código fuente) sino tambien directamente archivos binarios como imagenes, ejecutables, videos, archivos de word y otros. Es importante aclarar, que si vamos a almacenar el archivo de texto plano, debemos hacerlo con creación de archivos binarios para no perder la codificación original que tenga el archivo.
Can you write Python code for it?

res = requests.get('http://www.programmableweb.com/sites/default/files/github-jupyter.jpg')
archivo_imagen = open('github-jupyter.jpg', 'wb')
for bloques in res.iter_content(100000):
    archivo_imagen.write(bloques)
archivo_imagen.close()


I am working on the file "Presentaciones/Notas/09_Extraccion_web.ipynb".
The first task is:
El resultado es una lista con todos los resultados obtenidos. Tambien es posible una búsqueda uno a uno, usando find en lugar de find_all.
Can you write Python code for it?


I am working on the file "Presentaciones/Notas/09_Extraccion_web.ipynb".
The first task is:
En el bloque anterior, el método iter_content genera bloques del archivo con el tamaño indicado en su argumento. Esto conviene para la escritura de archivos de gran tamaño.
Can you write Python code for it?


I am working on the file "Presentaciones/Notas/09_Extraccion_web.ipynb".
The first task is:
Resulta bastante útil el uso de selenium no tanto en los casos que requieran de interacción sino en los casos donde los contenidos (incluye elementos de interacción) son de generación dinámica o tras la interacción el nuevo enlace o contenido tiene retrasos apreciables, lo cual evitaría que Request obtenga el código adecuado. Podemos extraer el código fuente de la página en la cual se encuentra el foco del navegador de la siguiente forma.
Can you write Python code for it?

codigo = browser.page_source
print(codigo)


I am working on the file "Presentaciones/Notas/09_Extraccion_web.ipynb".
The first task is:
Sin embargo, la labor de extracción web depende de obtener el código fuente o elementos disponibles en las páginas, lo cual es imposible con solo abrir el navegador. Para este fin es posible usar urllib o como lo haremos en esta sesión, con request.
Can you write Python code for it?

import requests
res = requests.get('http://www.gutenberg.org/files/18251/18251-0.txt')
res.status_code == requests.codes.ok # Validar código 200 (ok)
type(res)
len(res.text)
print(res.text[:250])


I am working on the file "Presentaciones/Notas/09_Extraccion_web.ipynb".
The first task is:
Usarmos ahora bs4 (forma de importar Beautiful Soup), lo cual nos permitirá la búsqueda de texto y estructuras html especificas. Este es más conveniente que usar expresiones regulares directamente en el código fuente.
Al crear el objeto, debemos indicar el texto sobre el cual actuará (puede ser obtenido directamente de un archivo abierto tambien) y el tipo de analizador sintactico, en este caso lxml.
Can you write Python code for it?

res = requests.get('https://github.com/cosmoscalibur/herramientas_computacionales')
gh = bs4.BeautifulSoup(res.text, "lxml")
type(gh)


I am working on the file "Presentation.ipynb".
The first task is:
...and we can read the source code of built-in functions by downloading the source from the Python.org Mercurial repositories): https://hg.python.org/
Can you write Python code for it?

# import inspect
# inspect.getsourcefile(range) # doesn't work for built-in functions

# Python's built-in "Counter" class defines the control flow of iterators,
# used in functions such as "for i in range(0,10) ..."

class Counter(object):
    def __init__(self, low, high):
        self.current = low
        self.high = high

    def __iter__(self):
        'Returns itself as an iterator object'
        return self

    def __next__(self):
        'Returns the next value till current is lower than high'
        if self.current > self.high:
            raise StopIteration
        else:
            self.current += 1
            return self.current - 1


I am working on the file "Presentation.ipynb".
The first task is:
2. Tutorial 2: Never get in a data battle without Numpy arrays (Valerio Maggio)
IPython Notebook: https://github.com/leriomaggio/numpy_euroscipy2015
Arrays and Data Types
Can you write Python code for it?

# We can infer the data type of an array structure (but not of int, list, etc.)
import numpy as np

a = np.array([1, 2, 3], dtype=np.int16)
a.dtype

arev = a[::-1]
arev

# Typecast variables into float, complex numbers,
b = np.float64(64)
c = np.complex(b)
print "R(c) = ", c.real
print "I(c) = ", c.imag

# Specify type of array elements
x = np.ones(4, 'int8')
x

# Wrap-around
x[0] = 256
x

# Define a new record and create an array of corresponding data types
rt = np.dtype([('artist', np.str_, 40),('title', np.str_, 40), ('year', np.int16)])
music = np.array([('John Cage','4\'33\'\'',1952)], dtype=rt)
music


I am working on the file "Presentation.ipynb".
The first task is:
Accessing Python's Source Code
cf. http://stackoverflow.com/questions/8608587/finding-the-source-code-for-built-in-python-functions
We can get help for a built-in Python function, such as range, with a single question mark:
Can you write Python code for it?


I am working on the file "Presentation.ipynb".
The first task is:
Creating model in pymc3
Can you write Python code for it?

x_test_data = [test_obj.loc[21226,'Log(N_II/H_Alpha)'],test_obj.loc[21226,'Log(N_II/H_Alpha)_Sigma'],
               test_obj.loc[21226,'N_II'],test_obj.loc[21226,'N_II_Sigma'],
               test_obj.loc[21226,'H_Alpha'],test_obj.loc[21226,'H_Alpha_Sigma']
              ]
y_test_data = [test_obj.loc[21226,'Log(O_III/H_Beta)'],test_obj.loc[21226,'Log(O_III/H_Beta)_Sigma'],
               test_obj.loc[21226,'O_III'],test_obj.loc[21226,'O_III_Sigma'],
               test_obj.loc[21226,'H_Beta'],test_obj.loc[21226,'H_Beta_Sigma']
              ]

@sampled
def oiii_limit(xData,yData):
    x,x_sigma,x_a,x_a_sigma,x_b,x_b_sigma = xData
    y,y_sigma,y_a,y_a_sigma,y_b,y_b_sigma = yData

    model_x_a = pm.Normal('N_II', mu=x_a, sd = x_a_sigma)
    model_x_b = pm.Normal('H_Alpha', mu=x_b, sd = x_b_sigma)
    model_y_b = pm.Normal('H_Beta', mu=y_b, sd = y_b_sigma)
    #the upper bound of 0.94 comes from solving ratio OIII/HBeta
    model_y_a_limit = pm.Uniform('OIII_Uniform',lower =0 ,upper=0.9417)
    model_y_a_gauss = pm.Normal('OIII_Normal',mu = 0.9417, sd = y_a_sigma)
    y_use_limit = pm.Binomial('OIII_swap', n=1,p=0.5)
    model_y_a = pm.Normal('OIII', mu = tt.switch(y_use_limit,model_y_a_limit,model_y_a_gauss), sd =1)

    model_NII_HAlpha = pm.Deterministic('NII/HAlpha', model_x_a/model_x_b)
    model_OIII_HBeta = pm.Deterministic('OIII/HBeta', model_y_a/model_y_b)


I am working on the file "Presentation.ipynb".
The first task is:
Dane pochodzą z ręcznie tagowanego treebanku (korpusu anotowanego składniowo) opracowanego przez Zespół Inżynierii Lingwistycznej IPI PAN na bazie Narodowego Korpusu Języka Polskiego (Wawer, 2015).
Treebank liczy około 1431 zdania. (To dość mało).
Can you write Python code for it?

w2v_model = Word2Vec.load('w2v_allwiki_nkjp300_200.model')

# Import w2v's dictionary to a bag-of-words model
w2v_vocabulary = Dictionary()
w2v_vocabulary.doc2bow(w2v_model.vocab.keys(), allow_update=True)
print w2v_vocabulary.items()[:10]

# Initialize dicts for representing w2v's dictionary as indices and 200-dim vectors
w2indx = {v: k+1 for k, v in w2v_vocabulary.items()}
w2vec = {word: w2v_model[word] for word in w2indx.keys()}

w2v_vocabulary_size = len(w2indx) + 1
w2v_vocabulary_dimension = len(w2vec.values()[0])

def map_treebank_words_to_w2v_indices(treebank_data, w2indx):
    treebank_data_vec = []
    for sentence in treebank_data:
        vectorized_sentence = []
        for word in sentence:
            try:
                vectorized_sentence.append(w2indx[word])
            except KeyError:  # words absent in w2v model will be indexed as 0s
                vectorized_sentence.append(0)
        treebank_data_vec.append(vectorized_sentence)
    return treebank_data_vec

X_train = map_treebank_words_to_w2v_indices(X_train_raw, w2indx)
X_test = map_treebank_words_to_w2v_indices(X_test_raw, w2indx)

print X_test[4]

# Define numpy weights matrix for embedding layer
embedding_weights = np.zeros((w2v_vocabulary_size , w2v_vocabulary_dimension))
for word, index in w2indx.items():
    embedding_weights[index, :] = w2vec[word]

# max sentence length
max(
    len(max(X_train, key=lambda sentence: len(sentence))),
    len(max(X_test, key=lambda sentence: len(sentence)))
)

# Normalize sequences length to 40 (will be extended with 0s)
sentence_length = 40
X_train = sequence.pad_sequences(X_train, maxlen=sentence_length)
X_test = sequence.pad_sequences(X_test, maxlen=sentence_length)

y_train = sequence.pad_sequences(y_train_raw, maxlen=sentence_length, value=[0, 1])
y_test = sequence.pad_sequences(y_test_raw, maxlen=sentence_length, value=[0, 1])

# print X_train[2]
# print y_train[2]

inputs = Input(shape=(sentence_length,), dtype='int32')

x = Embedding(
    input_dim=w2v_vocabulary_size,
   ...


I am working on the file "Presentation.ipynb".
The first task is:
Handling Limits

Creating the model
Example Case:
Object has a limit in OIII so this is an upper limit in our y
Can you write Python code for it?

test_obj = df.iloc[[21226]]
test_obj[['Log(O_III/H_Beta)','Log(O_III/H_Beta)_Sigma','Log(N_II/H_Alpha)','Log(N_II/H_Alpha)_Sigma', 'O_III/H_Beta', 'H_Beta', 'H_Beta_Sigma', 'O_III']]

graph.BPT(test_obj)
test_obj[lim_columns]


I am working on the file "Presentation.ipynb".
The first task is:
How many object we lose if we only take well defined objects
Can you write Python code for it?

print('Number of Objects\n-------------\nAll Data: {}\nWell Def: {}'.format(df.shape[0],no_limits.shape[0]))
print('Loss Percentage: {0:.3f}%'.format((1-no_limits.shape[0]/df.shape[0])*100))


I am working on the file "Presentation.ipynb".
The first task is:
Images, data files, etc. should be stored in a subdirectory of the Notebook host.
1. Tutorial #1: An Introduction to Python (Joris Vankerschaver)
IPython Notebook: https://github.com/jvkersch/python-tutorial-files
An overview of basic Python syntax and data structures, including:
- lists, tuples, dictionaries
- mutable vs immutable objects
- set, enumerate
- read from / write to files
- namespaces
Can you write Python code for it?

# Mutable objects can be changed in place (e.g. lists),
# Immutable objects can NOT (e.g. ints, strings, tuples)
tup = ('a','0','@')
tup

tup[0] = 8

record = {}
record['first'] = 'Alan'
record['last'] = 'Turing'
record

record.update({'workplace':'Bletchley Park'})
record

# Comma after print statement removes implicit \n, prints to same line
for x in range(0,4):
    print x,

# xrange(#) more efficient than range(#), because:
# range() creates the whole sequence of numbers,
# while xrange() creates them as needed!

%timeit range(1000000)

%timeit xrange(1000000)

# Namespaces are evil
pi = 3.14
from numpy import pi
pi


I am working on the file "Presentation.ipynb".
The first task is:
Introduction
Meetup is a online social networking website that allows people to organise offline group meetings revolving around a common interest.
Platforms like Meetup and Eventbrite can help researchers investigate interest related to a certain topic, as it changes in time and in different geographic areas. An earlier
analysis by Nesta, which focussed on tech-related meetings, has shown how it is possible to analyse their geography, and how the group creation trends can be linked to the emergence of new technologies.
In this blog post I will present a preliminary investigation of Meetup data, investigating the following questions:

Where and when do people meet?
Is there a difference between results obtained from tech-related and non-tech related meetings?

Data
Since Meetup API Python library has been discontinued, I wrote an alternative API which includes rate-limiting and allows to obtain extensive amount of data from Meetup API.
I then collected the 200 most-populated cities and towns in the UK and in the world. I also collected meetup groups in the UK and event information, obtaining a dataset of 13844 groups, of which 1559 are tech-related and others belong to one of the following categories:
arts-culture career-business cars-motorcycles community-environment dancing education-learning fashion-beauty fitness food-drink games lgbt government-politics health-wellbeing hobbies-crafts language lifestyle book-clubs movies-film music new-age-spirituality outdoors-adventure paranormal parents-family pets-animals photography religion-beliefs sci-fi-fantasy singles socializing sports-recreation support writing

While I was also interested in obtaining the attendance information for all categories, to investigate the role of structural holes as inter-categories information brokers, I had to refrain from this because it would have taken more than a week to download this data.
In which cities is Meetup most popular?
The first question of this analysis is: how are members of Meetup distributed among cities? This...
Can you write Python code for it?


I am working on the file "Presentation.ipynb".
The first task is:
Limit cases

Limit:
$\text{flux} < \text{flux error}$
Lower Limit:
Let  $f = \frac{A}{B} \text{ where } B < \sigma_B \text{, so  } f = \frac{A}{\sigma_B} \text{ is an lower limit.}$
Upper Limit:
Let  $f = \frac{A}{B} \text{ where } A < \sigma_A \text{, so } f = \frac{\sigma_A}{B} \text{ is an upper limit.}$
1D Limit
Is a limit in only one ratio. For example a lower limit in x would have the following shape,

2D Limit
Is a limit in each ratio.  For example a lower limit in x and y would have the following shape,

Number of Limit cases

1D Limits:
Can you write Python code for it?


I am working on the file "Presentation.ipynb".
The first task is:
Matrix Tricks
Can you write Python code for it?

# Flatten a matrix into a 1-D array
r = np.array([[1, 2, 3], [4, 5, 6]])
r.ravel()

# Save a .csv file using arbitrary precision
M = np.random.rand(3,3)
np.savetxt("data/random-matrix2.csv", M, fmt='%.5f')

# Create a matrix using list comprehension
coolmx = np.array([[10*j+i for i in range(6)] for j in range(6)])
coolmx


I am working on the file "Presentation.ipynb".
The first task is:
Memmapping
@ https://github.com/leriomaggio/numpy_euroscipy2015/blob/master/05_Memmapping.ipynb
Machine Learning with SciKit-Learn

Applying the K-means algorithm to the Iris dataset
@ https://github.com/leriomaggio/numpy_euroscipy2015/blob/master/07_0_MachineLearning_Data.ipynb
Can you write Python code for it?

# Machine Learning in Python is a oneliner!
centroids, variance = vq.kmeans(data, 3)
#... after preparing the data and importing the scikit-learn package


I am working on the file "Presentation.ipynb".
The first task is:
Nie wygląda imponująco, ale...
* Przy zgadywaniu trafność wynosiłaby 0,0625% (= $1/40^2$),
* Maksymalny możliwy wynik oscyluje wokół 80%, bo taka jest średnia zgodność ludzkich anotacji (Ogneva, 2012).
Plany na przyszłość

Zwiększenie trafności przewidywań

Dodanie drugiej (i kolejnych) warstwy LSTM powinno umożliwić budowanie przez sieć bardziej złożonych hierarchicznych reprezentacji zależności w zdaniach
Wydłużenie treningu sieci przy wykonywaniu obliczeń przez GPU lub szybszą maszynę

Rozszerzenie problemu o inne trenowanie innych klasyfikatorów:

Ekstrakcja obiektów, których aspektom przypisywany jest wydźwięk
Klasyfikowanie wydźwięku (pozytywny lub negatywny) dla pary <obiekt, wydźwięk>

Kod źródłowy
Repozytorium jest dostępne pod adresem: https://github.com/tomekkorbak/lstm-for-aspect-based-sentiment-analysis
Użyte oprogramowanie

Keras (wysokopoziomowy wrapper na Theano)
Theano (implementacja sieci i jej treningu)
Scikit-learn (walidacja)
Gensim (model językowy Word2Vec)
Numpy (pomocnicze obliczenia numeryczne)

Bibliografia

Hochreiter, S. i Schmidhuber, J, (1997). Long Short-Term Memory, „Neural Computation”, 9 (8), ss. 1735-1780.
Mikolov, T.,  Sutskever, I.,  Chen, K., Corrado, G., i Dean, J. (2013a). Distributed Representations of Words and Phrases and their Compositionality. „Advances in Neural Information Processing Systems 26”, ss. 3111-3119.
Mikolov, T., Chen, K., Corrado, G. i Dean, J. (2013b). Efficient estimation of word representations in vector space.
Ogneva, M. (2012). How Companies Can Use Sentiment Analysis to Improve Their Business
Wawer, A. (2015). Towards Domain-Independent Opinion Target Extraction. „IEEE 15th International Conference on Data Mining Workshops”, ss. 1326-1331.

Dziękuję za uwagę
Can you write Python code for it?

hist.history

score, acc = model.evaluate(X_test, y_test,
                            batch_size=batch_size)
print('Test score:', score)
print('Test accuracy:', acc)


I am working on the file "Presentation.ipynb".
The first task is:
No limits in any ratio
Can you write Python code for it?

no_limits = df[(oiii_healthy & nii_healthy)]
graph.BPT(no_limits)


I am working on the file "Presentation.ipynb".
The first task is:
OIII in depth
Can you write Python code for it?

pm.traceplot(base_trace, varnames = ['OIII_Normal','OIII_Uniform'])

def plot_jointkde(x_pdf,y_pdf):
    def StarForming(x): return (1.3 + 0.61 / (x - 0.04))
    g = sns.jointplot(np.log10(x_pdf),np.log10(y_pdf),kind = 'kde',size=10)
    xSpan = np.linspace(-0.6, -0.1, num=200)
    g.ax_joint.plot(xSpan, StarForming(xSpan), 'k--',-0.31,-0.37, 'bo')
    g.set_axis_labels('Log(N_II/H_Aplha)','Log(O[III]/H_Beta)')


I am working on the file "Presentation.ipynb".
The first task is:
P(AGN) Result

Old:
Can you write Python code for it?


I am working on the file "Presentation.ipynb".
The first task is:
Point location Result

Old:
$$\log_{10}(\frac{\text{O}{\text{III}}}{\text{H}{\beta}}) =   -0.37$$
$$\log_{10}(\frac{\text{N}{\text{II}}}{\text{H}{\alpha}}) = -0.31 $$
New:
$$\log_{10}(\frac{\text{O}{\text{III}}}{\text{H}{\beta}}) = -0.67$$
$$\log_{10}(\frac{\text{N}{\text{II}}}{\text{H}{\alpha}}) = -0.31$$
Can you write Python code for it?

def test_percentAGN(x,x_sig,y,y_sig):
    def AGNLine(x): return 1.19 + 0.61 / (x - 0.47)
    def SFLine(x): return 1.3 + 0.61 / (x - 0.04)
    lineCorrectionAGN = 0.2
    lineCorrectionSF = -0.2

    x0_array = np.float64(np.ones((100,)) * -x)
    y0_array = np.float64(np.ones((100,)) * -y)
    xLineSpace = np.linspace((x - x_sig), (x + x_sig), 100)
    yLineSpace = np.linspace((y - y_sig), (y + y_sig),
                             100)[:, None]
    ellipse = ((xLineSpace - x) / x_sig)**2 + ((yLineSpace - y) / y_sig)**2 <= 1
    boolean_AGN_line = np.subtract(
        yLineSpace, AGNLine(xLineSpace)) > 0  # AGN fit line
    boolean_AGN_Correction = (np.subtract(xLineSpace, np.ones(
        (100, 100)) * lineCorrectionAGN) > 0)  # Fixes problem with the fit line
    # combines fit line and correction into single boolean array
    boolean_AGN_Mask = np.where(np.logical_or(
        boolean_AGN_line, boolean_AGN_Correction), True, False)

    # plt.imshow(boolean_AGN_Mask, origin = 'lower') #for debugging

    # Starforming Fit Lines
    boolean_SF_line = np.subtract(yLineSpace, SFLine(
        xLineSpace)) < 0  # Starforming fit line
    boolean_SF_Correction = (np.subtract(xLineSpace, np.ones(
        (100, 100)) * lineCorrectionSF) < 0)  # Fixes problem with the fit line
    # combines fit line and correction into single boolean array
    boolean_SF_Mask = np.where(np.logical_and(
        boolean_SF_line, boolean_SF_Correction), True, False)

    # Neither mask indicates areas that are neither AGN or SF
    boolean_Neither_Mask = np.where(np.logical_or(
        boolean_SF_Mask, boolean_AGN_Mask), False, True)
    # plt.imshow(boolean_SF_Mask, origin = 'lower') #for debugging

    # Creat probability distribution function based on array index locations
    xWeight = (np.exp(-(np.add(xLineSpace, x0_array)**2) *
                      (2 * x_sig**2)) * (2 * math.pi * x_sig**2))**2
    yWeight = (np.exp(-(np.add(yLineSpace, y0_array)**2) *
                      (2 * y_sig**2)) * (2 * math.pi * y_sig**2))**2
  ...


I am working on the file "Presentation.ipynb".
The first task is:
Run model
Can you write Python code for it?

with oiii_limit(xData = x_test_data,yData =y_test_data):
    base_trace = pm.sample(10000)


I am working on the file "Presentation.ipynb".
The first task is:
The spatial distribution of 1000 randomly chosen tech-related events.
Can you write Python code for it?


I am working on the file "Presentation.ipynb".
The first task is:
Uczenie
<img src="plot.png">
Can you write Python code for it?

# plt.rcParams['figure.figsize'] = (10,10)

# axes = plt.gca()
# x_min = hist.epoch[0]
# x_max = hist.epoch[-1]+1
# axes.set_xlim([x_min,x_max])

# plt.scatter(hist.epoch, hist.history['acc'], color='r')
# plt.plot(hist.history['acc'], color='r', label=u'Trafność mierzona na zbiorze treningowym')
# plt.scatter(hist.epoch, hist.history['val_acc'], color='c')
# plt.plot(hist.history['val_acc'], color='c', label=u'Trafność mierzona na zbiorze walidacyjnym')
# plt.xlabel('epoki')
# plt.ylabel(u'Trafność')
# plt.title(u'Trafność w kolejnych epokach')
# plt.legend()
# plt.show()


I am working on the file "Presentation.ipynb".
The first task is:
Wartość bardzo przeszacowana ze względu na nierównomierną częśtość występowania klas (1:39 dla wydźwięku vs nie-wydźwięku)
Bardziej adekwatna metryka
Can you write Python code for it?

# Ułamek tokenów-wydźwięków, które poprawnie rozpoznano jako wydźwięki
float(accurately_evaluated_samples())/y_test.shape[0]


I am working on the file "Presentation.ipynb".
The first task is:
We can view the source code for a particular function, such as range, in the Python.org Mercurial Repository: https://hg.python.org/cpython/file/c6880edaf6f3/Objects/rangeobject.c
Can you write Python code for it?

for i in range(0,10):
    print i*i

# List comprehension (with filter)
[a*2 for a in range(0,10) if a>3]


I am working on the file "Presentation.ipynb".
The first task is:
Word embeddings
Word embedding to rodzaj statystycznego modelu językowego, który reprezentuje słowa (rzadziej złożone frazy lub całe dokumenty) jako punkty w n-wymiarowej przestrzeni liniowej.
Bardzo pożądaną cechą tego mapowania jest to, że relacje geometryczne między punktami tej przestrzeni odwzorowują relacje semantyczne między kodowanymi słowami.
Model taki trenuje się na bardzo dużych korpusach, każąc mu rozpoznawać wzorce współwystępowania słów. Najczęściej używanym algorytmem jest Word2Vec, opracowany przez Google (Mikolov et al., 2013a)
Skorzystaliśmy z gotowego embeddingu opracowanego przez IPI PAN, wytrenowanego (z użyciem Word2Vec) na całej polskojęzycznej Wikipedii oraz trzystumilionowym zbalansowanym podkorpusie NKJP.
Can you write Python code for it?

# w modelu, który wykorzystaliśmy, słowa są reprezentowane jako
# 200-elementowe wektory 32-bitowych liczb zmiennoprzecinkowych
w2v_model['filozofia']

w2v_model['filozofia'].shape

w2v_model.similarity(u'filozofia', u'inżynieria')

w2v_model.similarity(u'filozofia', u'nauka')

w2v_model.similarity(u'filozofia', u'literatura')

# wskaż słowo niepasujące do pozostałych
w2v_model.doesnt_match(['Kant', 'Leibniz', 'Derrida', 'Wittgenstein'])

# Kobieta + król - mężczyzna = królowa
# Medialny przykład z (Mikolov et al., 2013b)
w2v_model.most_similar(positive=[u'kobieta', u'król'], negative=[u'mężczyzna'])

# Paryż - Francja + Polska = Warszawa
w2v_model.most_similar(positive=[u'Paryż', u'Polska'], negative=[u'Francja'])

# filozofia - logika = literatura
w2v_model.most_similar(positive=[u'filozofia',], negative=[u'logika'])

# filozofia - postmodernizm = wiedza
w2v_model.most_similar(positive=[u'filozofia',], negative=[u'postmodernizm'])


I am working on the file "Presentations/BridgingTheGap.ipynb".
The first task is:
Aha, this is good.  You can't value a FRA with a discounting model because its cashflow depends on 3 month Jibar and your model does not know anything about 3 month Jibar.
With this type of constraint (which is deeply embedded in the library):

You will never work under the wrong numeraire again
You will never use the wrong curve to forecast a rate or asset price
You will never incorrectly combine cashflows in different currencies
etc.

For our problem at hand we need to fix the model by setting it up to forecast some rates:
Can you write Python code for it?

# add a forecast curve
forwardRates = [0.070614, 0.070614]
forecastCurve = ForecastCurve(valueDate, FloatingIndex.JIBAR3M, dates, forwardRates) # use flat 7% rates for forecasting
numeraireModel.AddRateForecast(forecastCurve) # add the forecast curve to the model

# value the product
portfolio = [fra]
value = coordinator.Value(portfolio, valueDate)
print("value is: {:.2f}".format(value))


I am working on the file "Presentations/BridgingTheGap.ipynb".
The first task is:
And just like that the cashflow definition can be turned into a value.
Same Product Different Model
I have hinted that models and products are independent.
Here is a demonstration of the same FRA with a Hull White model instead of deterministic curve discounting:
Can you write Python code for it?

valueDate = Date(2016, 9, 17)
flatRate = 0.07
newModel = HullWhite1F(Currency.ZAR, 0.05, 0.01, flatRate, flatRate, valueDate)
# tell HW model it is allowed to make some forecasts
newModel.AddForecast(FloatingIndex.JIBAR3M)

newCoordinator = Coordinator(newModel, List[Simulator](), 100000)
value = newCoordinator.Value(portfolio, valueDate)
print("value with the new model is: {:.2f}".format(value))


I am working on the file "Presentations/BridgingTheGap.ipynb".
The first task is:
Early exercise
For completeness we should at least mention early exercise products.
Stopping times in quant finance are not complicated stochastic control problems.
The only exercise decisions that I have ever seen involve deciding to exercise or not at a set of dates.
Always, one knows the cashflows as functions of states of the world both
 * when one exercises, and
 * when one does not exercise .
The optimal stopping time for the person who owns this right is the one that chooses the alternative with the higher expected value.
This is again a general problem that does not need to be solved for each product and model.
Recall that we defined a product as:
$$ P = \left{ \left(X_1, u_1\right), ..., \left(X_M, u_M\right) \right} $$
Similarly we can define a product with early exercise as
$$
O = P_{noex} \text{ and } \left{ \left(Q_1, e_1\right), ..., \left(Q_M, e_M\right) \right}
$$
Where
 * the $Q_i$ are the products that will be exercised into if the optimal stopping time is equal to $e_i$, and
 * the cashflows in $P_{noex}$ will stop at the optimal stopping time.
The extra pieces that need to be implemented on a product are then:
cs
List<Product> GetPostExProducts();
List<Date> GetExerciseDates()
Code Example: Bermudan Swaption
For a Bermudan swaption, there are no cashflows before exercise.
Once exercise has taken place the cashflows are those of a swap.
We have a swap from above, this forms the $Q_i$ at each of the exercise dates
We simply need to set up the exercise dates:
Can you write Python code for it?

# Make a Bermudan Swaption
exDates = List[Date]()
exDates.Add(Date(2017, 9, 17))
exDates.Add(Date(2018, 9, 17))
exDates.Add(Date(2019, 9, 17))
exDates.Add(Date(2020, 9, 17))
bermudan = BermudanSwaption(swap, exDates, True)

# Get the simulated forward values and the regressors used to obtain them
valuePaths = hwCoordinator.GetValuePaths([bermudan], valueDate, csDates)
fwdCashflowPVs = qsa.getnumpy(valuePaths.Get("fwdCashflowPVs"))
regressor0 = qsa.getnumpy(valuePaths.Get("regressor0"))
regressedFwdsPVs = qsa.getnumpy(valuePaths.Get("regressedFwdsPVs"))

# Examine the values after optimal exercise rule is applied
fig = plt.figure()
ax = fig.add_subplot(1, 1, 1)
ax.set_xlim([0, 0.15])
ax.set_ylim([-150000, 100000])
col = 20 # up to 181.
plt.plot(regressor0[:,col], fwdCashflowPVs[:,col],'.')
plt.plot(regressor0[:,col], regressedFwdsPVs[:,col],'k.')


I am working on the file "Presentations/BridgingTheGap.ipynb".
The first task is:
Higher Order Measures
xVA etc.
Most "valuations" that we perform are based on:
 * Fair value simulations,
 * Possibly some other market observables (eg. an FVA could need a funding rate)
 * A operation on these two
Example 1: The expected positive exposure depends on
 * expected future fair values
 * no other market observables
 * a simple positive operator
Example 2: The CVA depends on:
 * expected future fair values
 * A default indicator on each path at each time
 * The product of default on a path at a time and the positive part of the future value at the same point
Again these are separate from the models and the products.
If you implement a way to estimate, say, the pricing impact of initial margin it will work with anybody else's models and or products.
Code Example: Expected Positive Exposure
Can you write Python code for it?

positive_mtm = regressedFwdsPVs
positive_mtm[positive_mtm<0] = 0
epe = np.mean(positive_mtm, 0)
plt.plot(epe)


I am working on the file "Presentations/BridgingTheGap.ipynb".
The first task is:
Implementing a Model
I won't spend much time describing how to implement models, that remains roughly the same as in the "olden days":

You calibrate to market data,
obtain some parametrization of the stochastic processes $\textbf{W}(t)$ and functions $S_i(t)$, and then
simulate the $S$s.

The coordinating component moves the simulated values and cashflows between the products and models.
The canonical numerical scheme is Monte Carlo.
 * For many real world pricing problems Monte Carlo is the only feasible option once
     * sufficiently realistic dynamics and
     * enough of the side effects are modelled.
Other numerical schemes can be implemented:
 * The first FRA model above uses the same machinery but is a curve based valuation.
     * The curve based model is fast even in this general framework:
     * In a test it valued a large portfolio of swaps averaging 0.5ms per swap.
 * In principle other numerical schemes could also be possible
Expectation via Simulation
The simulation based valuation boils down to estimating:
$$ V(t_0) =  \mathbb{E}^\mathbb{Q} \left[ \sum_{u_i>t_0}{\frac{S_{xn}(u_i)X_i}{N(u_i)}} \right] $$
as
$$ V(t_0) \approx \frac{1}{j_{max}+1}  \sum_{j=0}^{j_{max}} \left[ \sum_{u_i>t_0}{\frac{S^{(j)}_{xn}(u_i)X^{(j)}_i}{N^{(j)}(u_i)}} \right] $$
The deterministic model does one simulation
Forward Values
In addition to the cashflows on a product that are explicitly defined by the bilateral contract, there are many other financial effects of trading a product such as:

The need to fund the cashflows
The capital required to be held against the position
The loss in the event the counterparty defaults
The need to place collateral and fund that collateral position
The gain in the event that we default ourselves...

In general these depend on the bank's fair value of these products at future dates
Simulating forward values
If we require the forward value at time $t_i > t_0$ we need to evaluate
$$
V(t_i) = N(t_i)\mathbb{E}^\mathbb{Q}\left[ \sum_{u_i>t_i}{\frac{S_{xn}(u_i)X_i}{N(u_i)}} \middle|...
Can you write Python code for it?

# Set up a swap, which has a more interesting value profile than a FRA
rate = 0.08
payFixed = True
notional = 1000000
startDate = Date(2016, 9, 17)
tenor = Tenor.Years(5)
swap = IRSwap.CreateZARSwap(rate, payFixed, notional, startDate, tenor)
print("Now we have a swap:")
print(swap)

# Set up a stochastic model
valueDate = Date(2016, 9, 17)
a = 0.05
vol = 0.01
flatCurveRate = 0.07
hullWiteSim = HullWhite1F(Currency.ZAR, a, vol, flatCurveRate, flatCurveRate, valueDate)
hullWiteSim.AddForecast(FloatingIndex.JIBAR3M)
hwCoordinator = Coordinator(hullWiteSim, List[Simulator](), 5000)

print("A stochastic rate model is ready (Hull White).")

# make the forward dates on which the values are required
from datetime import datetime
from datetime import timedelta
step = timedelta(days=10)
pyDates = []
date = datetime(2016, 9, 17)
endDate = datetime(2021,9,17)
while date<endDate:
    date += step
    pyDates.append(date)
csDates = [Date(d.year, d.month, d.day) for d in pyDates]

# Get the simulated forward values and the regressors used to obtain them
hwCoordinator.SetThreadedness(True)
valuePaths = hwCoordinator.GetValuePaths([swap], valueDate, csDates)
print("Available data:")
for s in valuePaths.GetNames():
    print("   " + s)

import sys
sys.path.insert(0, r'..\Python')
import quantsa as qsa

fwdCashflowPVs = qsa.getnumpy(valuePaths.Get("fwdCashflowPVs"))
regressor0 = qsa.getnumpy(valuePaths.Get("regressor0"))
regressedFwdsPVs = qsa.getnumpy(valuePaths.Get("regressedFwdsPVs"))

%matplotlib inline
import matplotlib.pyplot as plt

fig = plt.figure()
ax = fig.add_subplot(1, 1, 1)
ax.set_xlim([0, 0.15])
ax.set_ylim([-150000, 100000])
col = 17 # up to 181
plt.plot(regressor0[:,col], fwdCashflowPVs[:,col],'.')
plt.plot(regressor0[:,col], regressedFwdsPVs[:,col],'k.')


I am working on the file "Presentations/BridgingTheGap.ipynb".
The first task is:
Is the value right?
Can you write Python code for it?

# check the value
import numpy as np
date = Date(2017, 8, 28)
t = (date.value - valueDate.value) / 365.0 # C# operator overloading does not work in Python
dt = 91.0 / 365.0
fixedRate = 0.069
notional = 1000000.0
fwdRate = 0.070614
refValue = (notional * (fwdRate - fixedRate) * dt / (1 + fwdRate * dt) *
            np.exp(-t * 0.07))

print("value is: {:.2f}.  Expected {:.2f}".format(value, refValue))


I am working on the file "Presentations/BridgingTheGap.ipynb".
The first task is:
Making a product
Can you write Python code for it?

source = """Date date = new Date(2017, 8, 28);
FloatingIndex jibar = FloatingIndex.JIBAR3M;
double dt = 91.0/365.0;
double fixedRate = 0.069;
double notional = 1000000.0;
Currency currency = Currency.ZAR;

public override List<Cashflow> GetCFs()
{
    double reset = Get(jibar, date);
    double cfAmount = notional * (reset - fixedRate)*dt/(1+dt*reset);
    return new List<Cashflow>() { new Cashflow(date, cfAmount, currency) };
}
"""
# Make a product at runtime
fra = RuntimeProduct.CreateFromString("MyFRA", source);
print("Now we have a FRA:")
print(fra)


I am working on the file "Presentations/BridgingTheGap.ipynb".
The first task is:
Setting up a model
We haven't described how to make a model or exactly what it does but the following code is fairly easy to understand:
Can you write Python code for it?

# Set up the model
valueDate = Date(2016, 9, 17)
maximumDate = Date(2026, 9, 17)

dates = [Date(2016, 9, 17) , Date(2026, 9, 17)]
rates = [ 0.07, 0.07 ]
discountCurve = DatesAndRates(Currency.ZAR, valueDate, dates, rates, maximumDate)

numeraireModel = DeterminsiticCurves(discountCurve);
otherModels = List[Simulator]() # no model other than discounting for now.
coordinator = Coordinator(numeraireModel, otherModels, 1) # the magic ingredient that gets
                                                          # models and products to work
                                                          # together

print("A model is ready.")


I am working on the file "Presentations/BridgingTheGap.ipynb".
The first task is:
Valuing the product with the model
Can you write Python code for it?

# Run the valuation
portfolio = [fra]

try:
   value = coordinator.Value(portfolio, valueDate)
except Exception as e:
    print(e)


I am working on the file "Presentations/Python_V_R/Python.ipynb".
The first task is:
Appears that siblings/spouses and parents/children aboard isn't that useful.
Can you write Python code for it?

# drop the two cols mentioned above and name
df_model = df.drop(df.columns[[-2,-3,2]], axis=1)
df_model.head()

# split into data (X) and target (y)
X = df_model.drop(['Survived'], axis=1)
y = df['Survived']
print("X: ", X.shape)
print("y: ", y.shape)

# split into training and test
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# define custom function to output test and training accuracy
def ACCscores(model):
    train_preds = model.predict(X_train)
    train_acc = acc(y_train, train_preds)
    print("Train Accuracy: ", train_acc)
    test_preds = model.predict(X_test)
    test_acc = acc(y_test, test_preds)
    print("Test Accuracy: ", test_acc)
    return (train_acc,test_acc)

# Logistic Regression
log_reg = LogisticRegression()
log_reg.fit(X_train, y_train)
ACCscores(log_reg)

log_reg.get_params

log_reg.coef_

import statsmodels.formula.api as smf
import statsmodels.api as sm

formula = "Survived~ Pclass + Sex + Age + Fare"
data = df[['Survived','Pclass','Sex','Age','Fare']]

model = smf.glm(formula=formula, data=data, family=sm.families.Binomial()).fit()
model.summary()


I am working on the file "Presentations/Python_V_R/Python.ipynb".
The first task is:
Exploratory Data Analysis
Let's explore the data!
Can you write Python code for it?

# check structure
print(df.shape)
print(df.dtypes)

# get the summary statistics (only for numeric)
df.describe()

# let's look at missing
df.isnull().sum()

# look at correlations
tmp = df.select_dtypes(include=["number"]).copy()
tmp_cor = tmp.corr()
plt.figure(figsize=(8,8))
sns.heatmap(tmp_cor, cmap="plasma")

# pairwise plot
plt.figure(figsize=(15,15))
sns.pairplot(tmp, hue="Survived", diag_kind="kde")

# investigate relationship between sex, survival rates and age
plt.figure(figsize=(8,8))
g = sns.swarmplot(x="Survived", y="Age",
                  hue="Sex", data=df)
g.set_title("Sex, Age and Survival")

# grouping and aggregating
df.groupby("Sex", as_index=False)[['Age','Fare']].agg(['mean'])

# filtering -
# I want to find all the female 3rd class passengers who were above 30
# how many survived?

df_filter = df[(df.Pclass == 3) & (df.Sex == 0) & (df.Age > 30)]
df_filter.Survived.value_counts(normalize=True)

# count table of survival by gender
cnt_table = pd.crosstab(df.Survived, df.Sex)
cnt_table.index = ['died','survived']
cnt_table

# frequency table of above
freq_table = pd.crosstab(df.Survived, df.Sex, margins=True)
freq_table.index = ['died','survived', 'total']
freq_table = freq_table/freq_table.loc['total']
freq_table

# convert sex to 1 (male) and 0 (female)
# using replace and dict method
cleanup_sex = {"Sex":{"male": 1, "female": 0},
                }
df.replace(cleanup_sex, inplace=True)
df.Sex.value_counts()

# look at groupby stats for passenger class and gender
stat = df.groupby(["Sex","Pclass"])['Survived'].agg(['count','sum'])
stat['prop'] = stat['sum']/stat['count']
stat.columns = ['Total','Survived','Proportion']
print(stat)
stat.plot(y='Proportion', kind='line', color='orange')


I am working on the file "Presentations/Python_V_R/Python.ipynb".
The first task is:
The story is the same for both genders, the lower the passenger class, the lower the survival rate
Modeling
Let's train some models and see how they perform in predicting Survival.
Can you write Python code for it?

from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score as acc
from sklearn.linear_model import LogisticRegression

# prepare data for modeling

tmp_cor = df.corr()
tmp_cor


I am working on the file "Pressureless_flow.ipynb".
The first task is:
Keep a close eye on the scale of the h-axis in the phase plane plot, and notice again that as $g \to 0$ the hugoniot loci become nearly parallel (horizontal in the $h-u$ plane).  This means that their intersection occurs at a very large value of $h$, so the middle state depth goes to $\infty$ as $g\to 0$.  Meanwhile, the speed of both shocks approaches a single value, so the region occupied by the middle state gets ever narrower.  In the limit, the solution for the depth involves a delta function.
To make this even clearer, here's an example with uniform initial depth and both flows directed inward, very similar to the first example we considered in the initial chapter on shallow water.
Can you write Python code for it?

q_l  = State(Depth = 1.,
             Momentum = 2.)
q_r = State(Depth = 1.,
            Momentum = -1.)

plot_riemann_SW(q_l,q_r)


I am working on the file "Pressureless_flow.ipynb".
The first task is:
Notice the following behaviors as $g \to 0$:

The integral curves and hugoniot loci become parallel.
All four curves become horizontal lines in the $h-u$ plane.

What happens to the hyperbolic structure of this system as $g \to 0$?  Recall that the flux jacobian is
\begin{align}
f'(q) & = \begin{pmatrix} 0 & 1 \ -(q_2/q_1)^2 + g q_1 & 2 q_2/q_1 \end{pmatrix}
        = \begin{pmatrix} 0 & 1 \ -u^2 + g h & 2 u \end{pmatrix},
\end{align}
with eigenvalues
\begin{align}
    \lambda_1 & = u - \sqrt{gh} & \lambda_2 & = u + \sqrt{gh},
\end{align}
and corresponding eigenvectors
\begin{align}
    r_1 & = \begin{bmatrix} 1 \ u-\sqrt{gh} \end{bmatrix} &
    r_2 & = \begin{bmatrix} 1 \ u+\sqrt{gh} \end{bmatrix}.
\end{align}
As $g \to 0$, the eigenvalues both approach a single value of $\lambda_0 = u$ and the eigenvectors both approach
\begin{align}
    r_0 & = \begin{bmatrix} 1 \ u \end{bmatrix}.
\end{align}
Thus for $g=0$ there is a single eigenvalue $\lambda_0=u$ with algebraic multiplicity 2 but geometric multiplicity 1.  The flux Jacobian is defective and the system is no longer hyperbolic.  The integral curves are everywhere parallel to the eigenvector $r_0$, but this eigenvector points in a direction of constant $u$ and depends only on $u$, so the integral curves in the $h-hu$ plane are straight lines of constant $u$, all passing through the origin.
What about the Hugoniot loci?  Recall from the shallow water chapter that a shock with a jump in the depth of $\alpha = h-h_*$ must have a jump in the momentum of
\begin{align}
    h u & = h_ u_ + \alpha \left[u_ \pm \sqrt{gh_ \left(1+\frac{\alpha}{h_}\right)\left(1+\frac{\alpha}{2h_}\right)}\right]
\end{align}
Setting $g=0$ we obtain $hu = h_ u_ + (h-h_)u_$, so that $u=u_*$.  Thus the Hugoniot loci are the same as the integral curves and are lines of constant $u$.
In the Riemann problem, if the left and right states have the same initial velocity, then we can still connect them as $g \to 0$; in the limit, everything just moves along at that constant velocity.  Here's...
Can you write Python code for it?

q_l  = State(Depth = 3.,
             Momentum = 1.)
q_r = State(Depth = 1.,
            Momentum = 1./3)

plot_riemann_SW(q_l,q_r)


I am working on the file "Pressureless_flow.ipynb".
The first task is:
Now suppose we have a Riemann problem whose initial states have different velocities.  The only way to connect them is through a pair of rarefactions with a dry state in between.  Recall from our earlier treatment of the shallow water equations that a middle dry state occurred if $u_l + 2 \sqrt{gh_l} < u_r-2\sqrt{gh_r}$; for $g=0$ this reduces simply to the condition $u_l < u_r$.
Here's an example with $u_l < u_r$; use the second slider to see what happens as $g \to 0$.
Also, turn on the plots of the two characteristic families and notice how they become parallel as $g \to 0$.
Can you write Python code for it?

q_l  = State(Depth = 3.,
             Momentum = -1.)
q_r = State(Depth = 1.,
            Momentum = 2.)

plot_riemann_SW(q_l,q_r)