I am working on the file "BAMM.101x/datetime_objects.ipynb".
The first task is:

Obviously that's not going to work.

We can't do date operations on strings

Let's see what happens with datetime

Can you write Python code for it?

import datetime
d1 = datetime.date(2016,11,24)
d2 = datetime.date(2017,10,24)
max(d1,d2)

print(d2 - d1)


I am working on the file "BAMM.101x/datetime_objects.ipynb".
The first task is:

You can do arithmetic operations on datetime objects

You can use timedelta objects to calculate new dates or times from a given date
Can you write Python code for it?

today=datetime.date.today()
five_days_later=today+datetime.timedelta(days=5)
print(five_days_later)

now=datetime.datetime.today()
five_minutes_and_five_seconds_later = now + datetime.timedelta(minutes=5,seconds=5)
print(five_minutes_and_five_seconds_later)

now=datetime.datetime.today()
five_minutes_and_five_seconds_earlier = now+datetime.timedelta(minutes=-5,seconds=-5)
print(five_minutes_and_five_seconds_earlier)


I am working on the file "BAMM.101x/datetime_objects.ipynb".
The first task is:

datetime objects can check validity

A ValueError exception is raised if the object is invalid

Can you write Python code for it?

some_date=datetime.date(2015,2,29)
#some_date =datetime.date(2016,2,29)
#some_time=datetime.datetime(2015,2,28,23,60,0)


I am working on the file "BAMM.101x/datetime_objects.ipynb".
The first task is:

datetime.strftime

The strftime function flips the strptime function. It converts a datetime object to a string

with the specified format
Can you write Python code for it?

now = datetime.datetime.now()
string_now = datetime.datetime.strftime(now,'%m/%d/%y %H:%M:%S')
print(now,string_now)
print(str(now)) #Or you can use the default conversion


I am working on the file "BAMM.101x/datetime_objects.ipynb".
The first task is:
<li>But you can't use timedelta on time objects. If you do, you'll get a TypeError exception
Can you write Python code for it?

time_now=datetime.datetime.now().time() #Returns the time component (drops the day)
print(time_now)
thirty_seconds=datetime.timedelta(seconds=30)
time_later=time_now+thirty_seconds
#Bug or feature?

#But this is Python
#And we can always get around something by writing a new function!
#Let's write a small function to get around this problem
def add_to_time(time_object,time_delta):
    import datetime
    temp_datetime_object = datetime.datetime(500,1,1,time_object.hour,time_object.minute,time_object.second)
    print(temp_datetime_object)
    return (temp_datetime_object+time_delta).time()

#And test it
time_now=datetime.datetime.now().time()
thirty_seconds=datetime.timedelta(seconds=30)
print(time_now,add_to_time(time_now,thirty_seconds))


I am working on the file "BAMM.101x/datetime_objects.ipynb".
The first task is:
<li>How much time has passed?
Can you write Python code for it?


I am working on the file "BAMM.101x/datetime_objects.ipynb".
The first task is:

datetime objects understand time

The datetime library contains several useful types

date: stores the date (month,day,year)

time: stores the time (hours,minutes,seconds)

datetime: stores the date as well as the time (month,day,year,hours,minutes,seconds)

timedelta: duration between two datetime or date objects

datetime.date

Can you write Python code for it?

import datetime
century_start = datetime.date(2000,1,1)
today = datetime.date.today()
print(century_start,today)
print("We are",today-century_start,"days into this century")
print(type(century_start))
print(type(today))


I am working on the file "BAMM.101x/machine_learning_using_regression.ipynb".
The first task is:

Build the model and fit the training data

Can you write Python code for it?

model = linear_model.LinearRegression()
model.fit(x_train,y_train)


I am working on the file "BAMM.101x/machine_learning_using_regression.ipynb".
The first task is:

Confusion matrix (and hence precision, recall etc.) depend on the selected threshold

As the threshold changes, we will need to tradeoff precision and recall

Can you write Python code for it?

[tp, fn, fp, tn] = confusion_matrix(testing_predictions,np.array(y_test),0.9)
precision = tp/(tp+fp)
recall = tp/(tp+fn)
f_score = 2 * (precision * recall)/(precision + recall)
print(precision,recall,f_score)


I am working on the file "BAMM.101x/machine_learning_using_regression.ipynb".
The first task is:

Training a classifier on Rocks vs Mines

Can you write Python code for it?

import numpy
import random
from sklearn import datasets, linear_model
from sklearn.metrics import roc_curve, auc
import pylab as pl

import pandas as pd
from pandas import DataFrame
url="https://archive.ics.uci.edu/ml/machine-learning-databases/undocumented/connectionist-bench/sonar/sonar.all-data"
df = pd.read_csv(url,header=None)
df.describe()


I am working on the file "BAMM.101x/machine_learning_using_regression.ipynb".
The first task is:

We want to predict categories: Rocks or Mines

But we're actually getting a continuous value

Not the same thing. So R-Square probably doesn't mean a whole lot

We need to convert the conitnuous values into  categorical 1s and 0s. We can do this by fixing a threshold value between 0 and 1

Values greater than the threshold are 1 (Mines). Values less than or equal to the threshold are 0 (Rocks)

Confusion matrix

Reports the proportion of

true positive: predicts mine and is a mine

false positive: predicts mine and is not a mine

true negative: predicts not mine and is not a mine

false negative:Predicts not mine but turns out to be a mine (BOOM!)
Can you write Python code for it?

def confusion_matrix(predicted, actual, threshold):
    if len(predicted) != len(actual): return -1
    tp = 0.0
    fp = 0.0
    tn = 0.0
    fn = 0.0
    for i in range(len(actual)):
        if actual[i] > 0.5: #labels that are 1.0  (positive examples)
            if predicted[i] > threshold:
                tp += 1.0 #correctly predicted positive
            else:
                fn += 1.0 #incorrectly predicted negative
        else:              #labels that are 0.0 (negative examples)
            if predicted[i] < threshold:
                tn += 1.0 #correctly predicted negative
            else:
                fp += 1.0 #incorrectly predicted positive
    rtn = [tp, fn, fp, tn]

    return rtn

testing_predictions = model.predict(x_test)

testing_predictions = model.predict(x_test)
confusion_matrix(testing_predictions,np.array(y_test),0.5)


I am working on the file "BAMM.101x/machine_learning_using_regression.ipynb".
The first task is:

Example: Let's say

Everything classified as a rock will be assumed a rock and if wrong, will cost 5000 in injuries

Everything classified as a mine will be left as is (no one will walk on it!)

Can you write Python code for it?

cm = confusion_matrix(testing_predictions,np.array(y_test),.1)
cost1 = 0*cm[0] + 0 * cm[2] + 5000 * cm[1] + 0 * cm[3]
cm = confusion_matrix(testing_predictions,np.array(y_test),.9)
cost2 = 0*cm[0] + 0 * cm[2] + 5000 * cm[1] + 0 * cm[3]
print(cost1,cost2)


I am working on the file "BAMM.101x/machine_learning_using_regression.ipynb".
The first task is:

Misclassification rate = (fp + fn)/number of cases

Can you write Python code for it?

cm = confusion_matrix(testing_predictions,np.array(y_test),0.5)
misclassification_rate = (cm[1] + cm[2])/len(y_test)
misclassification_rate


I am working on the file "BAMM.101x/machine_learning_using_regression.ipynb".
The first task is:

Pandas scatter matrix function helps visualize the relationship between features

Use with care though, because it is processor intensive
Can you write Python code for it?

from pandas.tools.plotting import scatter_matrix
p=scatter_matrix(w_df, alpha=0.2, figsize=(12, 12), diagonal='kde')


I am working on the file "BAMM.101x/machine_learning_using_regression.ipynb".
The first task is:

Examine correlations

Can you write Python code for it?

df.corr()

import matplotlib.pyplot as plot
plot.pcolor(df.corr())
plot.show()

df.corr()[0].plot()


I am working on the file "BAMM.101x/machine_learning_using_regression.ipynb".
The first task is:

Examine the distribution of the data in column 4

Quartile 1: from .0067 to .03805

Quartile 2: from .03805 to .0625

Quartile 3: from .0625 to .100275

Quartile 4: from .100275 to .401

Quartile 4 is much larger than the other quartiles. This raises the possibility of outliers

 A Quantile - Quantile (qq) plot can help identify outliers

y-axis contains values

x-axis is the cumulative normal density function plotted as a straight line (-3 to +3)

y-axis is the values ordered from lowest to highest

the closer the curve is to the line, the more it reflects a normal distribution
Can you write Python code for it?

import numpy as np
import pylab
import scipy.stats as stats
import matplotlib
import matplotlib.pyplot as plt
matplotlib.style.use('ggplot')
%matplotlib inline

stats.probplot(df[4], dist="norm", plot=pylab)
pylab.show()


I am working on the file "BAMM.101x/machine_learning_using_regression.ipynb".
The first task is:

In-sample ROC Curve

Can you write Python code for it?

(fpr, tpr, thresholds) = roc_curve(y_train,training_predictions)
area = auc(fpr,tpr)
pl.clf() #Clear the current figure
pl.plot(fpr,tpr,label="In-Sample ROC Curve with area = %1.2f"%area)

pl.plot([0, 1], [0, 1], 'k') #This plots the random (equal probability line)
pl.xlim([0.0, 1.0])
pl.ylim([0.0, 1.0])
pl.xlabel('False Positive Rate')
pl.ylabel('True Positive Rate')
pl.title('In sample ROC rocks versus mines')
pl.legend(loc="lower right")
pl.show()


I am working on the file "BAMM.101x/machine_learning_using_regression.ipynb".
The first task is:

See all columns

Can you write Python code for it?


I am working on the file "BAMM.101x/pandas.ipynb".
The first task is:

Getting historical stock prices from Google financs

Usage: DataReader(ticker,source,startdate,enddate)

Unfortunately, the Yahoo finance datareader has stopped working because of a change to Yahoo's website
Can you write Python code for it?

from pandas_datareader import data
import datetime as dt
start=dt.datetime(2017, 1, 1)
end=dt.datetime.today()

print(start,end)

df = data.DataReader('IBM', 'google', start, end)

df


I am working on the file "BAMM.101x/pandas.ipynb".
The first task is:

Linear regression with pandas

Example: TAN is the ticker for a solar ETF. FSLR, RGSE, and SCTY are tickers of companies that build or lease solar panels. Each has a different business model. We'll use pandas to study the risk reward tradeoff between the 4 investments and also see how correlated they are

Can you write Python code for it?

import datetime
import pandas_datareader as data
start = datetime.datetime(2015,7,1)
end = datetime.datetime(2016,6,1)
solar_df = data.DataReader(['FSLR', 'TAN','RGSE','SCTY'],'google', start=start,end=end)['Close']

solar_df


I am working on the file "BAMM.101x/pandas.ipynb".
The first task is:

Pandas datareader

Access data from html tables on any web page

Get data from google finance

Get data from the federal reserve

HTML Tables

Pandas datareader can read a table in an html page into a dataframe

the read_html function returns a list of all dataframes with one dataframe for each html table on the page

Example: Read the tables on the google finance page

Can you write Python code for it?

#df_list = pd.read_html('http://www.bloomberg.com/markets/currencies/major')
df_list = pd.read_html('http://www.waihuipaijia.cn/'
                      , encoding='utf-8')
print(len(df_list))


I am working on the file "BAMM.101x/pandas.ipynb".
The first task is:

Regressions

http://statsmodels.sourceforge.net/

Steps for regression

Construct y (dependent variable series)

Construct matrix (dataframe) of X (independent variable series)

Add intercept

Model the regression

Get the results

The statsmodels library contains various regression packages. We'll use the OLS (Ordinary Least Squares) model
Can you write Python code for it?

import numpy as np
import statsmodels.api as sm
X=solar_df[['FSLR','RGSE','SCTY']]
X = sm.add_constant(X)
y=solar_df['TAN']
model = sm.OLS(y,X,missing='drop')
result = model.fit()
print(result.summary())


I am working on the file "BAMM.101x/pandas.ipynb".
The first task is:

The structure of a dataframe

Can you write Python code for it?


I am working on the file "BAMM.101x/pandas.ipynb".
The first task is:

Accessing columns and rows

Can you write Python code for it?

df = pd.DataFrame([['r1','00','01','02'],['r2','10','11','12'],['r3','20','21','22']],columns=['row_label','A','B','C'])
print(id(df))
df.set_index('row_label',inplace=True)
print(id(df))
df

data = {'nationality': ['UK', 'China', 'US', 'UK', 'Japan', 'China', 'UK', 'UK', 'Japan', 'US'],
        'age': [25, 30, 15, np.nan, 25, 22, np.nan,45 ,18, 33],
        'type': [1, 3, 2, 3, 2, 3, 1, 1, 2, 1],
        'diabetes': ['yes', 'yes', 'no', 'yes', 'no', 'no', 'no', 'yes', 'no', 'no']}

labels = ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j']
df=pd.DataFrame(data=data,index=labels)
#print(df[df['age'].between(20,30)])
#print(df.groupby('nationality').mean()['age'])
#print(df.sort_values(by=['age','type'],ascending=[False,True]))
df['nationality'] = df['nationality'].replace('US','United States')
print(df)

df.ix[1]


I am working on the file "BAMM.101x/pandas.ipynb".
The first task is:

Basic risk analysis

We'll plot the mean and std or returns for each ticker to get a sense of the risk return profile

Can you write Python code for it?

plt.scatter(rets.mean(), rets.std())
plt.xlabel('Expected returns')
plt.ylabel('Standard deviations')
for label, x, y in zip(rets.columns, rets.mean(), rets.std()):
    plt.annotate(
        label,
        xy = (x, y), xytext = (20, -20),
        textcoords = 'offset points', ha = 'right', va = 'bottom',
        bbox = dict(boxstyle = 'round,pad=0.5', fc = 'yellow', alpha = 0.5),
        arrowprops = dict(arrowstyle = '->', connectionstyle = 'arc3,rad=0'))
plt.show()


I am working on the file "BAMM.101x/pandas.ipynb".
The first task is:

Getting a row by row number

Can you write Python code for it?


I am working on the file "BAMM.101x/pandas.ipynb".
The first task is:

Getting column data

Can you write Python code for it?


I am working on the file "BAMM.101x/pandas.ipynb".
The first task is:

Getting row data

Can you write Python code for it?


I am working on the file "BAMM.101x/pandas.ipynb".
The first task is:

Rolling windows

"rolling" function extracts rolling windows

For example, the 21 period rolling window of the 13 period percent change
Can you write Python code for it?


I am working on the file "BAMM.101x/pandas.ipynb".
The first task is:

Slicing

Can you write Python code for it?


I am working on the file "BAMM.101x/pandas.ipynb".
The first task is:

Calculate the percentage of days that the stock has closed higher than its open

Can you write Python code for it?


I am working on the file "BAMM.101x/pandas.ipynb".
The first task is:
<h4>Finally plot the fitted line with the actual y values
Can you write Python code for it?

fig, ax = plt.subplots(figsize=(8,6))
ax.plot(y)
ax.plot(result.fittedvalues)


I am working on the file "BAMM.101x/pandas.ipynb".
The first task is:

Let's calculate returns (the 1 day percent change)

Can you write Python code for it?

rets = solar_df.pct_change()
print(rets)


I am working on the file "BAMM.101x/pandas.ipynb".
The first task is:

Note that the read_html function has automatically detected the header columns

If an index is necessary, we need to explicitly specify it

Can you write Python code for it?


I am working on the file "BAMM.101x/pandas.ipynb".
The first task is:

The correlation matrix

Can you write Python code for it?

solar_corr = rets.corr()
print(solar_corr)


I am working on the file "BAMM.101x/pandas.ipynb".
The first task is:

The page contains only one table so the read_html function returns a list of one element

Can you write Python code for it?


I am working on the file "BAMM.101x/sql_and_python.ipynb".
The first task is:

Set up a connection and create a cursor object

Can you write Python code for it?

db = pymysql.connect("localhost","root","None" ,database="schooldb")
cursor = db.cursor()


I am working on the file "BAMM.101x/text_mining.ipynb".
The first task is:

Often, a comparitive analysis helps us understand text better

Let's look at US Presidentinaugural speeches

Copy the files 2013-Obama.txt and 2017-Trump.txt to the nltk_data/corpora/inaugural directory. nltk_data should be under your home directory

Can you write Python code for it?


I am working on the file "BAMM.101x/text_mining.ipynb".
The first task is:

Similarity

Given a corpus of documents, when a new document arrives, find the document that is the most similar

Can you write Python code for it?

doc_list = [community_data,le_monde_data,amigos_data,heights_data]
all_text = community_data.raw() + le_monde_data.raw() + amigos_data.raw() + heights_data.raw()

documents = [doc.raw() for doc in doc_list]
texts = [[word for word in document.lower().split()
        if word not in STOPWORDS and word.isalnum()]
        for document in documents]
dictionary = corpora.Dictionary(texts)
corpus = [dictionary.doc2bow(text) for text in texts]

from gensim.similarities.docsim import Similarity
from gensim import corpora, models, similarities
lsi = models.LsiModel(corpus, id2word=dictionary, num_topics=2)
doc = """
Many, many years ago, I used to frequent this place for their amazing french toast.
It's been a while since then and I've been hesitant to review a place I haven't been to in 7-8 years...
but I passed by French Roast and, feeling nostalgic, decided to go back.

It was a great decision.

Their Bloody Mary is fantastic and includes bacon (which was perfectly cooked!!), olives,
cucumber, and celery. The Irish coffee is also excellent, even without the cream which is what I ordered.

Great food, great drinks, a great ambiance that is casual yet familiar like a tiny little French cafe.
I highly recommend coming here, and will be back whenever I'm in the area next.

Juan, the bartender, is great!! One of the best in any brunch spot in the city, by far.
"""
vec_bow = dictionary.doc2bow(doc.lower().split())
vec_lsi = lsi[vec_bow]
index = similarities.MatrixSimilarity(lsi[corpus])
sims = index[vec_lsi]
sims = sorted(enumerate(sims), key=lambda item: -item[1])

sims

doc="""
I went to Mexican Festival Restaurant for Cinco De Mayo because I had been there years
prior and had such a good experience. This time wasn't so good. The food was just
mediocre and it wasn't hot when it was brought to our table. They brought my friends food out
10 minutes before everyone else and it took forever to get drinks. We let it slide because the place was
packed with people and it was Cinco De Mayo. Also, the margaritas we had...


I am working on the file "BAMM.101x/text_mining.ipynb".
The first task is:

Topic modeling

The goal of topic modeling is to identify the major concepts underlying a piece of text

Topic modeling uses "Unsupervised Learning". No apriori knowledge is necessary

Though it is helpful in cleaning up results!

LDA: Latent Dirichlet Allocation Model

Identifies potential topics using pruning techniques like 'upward closure'

Computes conditional probabilities for topic word sets

Identifies the most likely topics

Does this over multiple passes probabilistically picking topics in each pass

Good intuitive explanation: http://blog.echen.me/2011/08/22/introduction-to-latent-dirichlet-allocation/
Can you write Python code for it?

from gensim import corpora
from gensim.models.ldamodel import LdaModel
from gensim.parsing.preprocessing import STOPWORDS
import pprint


I am working on the file "BAMM.101x/text_mining.ipynb".
The first task is:

Working with text!

Sentiment Analysis

Identify entities and emotions in a sentence and use these to determine if the entity is being viewed positively or negatively

Easy examples

I had an excellent souffle at the restaurant Cavity Maker

Excellent is a positive word for both the souffle as well as for the restaurant

Not so easy examples

Often, looking at words alone is not enough to figure out the sentiment

The Girl on the Train is an excellent book for a ‘stuck at home’ snow day

This one is easy since it includes an explicit positive opinion using a positive word

The Girl on the Train is an excellent book for using as a liner for your cat’s litter box

Not so simple! The positive word "excellent" is used with a negative connotation.

The Girl on the Train is better than Gone Girl

The positive word is used as a comparator. Whether the writer likes The Girl on the Train or not depends on what he or she thinks of Gone Girl

Bottom line

Sentiment analysis is generally a starting point in analyzing a text and is then coupled with other techniques (e.g., topic analysis)

Sentiment analysis is usually done using a corpus of positive and negative words

Some sources compile lists of positive and negative words

Others include the polarity - the degree of positivity or negativity - of each word

Sources of sentiment coded words

Hu and Liu's sentiment analysis lexicon: words coded as either positive or negative

http://ptrckprry.com/course/ssd/data/positive-words.txt

http://ptrckprry.com/course/ssd/data/negative-words.txt

NRC Emotion Lexicon: words coded into emotional categories (many languages)

http://saifmohammad.com/WebPages/NRC-Emotion-Lexicon.htm

SentiWordNet: Lists of words weighted...
Can you write Python code for it?

def get_words(url):
    import requests
    words = requests.get(url).content.decode('latin-1')
    word_list = words.split('\n')
    index = 0
    while index < len(word_list):
        word = word_list[index]
        if ';' in word or not word:
            word_list.pop(index)
        else:
            index+=1
    return word_list

#Get lists of positive and negative words
p_url = 'http://ptrckprry.com/course/ssd/data/positive-words.txt'
n_url = 'http://ptrckprry.com/course/ssd/data/negative-words.txt'
positive_words = get_words(p_url)
negative_words = get_words(n_url)


I am working on the file "BAMM.101x/text_mining.ipynb".
The first task is:

dispersion plots

Dispersion plots show the relative frequency of words over the text

Let's see how the frequency of some words has changed over the course of the republic

That should give us some idea of how the focus of the nation has changed

Can you write Python code for it?

text4.dispersion_plot(["government", "citizen", "freedom", "duties", "America",'independence','God','patriotism'])


I am working on the file "BAMM.101x/text_mining.ipynb".
The first task is:

A function that analyzes emotions

Can you write Python code for it?

def emotion_analyzer(text,emotion_dict=emotion_dict):
    #Set up the result dictionary
    emotions = {x for y in emotion_dict.values() for x in y}
    emotion_count = dict()
    for emotion in emotions:
        emotion_count[emotion] = 0

    #Analyze the text and normalize by total number of words
    total_words = len(text.split())
    for word in text.split():
        if emotion_dict.get(word):
            for emotion in emotion_dict.get(word):
                emotion_count[emotion] += 1/len(text.split())
    return emotion_count


I am working on the file "BAMM.101x/text_mining.ipynb".
The first task is:

Create the model

Can you write Python code for it?

lda = LdaModel(by_address_corpus,
              id2word=dictionary,
              num_topics=20,
              passes=10)

pp = pprint.PrettyPrinter(indent=4)
pp.pprint(lda.print_topics(num_words=10))


I am working on the file "BAMM.101x/text_mining.ipynb".
The first task is:

Matching topics to documents

Sort topics by probability

We're using sentences as documents here, so this is less than ideal

Can you write Python code for it?


I am working on the file "BAMM.101x/text_mining.ipynb".
The first task is:

Simple sentiment analysis using NRC data

NRC data codifies words with emotions

14,182 words are coded into 2 sentiments and 8 emotions

For example, the word abandonment is associated with anger, fear, sadness and has a negative sentiment

abandoned   anger   1

abandoned   anticipation    0

abandoned   disgust 0

abandoned   fear    1

abandoned   joy 0

abandoned   negative    1

abandoned   positive    0

abandoned   sadness 1

abandoned   surprise    0

abandoned   trust   0

Read the NRC sentiment data

Can you write Python code for it?

nrc = "data/NRC-emotion-lexicon-wordlevel-alphabetized-v0.92.txt"
count=0
emotion_dict=dict()
with open(nrc,'r') as f:
    all_lines = list()
    for line in f:
        if count < 46:
            count+=1
            continue
        line = line.strip().split('\t')
        if int(line[2]) == 1:
            if emotion_dict.get(line[0]):
                emotion_dict[line[0]].append(line[1])
            else:
                emotion_dict[line[0]] = [line[1]]


I am working on the file "BAMM.101x/text_mining.ipynb".
The first task is:

We can now compare presidential addresses by topic

Can you write Python code for it?

len(by_address_corpus)

from operator import itemgetter
sorted(lda.get_document_topics(by_address_corpus[0],minimum_probability=0,per_word_topics=False),key=itemgetter(1),reverse=True)

draw_wordcloud(lda,18)

print(lda.show_topic(12,topn=5))
print(lda.show_topic(18,topn=5))


I am working on the file "BAMM.101x/text_mining.ipynb".
The first task is:

Weighted word analysis using Vader

Vader contains a list of 7500 features weighted by how positive or negative they are

It uses these features to calculate stats on how positive, negative and neutral a passage is

And combines these results to give a compound sentiment (higher = more positive) for the passage

Human trained on twitter data and generally considered good for informal communication

10 humans rated each feature in each tweet in context from -4 to +4

Calculates the sentiment in a sentence using word order analysis

"marginally good" will get a lower positive score than "extremely good"

Computes a "compound" score based on heuristics (between -1 and +1)

Includes sentiment of emoticons, punctuation, and other 'social media' lexicon elements

Can you write Python code for it?

!pip install vaderSentiment

from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer

headers = ['pos','neg','neu','compound']
texts = restaurant_data
analyzer = SentimentIntensityAnalyzer()
for i in range(len(texts)):
    name = texts[i][0]
    sentences = sent_tokenize(texts[i][1])
    pos=compound=neu=neg=0
    for sentence in sentences:
        vs = analyzer.polarity_scores(sentence)
        pos+=vs['pos']/(len(sentences))
        compound+=vs['compound']/(len(sentences))
        neu+=vs['neu']/(len(sentences))
        neg+=vs['neg']/(len(sentences))
    print(name,pos,neg,neu,compound)


I am working on the file "BAMM.101x/text_mining.ipynb".
The first task is:

Comparing complexity of restaurant reviews won't get us anything useful

Let's look at something more useful

nltk: Python's natural language toolkit

ntlk documentation link:

http://www.nltk.org/api/nltk.html

Commands cheat sheet

https://blogs.princeton.edu/etc/files/2014/03/Text-Analysis-with-NLTK-Cheatsheet.pdf

nltk book

http://www.nltk.org/book/

nltk contains a large corpora of pre-tokenized text

Load it using the command:

nltk.download()

Import the corpora

Can you write Python code for it?


I am working on the file "BAMM.101x/text_mining.ipynb".
The first task is:

Making sense of the topics

Draw wordclouds

Can you write Python code for it?

def draw_wordcloud(lda,topicnum,min_size=0,STOPWORDS=[]):
    word_list=[]
    prob_total = 0
    for word,prob in lda.show_topic(topicnum,topn=50):
        prob_total +=prob
    for word,prob in lda.show_topic(topicnum,topn=50):
        if word in STOPWORDS or  len(word) < min_size:
            continue
        freq = int(prob/prob_total*1000)
        alist=[word]
        word_list.extend(alist*freq)

    from wordcloud import WordCloud, STOPWORDS
    import matplotlib.pyplot as plt
    %matplotlib inline
    text = ' '.join(word_list)
    wordcloud = WordCloud(stopwords=STOPWORDS,background_color='white',width=3000,height=3000).generate(' '.join(word_list))

    plt.imshow(wordcloud)
    plt.axis('off')
    plt.show()

draw_wordcloud(lda,2)


I am working on the file "BAMM.101x/text_mining.ipynb".
The first task is:

gensim: another text summarizer

Gensim uses a network with sentences as nodes and 'lexical similarity' as weights on the arcs between nodes

Can you write Python code for it?

from wordcloud import WordCloud, STOPWORDS
import matplotlib.pyplot as plt
%matplotlib inline
import nltk
from nltk.corpus import PlaintextCorpusReader
from nltk import sent_tokenize,word_tokenize
from nltk.book import *

import nltk
from nltk.corpus import PlaintextCorpusReader
community_root = "data/community"
le_monde_root = "data/le_monde"
community_files = "community.*"
le_monde_files = "le_monde.*"
heights_root = "data/heights"
heights_files = "heights.*"
amigos_root = "data/amigos"
amigos_files = "amigos.*"
community_data = PlaintextCorpusReader(community_root,community_files)
le_monde_data = PlaintextCorpusReader(le_monde_root,le_monde_files)
heights_data = PlaintextCorpusReader(heights_root,heights_files)
amigos_data = PlaintextCorpusReader(amigos_root,amigos_files)

type(community_data)

text = community_data.raw()
summary_sentences = []
candidate_sentences = {}
candidate_sentence_counts = {}
striptext = text.replace('\n\n', ' ')
striptext = striptext.replace('\n', ' ')

import gensim.summarization

#!pip install gensim

import gensim.summarization

summary = gensim.summarization.summarize(striptext, word_count=100)
print(summary)

print(gensim.summarization.keywords(striptext,words=10))

summary = '\n'.join(build_naive_summary(community_data.raw()))
print(summary)

text = le_monde_data.raw()
summary_sentences = []
candidate_sentences = {}
candidate_sentence_counts = {}
striptext = text.replace('\n\n', ' ')
striptext = striptext.replace('\n', ' ')
summary = gensim.summarization.summarize(striptext, word_count=100)
print(summary)
#print(gensim.summarization.keywords(striptext,words=10))


I am working on the file "BAMM.101x/text_mining.ipynb".
The first task is:

And functionalize this as well

Can you write Python code for it?

def vader_comparison(texts):
    from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer
    headers = ['pos','neg','neu','compound']
    print("Name\t",'  pos\t','neg\t','neu\t','compound')
    analyzer = SentimentIntensityAnalyzer()
    for i in range(len(texts)):
        name = texts[i][0]
        sentences = sent_tokenize(texts[i][1])
        pos=compound=neu=neg=0
        for sentence in sentences:
            vs = analyzer.polarity_scores(sentence)
            pos+=vs['pos']/(len(sentences))
            compound+=vs['compound']/(len(sentences))
            neu+=vs['neu']/(len(sentences))
            neg+=vs['neg']/(len(sentences))
        print('%-10s'%name,'%1.2f\t'%pos,'%1.2f\t'%neg,'%1.2f\t'%neu,'%1.2f\t'%compound)

vader_comparison(restaurant_data)


I am working on the file "BAMM.101x/text_mining.ipynb".
The first task is:

And let's functionalize the yelp stuff as well

Can you write Python code for it?

def analyze_nearby_restaurants(address,radius):
    lat,long = get_lat_lng(address)
    params = set_search_parameters(lat,long,radius)
    response = get_results(params)
    snippets = get_snippets(response)
    comparative_emotion_analyzer(snippets)

#And test it
analyze_nearby_restaurants("Community Food and Juice",200)

#Test it on some other place
analyze_nearby_restaurants("221 Baker Street",200)


I am working on the file "BAMM.101x/text_mining.ipynb".
The first task is:

Compute sentiment by looking at the proportion of positive and negative words in the text

Can you write Python code for it?

from nltk import word_tokenize
cpos = cneg = lpos = lneg = 0
for word in word_tokenize(community):
    if word in positive_words:
        cpos+=1
    if word in negative_words:
        cneg+=1
for word in word_tokenize(le_monde):
    if word in positive_words:
        lpos+=1
    if word in negative_words:
        lneg+=1
print("community {0:1.2f}%\t {1:1.2f}%\t {2:1.2f}%".format(cpos/len(word_tokenize(community))*100,
                                                        cneg/len(word_tokenize(community))*100,
                                                        (cpos-cneg)/len(word_tokenize(community))*100))
print("le monde  {0:1.2f}%\t {1:1.2f}%\t {2:1.2f}%".format(lpos/len(word_tokenize(le_monde))*100,
                                                        lneg/len(word_tokenize(le_monde))*100,
                                                        (lpos-lneg)/len(word_tokenize(le_monde))*100))


I am working on the file "BAMM.101x/text_mining.ipynb".
The first task is:

Construct a list of words after getting rid of unimportant ones and numbers

Can you write Python code for it?

words = word_tokenize(striptext)
lowercase_words = [word.lower() for word in words
                  if word not in stopwords.words() and word.isalpha()]


I am working on the file "BAMM.101x/text_mining.ipynb".
The first task is:

Construct word frequencies and choose the most common n (20)

Can you write Python code for it?

word_frequencies = FreqDist(lowercase_words)
most_frequent_words = FreqDist(lowercase_words).most_common(20)
pp = pprint.PrettyPrinter(indent=4)
pp.pprint(most_frequent_words)


I am working on the file "BAMM.101x/text_mining.ipynb".
The first task is:

Extract snippets

Can you write Python code for it?

all_snippets = list()
for business in response['businesses']:
    name = business['name']
    snippet = business['snippet_text']
    id = business['id']
    all_snippets.append((id,name,snippet))
all_snippets


I am working on the file "BAMM.101x/text_mining.ipynb".
The first task is:

Functionalize this

Can you write Python code for it?

def get_complexity(text):
    num_chars=len(text)
    num_words=len(word_tokenize(text))
    num_sentences=len(sent_tokenize(text))
    vocab = {x.lower() for x in word_tokenize(text)}
    return len(vocab),int(num_chars/num_words),int(num_words/num_sentences),len(vocab)/num_words

get_complexity(le_monde_data.raw())

for text in restaurant_data:
    (vocab,word_size,sent_size,vocab_to_text) = get_complexity(text[1])
    print("{0:15s}\t{1:1.2f}\t{2:1.2f}\t{3:1.2f}\t{4:1.2f}".format(text[0],vocab,word_size,sent_size,vocab_to_text))


I am working on the file "BAMM.101x/text_mining.ipynb".
The first task is:

Let's functionalize this

Can you write Python code for it?

def comparative_emotion_analyzer(text_tuples):
    print("%-20s %1s\t%1s %1s %1s %1s   %1s %1s %1s %1s"%(
            "restaurant","fear","trust","negative","positive","joy","disgust","anticip",
            "sadness","surprise"))

    for text_tuple in text_tuples:
        text = text_tuple[2]
        result = emotion_analyzer(text)
        print("%-20s %1.2f\t%1.2f\t%1.2f\t%1.2f\t%1.2f\t%1.2f\t%1.2f\t%1.2f\t%1.2f"%(
            text_tuple[1][0:20],result['fear'],result['trust'],
              result['negative'],result['positive'],result['joy'],result['disgust'],
              result['anticipation'],result['sadness'],result['surprise']))

#And test it
comparative_emotion_analyzer(all_snippets)


I am working on the file "BAMM.101x/text_mining.ipynb".
The first task is:

Let's look at the complexity of the speeches by four presidents

Can you write Python code for it?

texts = [('trump',inaugural.raw('2017-Trump.txt')),
         ('obama',inaugural.raw('2009-Obama.txt')+inaugural.raw('2013-Obama.txt')),
         ('jackson',inaugural.raw('1829-Jackson.txt')+inaugural.raw('1833-Jackson.txt')),
         ('washington',inaugural.raw('1789-Washington.txt')+inaugural.raw('1793-Washington.txt'))]
for text in texts:
    (vocab,word_size,sent_size,vocab_to_text) = get_complexity(text[1])
    print("{0:15s}\t{1:1.2f}\t{2:1.2f}\t{3:1.2f}\t{4:1.2f}".format(text[0],vocab,word_size,sent_size,vocab_to_text))


I am working on the file "BAMM.101x/text_mining.ipynb".
The first task is:

Now we can analyze the emotional content of the review snippets

Can you write Python code for it?

print("%-12s %1s\t%1s %1s %1s %1s   %1s %1s %1s %1s"%(
        "restaurant","fear","trust","negative","positive","joy","disgust","anticip",
        "sadness","surprise"))

for snippet in all_snippets:
    text = snippet[2]
    result = emotion_analyzer(text)
    print("%-12s %1.2f\t%1.2f\t%1.2f\t%1.2f\t%1.2f\t%1.2f\t%1.2f\t%1.2f\t%1.2f"%(
        snippet[1][0:10],result['fear'],result['trust'],
          result['negative'],result['positive'],result['joy'],result['disgust'],
          result['anticipation'],result['sadness'],result['surprise']))


I am working on the file "BAMM.101x/text_mining.ipynb".
The first task is:

Prepare the text

Can you write Python code for it?

text = PlaintextCorpusReader("data/","Nikon_coolpix_4300.txt").raw()
striptext = text.replace('\n\n', ' ')
striptext = striptext.replace('\n', ' ')
sentences = sent_tokenize(striptext)
#words = word_tokenize(striptext)
#tokenize each sentence into word tokens
texts = [[word for word in sentence.lower().split()
        if word not in STOPWORDS and word.isalnum()]
        for sentence in sentences]
len(texts)


I am working on the file "BAMM.101x/text_mining.ipynb".
The first task is:

Read the text being analyzed and count the proportion of positive and negative words in the text

Can you write Python code for it?

with open('data/community.txt','r') as f:
    community = f.read()
with open('data/le_monde.txt','r') as f:
    le_monde = f.read()


I am working on the file "BAMM.101x/text_mining.ipynb".
The first task is:

Then prep the text. Get did of end of line chars

Can you write Python code for it?

text = community_data.raw()
summary_sentences = []
candidate_sentences = {}
candidate_sentence_counts = {}
striptext = text.replace('\n\n', ' ')
striptext = striptext.replace('\n', ' ')


I am working on the file "BAMM.101x/text_mining.ipynb".
The first task is:

We can summarize George Washington's first inaugural speech

Can you write Python code for it?


I am working on the file "BAMM.101x/text_mining.ipynb".
The first task is:

We could also develop a affect calculator for common terms in our domain (e.g., food items)

Can you write Python code for it?

def get_affect(text,word,lower=False):
    import nltk
    from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer
    analyzer = SentimentIntensityAnalyzer()
    sent_detector = nltk.data.load('tokenizers/punkt/english.pickle')
    sentences = sent_detector.tokenize(text.strip())
    sentence_count = 0
    running_total = 0
    for sentence in sentences:
        if lower: sentence = sentence.lower()
        if word in sentence:
            vs = analyzer.polarity_scores(sentence)
            running_total += vs['compound']
            sentence_count += 1
    if sentence_count == 0: return 0
    return running_total/sentence_count

get_affect(community_data.raw(),'service',True)


I am working on the file "BAMM.101x/text_mining.ipynb".
The first task is:

We could do a word cloud comparison

We'll remove short words and look only at words longer than 6 letters
Can you write Python code for it?

texts = restaurant_data
from wordcloud import WordCloud, STOPWORDS
import matplotlib.pyplot as plt
%matplotlib inline
#Remove unwanted words
#As we look at the cloud, we can get rid of words that don't make sense by adding them to this variable
DELETE_WORDS = []
def remove_words(text_string,DELETE_WORDS=DELETE_WORDS):
    for word in DELETE_WORDS:
        text_string = text_string.replace(word,' ')
    return text_string

#Remove short words
MIN_LENGTH = 0
def remove_short_words(text_string,min_length = MIN_LENGTH):
    word_list = text_string.split()
    for word in word_list:
        if len(word) < min_length:
            text_string = text_string.replace(' '+word+' ',' ',1)
    return text_string

#Set up side by side clouds
COL_NUM = 2
ROW_NUM = 2
fig, axes = plt.subplots(ROW_NUM, COL_NUM, figsize=(12,12))

for i in range(0,len(texts)):
    text_string = remove_words(texts[i][1])
    text_string = remove_short_words(text_string)
    ax = axes[i%2]
    ax = axes[i//2, i%2] #Use this if ROW_NUM >=2
    ax.set_title(texts[i][0])
    wordcloud = WordCloud(stopwords=STOPWORDS,background_color='white',width=1200,height=1000,max_words=20).generate(text_string)
    ax.imshow(wordcloud)
    ax.axis('off')
plt.show()


I am working on the file "BAMM.101x/text_mining.ipynb".
The first task is:

We need to modify comparitive_emotion_analyzer to tell it where the restaurant name and the text is in the tuple

Can you write Python code for it?

def comparative_emotion_analyzer(text_tuples,name_location=1,text_location=2):
    print("%-20s %1s\t%1s %1s %1s %1s   %1s %1s %1s %1s"%(
            "restaurant","fear","trust","negative","positive","joy","disgust","anticip",
            "sadness","surprise"))

    for text_tuple in text_tuples:
        text = text_tuple[text_location]
        result = emotion_analyzer(text)
        print("%-20s %1.2f\t%1.2f\t%1.2f\t%1.2f\t%1.2f\t%1.2f\t%1.2f\t%1.2f\t%1.2f"%(
            text_tuple[name_location][0:20],result['fear'],result['trust'],
              result['negative'],result['positive'],result['joy'],result['disgust'],
              result['anticipation'],result['sadness'],result['surprise']))

#And test it
comparative_emotion_analyzer(all_snippets)

restaurant_data = [('community',community_data.raw()),('le monde',le_monde_data.raw())
                  ,('heights',heights_data.raw()), ('amigos',amigos_data.raw())]
comparative_emotion_analyzer(restaurant_data,0,1)


I am working on the file "BAMM.101x/text_mining.ipynb".
The first task is:

Write the function that queries yelp. We'll use rauth library to handle authentication

!pip install rauth
Can you write Python code for it?

def get_results(params):
    import rauth
    consumer_key = CONSUMER_KEY
    consumer_secret = CONSUMER_SECRET
    token = TOKEN
    token_secret = TOKEN_SECRET

    session = rauth.OAuth1Session(
    consumer_key = consumer_key
    ,consumer_secret = consumer_secret
    ,access_token = token
    ,access_token_secret = token_secret)

    request = session.get("http://api.yelp.com/v2/search",params=params)
    #Transforms the JSON API response into a Python dictionary
    data = request.json()
    session.close()

    return data

#Get the results
response = get_results(set_search_parameters(get_lat_lng("Community Food and Juice")[0],get_lat_lng("Community Food and Juice")[1],200))


I am working on the file "BASELINE/Google_Images_Baseline.ipynb".
The first task is:
Add the hash to each row in our hillary_clinton_remaining_images.csv and donald_trump_remaining_images.csv dataframes
Can you write Python code for it?

import shelve

# This is the same as script `3_baseline_imagesintoDF.py`

def add_hash_to_DF(shelf, data_csv, candidate):

    db = shelve.open(shelf + '.shelve')

    data = pd.read_csv(data_csv)

    for key in db.keys():   #  For every hash KEY
        for f in db[key]:   # for every file path name ITEM within the KEY
            for index, i in enumerate(data['image_file']):   # For every Image path in each row of my DF
                if f == i:                                   # If the ITEM file path is also in the IMAGE PATH of my DF
                    data.loc[index, 'image_hash'] = key      # Put the KEY into the 'image_hash' Column

    data.to_csv(candidate + "_baseline_hashedDF.csv", index=False)

    db.close()

add_hash_to_DF('hc_baseline', 'hillary_clinton_remaining_images.csv', 'HC' )

add_hash_to_DF('dt_baseline', 'donald_trump_remaining_images.csv', 'DT' )


I am working on the file "BASELINE/Google_Images_Baseline.ipynb".
The first task is:
Combining ratings from Allsides and Facebook, together with crowdsourced political bias ratings from Mondo Times and author's judgement
First, Facebook ratings were categoriesed into 2, 1, 0, -1, and -2 integers so as to be compatable with the Allsides data. Between-category thresholds were selected so as to make each bucket equal in float range of values.
Can you write Python code for it?

def convert_facebookbias_toInts(col):

    if col >= 0.6 and col <= 1:
        return 2
    elif col >= 0.2 and col < 0.6:
        return 1
    elif col > -0.2 and col < 0.2:
        return 0
    elif col > -0.6 and col <= -0.2:
        return -1
    elif col <= -0.6:
        return -2
    elif col == 999:
        return 999
    else:
        return 9992

HC['facebook_int'] = HC.facebookbias_rating.apply(convert_facebookbias_toInts)
DT['facebook_int'] = DT.facebookbias_rating.apply(convert_facebookbias_toInts)

HC.head()

HC.facebook_int.value_counts()

DT.facebook_int.value_counts()


I am working on the file "BASELINE/Google_Images_Baseline.ipynb".
The first task is:
Getting bias ratings from a study of news sharing on Facebook by (Bakshy et al. 2015)
Data is available here
Can you write Python code for it?

facebook = pd.read_csv('../Facebook_study.csv')

facebook.head()

cols = ['p', 'avg_align']
facebook = pd.read_csv('../Facebook_study.csv', usecols=cols)
facebook.head()

def tag_facebookbias_rating(candidate):

    candidate['facebook_p'] = ''
    candidate['facebookbias_rating'] = 999

    for i, valuei in enumerate(candidate.news_source_url):
        valuei = valuei.split('//')[1]
        valuei = valuei.split('/')[0]

        for j, valuej in enumerate(facebook.p):

            if valuej == valuei:
                print(valuei, valuej)
                candidate.loc[i, 'facebookbias_rating'] = facebook.loc[j, 'avg_align']
                candidate.loc[i, 'facebook_p'] = valuej

tag_facebookbias_rating(HC)
tag_facebookbias_rating(DT)

HC.head()

DT.facebookbias_rating[DT.facebookbias_rating < 3].plot.hist(alpha=0.5, bins=20, range=(-1,1), color='red')
HC.facebookbias_rating[HC.facebookbias_rating < 3].plot.hist(alpha=0.5, bins=20, range=(-1,1), color='blue')
#plt.savefig('baseline_facebookbias_hist.png')


I am working on the file "BASELINE/Google_Images_Baseline.ipynb".
The first task is:
Make list of unique news sources, combine lists together from HC and DT, and output to a csv:
Can you write Python code for it?

Unrated_newssource_list = HC_Unrated.news_source.unique().tolist()
DT_Unrated_newssource_list = DT_Unrated.news_source.unique().tolist()

print(len(Unrated_newssource_list))
print(len(DT_Unrated_newssource_list))

for i in DT_Unrated_newssource_list:
    if i not in Unrated_newssource_list:
        Unrated_newssource_list.append(i)

len(Unrated_newssource_list)

tmp = pd.DataFrame(Unrated_newssource_list, columns=["news_source"])
#tmp.to_csv('unrated_newssources.csv', index=False)


I am working on the file "BASELINE/Google_Images_Baseline.ipynb".
The first task is:
NEWS SOURCE INFORMATION
Can you write Python code for it?


I am working on the file "BASELINE/Google_Images_Baseline.ipynb".
The first task is:
Next, ratings from Allsides and the Facebook study were combined.
To combine, bias ratings had to to agree or be present only in one dataset.
If ratings between Allsides and Facebook data disagreed they were assigned Unknown / Unreliable
Can you write Python code for it?

def combine_ratings(candidate):

    candidate['combine_rating'] = ''

    for i, valuei in enumerate(candidate.allsides_bias_rating):

        # STATEMENTS FOR IF BOTH RATINGS AGREE:
        # Both bias ratings say LEFT
        if (valuei < 0) and (candidate.loc[i, 'facebook_int'] < 0):
            print(valuei, candidate.loc[i, 'facebook_int'],  "Left")
            candidate.loc[i, 'combine_rating'] = "Left"

        # Both bias ratings say CENTER
        elif (valuei == 0.0) and (candidate.loc[i, 'facebook_int'] == 0):
            print(valuei, candidate.loc[i, 'facebook_int'],  "Center")
            candidate.loc[i, 'combine_rating'] = "Center"

        # Both bias ratings say RIGHT
        elif (0 < valuei < 3) and (0 < candidate.loc[i, 'facebook_int'] < 3):
            print(valuei, candidate.loc[i, 'facebook_int'], "Right")
            candidate.loc[i, 'combine_rating'] = "Right"

        # STATEMENTS FOR IF RATINGS ARE ONLY PRESENT IN ONE (ALLSIDES OR FACEBOOK STUDY)
        # Only one scale has a rating of LEFT, while the other has no entry
        elif (valuei < 0 and candidate.loc[i, 'facebook_int'] == 999) or (valuei == 999 and candidate.loc[i, 'facebook_int'] < 0):
            print(valuei, candidate.loc[i, 'facebook_int'],  "Left")
            candidate.loc[i, 'combine_rating'] = "Left"

        # Only one scale has a rating of CENTER, while the other has no entry
        elif (valuei == 0 and candidate.loc[i, 'facebook_int'] == 999) or (valuei  == 999 and candidate.loc[i, 'facebook_int'] == 0):
            print(valuei, candidate.loc[i, 'facebook_int'],  "Center")
            candidate.loc[i, 'combine_rating'] = "Center"

        # Only one scale has a rating of RIGHT, while the other has no entry
        elif (0 < valuei < 3 and candidate.loc[i, 'facebook_int'] == 999) or (valuei  == 999 and 0 < candidate.loc[i, 'facebook_int'] < 3):
            print(valuei, candidate.loc[i, 'facebook_int'], "Right")
           ...


I am working on the file "BASELINE/Google_Images_Baseline.ipynb".
The first task is:
Scrape images and metadata from the saved web search source file
Run 1_extract_images_urls.py script

This script reads the htm source file saved from the Google Image search results page for the queries 'hillary clinton' or 'donald trump' for images from the 3rd September 2016 to the 28th October 2016.
output will be:
meta data saved in donald_trump_image_universe.csv or hillary_clinton_image_universe.csv
Images saved in donald_trump_image_universe or hillary_clinton_image_universe directories

Manually check images:

Remove images that are not of the candidate
Remove images where there are mulitple faces
Manually collect images that failed to download properly. This is sometimes because more time was needed to scrape it

Run 2_remove_rejected_from_data.py script

Removes rows from the donald_trump_image_universe.csv or hillary_clinton_image_universe.csv spreadsheets, so it correctly references the remaining images in the donald_trump_image_universe or hillary_clinton_image_universe image directories.

Programatically identify unique images

Read in each image, calculate a unique hash based on the image data, a bit like a fingerprint, and store the hash along with the image filename in a shelf (a persistent dictionary). Many images were repeats, so there were many image filenames stored with the same hash key.

The code is based on this blog post: https://realpython.com/blog/python/fingerprinting-images-for-near-duplicate-detection/ which I adapted to work with Python3
Can you write Python code for it?

%run ../index.py --dataset ./hillary_clinton_image_universe --shelve hc_baseline.shelve

%run ../index.py --dataset ./donald_trump_image_universe --shelve dt_baseline.shelve


I am working on the file "BASELINE/Google_Images_Baseline.ipynb".
The first task is:
unrated_newssource.csv was then opened in Excel, and the author went through each one manually, to determine political bias, if possible.
The final document was saved as Unrated_newssource_final.xlsx and used below.

Read in manual bias ratings:
Can you write Python code for it?

manual_rating = pd.read_excel('Unrated_newssource_final.xlsx')

manual_rating.head()

def merge_manual_ratings(candidate):

    candidate['final_rating'] = ''

    for i, valuei in enumerate(candidate.news_source):
        for j, valuej in enumerate(manual_rating.news_source):

            if (valuei == valuej):
                print(valuei, valuej, manual_rating.loc[j, 'final_rating'])

                if manual_rating.loc[j, 'final_rating'] < 0:
                    print("Left")
                    candidate.loc[i, 'final_rating'] = "Left"

                elif manual_rating.loc[j, 'final_rating'] == 0:
                    print("Center")
                    candidate.loc[i, 'final_rating'] = "Center"

                elif 999 > manual_rating.loc[j, 'final_rating'] > 0:
                    print("Right")
                    candidate.loc[i, 'final_rating'] = "Right"

                elif manual_rating.loc[j, 'final_rating'] == 999:
                    print("Unknown/Unreliable")
                    candidate.loc[i, 'final_rating'] = "Unknown / unreliable"

    for i, valuei in enumerate(candidate.final_rating):
        if valuei == '':
            print("currently empty. Let's fill it up!!")
            candidate.loc[i, 'final_rating'] = candidate.loc[i, 'combine_rating']

merge_manual_ratings(HC)
merge_manual_ratings(DT)

HC.head()

HC.final_rating.value_counts()

DT.final_rating.value_counts()

HC.to_csv('HC_baseline_full_ratings.csv', index=False)
DT.to_csv('DT_baseline_full_ratings.csv', index=False)


I am working on the file "BASELINE/ImageAPI_analysis_baseline.ipynb".
The first task is:
Examination of the images that triggered an error revealed that errors were caused by the API being unable to calculate an emotion due to the face being partially obscured or being in profile.
Can you write Python code for it?

HRC_universe[HRC_universe.fear == 'Error']

DJT_universe[DJT_universe.fear == 'Error']


I am working on the file "BASELINE/ImageAPI_analysis_baseline.ipynb".
The first task is:
Microsoft Emotion API Data
Run 4_check_img_size.py

This script checks the images are not too large for Micrsofts API, which has a limit of 1kb to 4MB
When it finds an image that is too large, it resizes it to 3.5MB, and saves as "original_filename_resized.jpg"
Check through the results, and remove original file if a resized one has been created. This step could be done programmatically... but being forced to actually look at your data can be good, too.

Run 5_microsoft_Face_API.py

You will need your own API keys. Free key subscription gets you a generous rate limit.
This script only collects face_id, age, smile, gender, and emotion.
Although this is the Face API, it also returned emotion, which was handy.
One JSON is output per image

Read in JSONS and make a dataframe
Can you write Python code for it?

def read_jsons(f, candidate):

    tmp_dict = {}

    with open(f) as json_file:
        data = json.load(json_file)
        data = json.loads(data)
        print(data)
        try:
            tmp_dict['age'] = data[0]['faceAttributes']['age']
            tmp_dict['gender'] = data[0]['faceAttributes']['gender']
            tmp_dict['smile'] = data[0]['faceAttributes']['smile']

            if data[0]['faceAttributes']['emotion']['anger'] > 0.55:   # this value is the confidence threshold
                tmp_dict['anger'] = data[0]['faceAttributes']['emotion']['anger']

            if data[0]['faceAttributes']['emotion']['contempt'] > 0.55:
                tmp_dict['contempt'] = data[0]['faceAttributes']['emotion']['contempt']

            if data[0]['faceAttributes']['emotion']['disgust'] > 0.55:
                tmp_dict['disgust'] = data[0]['faceAttributes']['emotion']['disgust']

            if data[0]['faceAttributes']['emotion']['fear'] > 0.55:
                tmp_dict['fear'] = data[0]['faceAttributes']['emotion']['fear']

            if  data[0]['faceAttributes']['emotion']['happiness'] > 0.55:
                tmp_dict['happiness'] = data[0]['faceAttributes']['emotion']['happiness']

            if data[0]['faceAttributes']['emotion']['neutral'] > 0.55:
                tmp_dict['neutral'] = data[0]['faceAttributes']['emotion']['neutral']

            if data[0]['faceAttributes']['emotion']['sadness'] > 0.55:
                tmp_dict['sadness'] = data[0]['faceAttributes']['emotion']['sadness']

            if data[0]['faceAttributes']['emotion']['surprise'] > 0.55:
                tmp_dict['surprise'] = data[0]['faceAttributes']['emotion']['surprise']

            tmp_dict['face_id'] = data[0]['faceId']

            tmp_dict['image_file'] = f.split('/')[1]

        # populating the row with `Error` allows us to check the original image to understand the caused of...


I am working on the file "BASELINE/ImageAPI_analysis_baseline.ipynb".
The first task is:
Visualising sentiment scores
Can you write Python code for it?

HRC_universe_sentiment.mean().plot(kind='bar', ylim=(0,1), color='blue', alpha=.5)
DJT_universe_sentiment.mean().plot(kind='bar',ylim=(0,1), color='red', alpha=.5)

print("anger: ", len(HRC_universe_sentiment[HRC_universe_sentiment.anger >= 0.55]))
print("contempt: ", len(HRC_universe_sentiment[HRC_universe_sentiment.contempt >= 0.55]))
print("disgust: ", len(HRC_universe_sentiment[HRC_universe_sentiment.disgust >= 0.55]))
print("fear: ", len(HRC_universe_sentiment[HRC_universe_sentiment.fear >= 0.55]))
print("happiness: ", len(HRC_universe_sentiment[HRC_universe_sentiment.happiness >= 0.55]))
print("neutral: ", len(HRC_universe_sentiment[HRC_universe_sentiment.neutral >= 0.55]))
print("sadness: ", len(HRC_universe_sentiment[HRC_universe_sentiment.sadness >= 0.55]))
print("surprise: ", len(HRC_universe_sentiment[HRC_universe_sentiment.surprise >= 0.55]))

print("anger: ", len(DJT_universe_sentiment[DJT_universe_sentiment.anger >= 0.55]))
print("contempt: ", len(DJT_universe_sentiment[DJT_universe_sentiment.contempt >= 0.55]))
print("disgust: ", len(DJT_universe_sentiment[DJT_universe_sentiment.disgust >= 0.55]))
print("fear: ", len(DJT_universe_sentiment[DJT_universe_sentiment.fear >= 0.55]))
print("happiness: ", len(DJT_universe_sentiment[DJT_universe_sentiment.happiness >= 0.55]))
print("neutral: ", len(DJT_universe_sentiment[DJT_universe_sentiment.neutral >= 0.55]))
print("sadness: ", len(DJT_universe_sentiment[DJT_universe_sentiment.sadness >= 0.55]))
print("surprise: ", len(DJT_universe_sentiment[DJT_universe_sentiment.surprise >= 0.55]))


I am working on the file "BASH.ipynb".
The first task is:
-E flag has to be added if one want to use expressions such as + or |.
Can you write Python code for it?

!dpkg -L iproute2 | grep -E '/bin|/sbin'

!dpkg -L iproute2 | grep -E '/bin|/sbin' | wc -l

!dpkg -L iproute2 | grep -E '/s?bin' | wc -l


I am working on the file "BASH.ipynb".
The first task is:
Another code which do exactly the same thing:
Can you write Python code for it?

%%bash
#!/bin/bash
# calculate the available % of disk space

echo "Current Mount Points:"
mount | egrep 'ext[234]|xfs' | cut -f 3 -d ' ' # -f: field; -d:delimiter

#read -p "Enter a Mount Point: " mntpnt
mntpnt="/home"

df_out="$(df $mntpnt | tail -1)"

set $df_out

availpct=$(echo "scale=4; ${4}/${2} * 100" | bc)

printf "There is %5.2f%% available in %s\n" $availpct $mntpnt

exit 0

%%bash

#!/bin/bash
# shift left is double, shift right is half

declare -i number

#read -p "Enter a number: " number
number=-4

echo " Double $number is: $((number << 1))"
echo " Half of $number is: $((number >> 1))"

exit 0


I am working on the file "BASH.ipynb".
The first task is:
In BASH, a line is excuted successfully if the exit status is 0.
Can you write Python code for it?

%%bash

:
echo "exit status=$?"

%%bash

ls /dhuoewyr242q
echo "exit status=$?"

%%bash

true
echo "exit status=$?"

%%bash

false
echo "exit status=$?"

%%bash

value=123
test $value=="123"
echo "exit status=$?"
echo""
test $value=="456"
echo "exit status=$?"


I am working on the file "BASH.ipynb".
The first task is:
Let's play iPython and BASH a bit
count number of paths in $PATH:
Can you write Python code for it?

path=!echo $PATH

print path

path[0].split(":")

print len(path[0].split(":"))


I am working on the file "BASH.ipynb".
The first task is:
Remark
?: showing once or 0 times
*: showing for any times (including 0 times)
+: showing at least once
Remark:
-i, --ignore-case
-v  --invert-match
-n, --line-number
-E: extended
Can you write Python code for it?

f=open("regular_express.txt", "r")
file=f.read()

print repr(file)

for line in open("regular_express.txt", "r"):
    print line

os.mkdir("tmp")

os.listdir(os.getcwd())

ls

os.removedirs("tmp")

os.listdir(".")


I am working on the file "BASH.ipynb".
The first task is:
See if an integer A is greater equal than another integer B:
Can you write Python code for it?

%%bash
i=5
if [ $i -ge 0 ];then echo "$i >= 0";fi


I am working on the file "BASH.ipynb".
The first task is:
```bash
1. ls -al /etc | awk '$1 ~ /^d/ {print "dir: ",$9}'

ll /etc | awk '$1 ~ /^d/ && $9 !~ /^./ {print "dir: ",$9}'

ll /sbin |awk '/^-/ && $2 > 1 {print "file:",$9 "\t links: ",$2}'

cat /etc/services | awk '$1 == "ssh" {print "service: ",$1,$2}'

ss -an | awk '/^ESTAB/ && $4 ~ /:22$/ {print "ssh from:",$5}'

mount | awk '$5 ~ /ext[234]/ || $5 == "xfs" {print $3,"("$5")"}'

ps -ef | awk '$2 == 1 , $2 == 10 {print}'
```
Can you write Python code for it?

!ls -al /home/chweng

!ls -al /home/chweng/ | awk '/^d/ {print "dir: ",$9}'

!mount |grep "ext"

!mount | awk '/ ext[234] / {print "device: ",$1,"\tmount point: " $3}'

!mount | awk '/ ext[234] / {print "device: %5s \tmount point: %5s",$1, $3}'

!ip addr show


I am working on the file "BASH.ipynb".
The first task is:
```bash
input data from nbafile file

awk '$3 == 82 {print $1," \t",$5}' nbafile

awk '$3 < 78' nbafile

awk '$2 ~ /c.*l/' nbafile

awk '$1 ~ /^s/ && $4 > 80 {print $1 "\t\t" $4}' nbafile
```
Can you write Python code for it?

%%bash

echo "This is a book" | awk '
   { print "length of the string : ",$0," is : ",length($0) }'


I am working on the file "BASH.ipynb".
The first task is:
alternatively, one can write it like this (with the help of )
Can you write Python code for it?

%%bash
i=5
if (($i >= 0));then echo "$i >= 0";fi

%%bash
i=5
if [ $i >= 0 ];then echo "$i >= 0";fi

%%bash
i=05
if [ $i -ge 0 ];then echo "$i >= 0";fi


I am working on the file "BASH.ipynb".
The first task is:
another way to create a loop, as what people normally do in Java:
Can you write Python code for it?

%%bash
#!/bin/bash

#echo $1
#file=$1

cd /home/chweng/code/exercises/
file="hello.sh"

if [ -f $file ]
then
  echo "the file $file exists"
fi

for((j=1;j<=5;j++))
do
  echo $j, "Hello World"
done


I am working on the file "BASH.ipynb".
The first task is:
arithmatic calculations are enclosed by (()):
Can you write Python code for it?

%%bash
echo $((7**2))

%%bash
echo $((7%3))

%%bash
#!/bin/bash
# calculate the available % of disk space

echo "Current Mount Points:"
mount | grep -E 'ext[234]|xfs' | cut -f 3 -d ' '

#read -p "Enter a Mount Point: " mntpnt
mntpnt="/home"

sizekb=$(df $mntpnt | tail -1 | tr -s ' ' | cut -f 2 -d ' ')
availkb=$(df $mntpnt | tail -1 | tr -s ' ' | cut -f 4 -d ' ')

availpct=$(echo "scale=4; $availkb/$sizekb * 100" | bc)

printf "There is %5.2f%% available in %s\n" $availpct $mntpnt

exit 0


I am working on the file "BASH.ipynb".
The first task is:
bash
chweng@VirtualBox:~$ sudo fdisk -l | awk '/^Disk \/dev\/[hs]d/ {print $2,$3,$4}'
/dev/sda: 1 TiB,
Can you write Python code for it?

!cat /etc/group | awk -F : '/^sudo/ {print $1 ,"users are:", $4}'

import os
os.chdir("/home/chweng/code")


I am working on the file "BASH.ipynb".
The first task is:
bash
chweng@chweng-VirtualBox:~$ env |grep "PWD"
PWD=/home/chweng
chweng@chweng-VirtualBox:~$
chweng@chweng-VirtualBox:~$ cd Desktop/
chweng@chweng-VirtualBox:~/Desktop$ env |grep "PWD"
PWD=/home/chweng/Desktop
OLDPWD=/home/chweng
Can you write Python code for it?

%%bash
#!/bin/bash
# Global and local variables inside a function.

func ()
{
        local loc_var=23                # Declared as local variable.
        echo                                    # Uses the 'local' builtin.
        echo "\"loc_var\" in function = $loc_var"
        global_var=999                  # Not declared as local.
                                                        # Therefore, defaults to global.
        echo "\"global_var\" in function = $global_var"
}

func

# Now, to see if local variable "loc_var" exists outside the function.

echo
echo "\"loc_var\" outside function = $loc_var"
                                      # $loc_var outside function =
                                      # No, $loc_var not visible globally.
echo "\"global_var\" outside function = $global_var"
                                      # $global_var outside function = 999
                                      # $global_var is visible globally.
echo

exit 0
#  In contrast to C, a Bash variable declared inside a function
#+ is local ONLY if declared as such.

%%bash
#!/bin/bash
# passing data to function

# DECLARE FUNCTIONS
shifter()       # function to demonstrate parameter
                        # list management in a function
{
        echo "$# parameters passed to $0"
        while (( $# > 0 ))
        do
                echo "$*"
                shift
        done
}

# MAIN
#read -p "Please type a list of five words (then press Return): " varlist
varlist="i my me mine myself"

set $varlist    # this creates positional parameters in the parent

shifter $*              # call the function and pass argument list

echo "$# parameters in the parent "
echo "Parameters: $*"

exit

%%bash
#!/bin/bash
# Functions and parameters

DEFAULT=default # Default param value.

func2 () {
        if [ -z "$1" ] # Is parameter #1 zero length?
        then
                echo "-Parameter #1 is zero length.-" # Or no parameter passed.
        else
                echo "-Parameter...


I am working on the file "BASH.ipynb".
The first task is:
bash
chweng@ubuntu221:~/code$ javac -d . TestDiceThrowEx1.java
chweng@ubuntu221:~/code$ ls
RoadLog                exercises  module03  module05  module07  tw
TestDiceThrowEx1.java  module02   module04  module06  module08
chweng@ubuntu221:~/code$ java tw.loop.TestDiceThrowEx1
diceNumber=5
Try Again.
diceNumber=5
Try Again.
diceNumber=6
Try Again.
diceNumber=6
Try Again.
diceNumber=4
Try Again.
diceNumber=5
Try Again.
diceNumber=6
Try Again.
diceNumber=2
You Win.
Can you write Python code for it?

%%bash
ls -l /home/chweng/code/exercises/ |awk '/^-/ && $2 = 1 {print "file:",$9 "\t links: ",$2}'


I am working on the file "BASH.ipynb".
The first task is:
break & continue:
Can you write Python code for it?

%%bash
#!/bin/bash
# break, continue usage

LIMIT=19        # Upper limit

echo
echo "Printing Numbers 1 through 20 (but not 3 and 11)."

a=0

while (( a <= LIMIT))
do
        ((a++))

        if [[ "$a" -eq 3  ||  "$a" -eq 11 ]] # Excludes 3 and 11.
        then
                continue        # Skip rest of this particular loop iteration.
        fi

        echo -n "$a " # This will not execute for 3 and 11.
done

# Exercise:
# Why does the loop print up to 20?

echo; echo

echo Printing Numbers 1 through 20, but something happens after 2.

##################################################################
# Same loop, but substituting 'break' for 'continue'.

a=0

while [ "$a" -le "$LIMIT" ]
do
        a=$(($a+1))

        if [ "$a" -gt 2 ]
        then
                break # Skip entire rest of loop.
        fi

        echo -n "$a "
done
exit 0

%%bash
#!/bin/bash
# The "continue N" command, continuing at the Nth level loop.

for outer in I II III IV V # outer loop
do
        echo; echo -n "Group $outer: "

        # --------------------------------------------------------------------
        for inner in 1 2 3 4 5 6 7 8 9 10 # inner loop
        do
                if [[ "$inner" -eq 7 && "$outer" = "III" ]]
                then
                        continue 2      # Continue at loop on 2nd level, that is "outer loop".
                                                # Replace above line with a simple "continue"
                                                # to see normal loop behavior.
                fi

                echo -n "$inner " # 7 8 9 10 will not echo on "Group III."
        done
        # --------------------------------------------------------------------
done

echo; echo

exit 0

%%bash
#!/bin/bash
# The "continue N" command, continuing at the Nth level loop.

for outer in I II III IV V # outer loop
do
        echo; echo -n "Group $outer: "

        # --------------------------------------------------------------------
        for inner in 1 2 3 4 5 6 7 8 9 10 # inner...


I am working on the file "BASH.ipynb".
The first task is:
change the language environment
Can you write Python code for it?

!locale

!export LANG='en_US.UTF-8'

!locale


I am working on the file "BASH.ipynb".
The first task is:
example: ex4-4.sh:
Can you write Python code for it?

!mkdir /home/chweng/a

!touch /home/chweng/a/123.txt

%%bash
#!/bin/bash
# using file test operator

DEST="~/b"
SRC="~/a"

# Make sure backup dir exits
if [ ! -d $DEST ]
then
        mkdir -p $DEST
fi

# If source directory does not exits, die...
if [ ! -d $SRC ]
then
        echo "$SRC directory not found. Cannot make backup to $DEST"
        exit 1
fi

# Okay, dump backup using tar
echo "Backup directory $DEST..."
echo "Source directory $SRC..."
/bin/tar -Jcf $DEST/backup.tar.xz $SRC 2>/dev/null

# Find out if backup failed or not
if [ $? -eq 0 ]
then
        echo "Backup done!"
else
        echo "Backup failed"
fi


I am working on the file "BASH.ipynb".
The first task is:
example: ex4-4.sh:
use the advanced-test:[[]]
to compare different integer strings in different forms (it could be that one in decimal and another in octal format)
Can you write Python code for it?

%%bash

#!/bin/bash
# using [ and [[

file=/etc/passwd

if [[ -e $file ]]
then
        echo "Password file exists."
fi

# [[ Octal and hexadecimal evaluation ]]
# Thank you, Moritz Gronbach, for pointing this out.

decimal=15
octal=017       # = 15 (decimal)
hex=0x0f        # = 15 (decimal)

if [ "$decimal" -eq "$octal" ]
then
        echo "$decimal equals $octal"
else
        echo "$decimal is not equal to $octal" # 15 is not equal to 017
fi      # Doesn't evaluate within [ single brackets ]!

if [[ "$decimal" -eq "$octal" ]]
then
        echo "$decimal equals $octal" # 15 equals 017
else
        echo "$decimal is not equal to $octal"
fi      # Evaluates within [[ double brackets ]]!

if [[ "$decimal" -eq "$hex" ]]
then
        echo "$decimal equals $hex" # 15 equals 0x0f
else
        echo "$decimal is not equal to $hex"
fi      # [[ $hexadecimal ]] also evaluates!


I am working on the file "BASH.ipynb".
The first task is:
get the line which is ended by '!':
Can you write Python code for it?


I am working on the file "BASH.ipynb".
The first task is:
get the line which is not started by any capital or lower-case alphabets:
Can you write Python code for it?

!grep -n '^[^A-Za-z]' regular_express.txt


I am working on the file "BASH.ipynb".
The first task is:
in the above example, \$(pwd) can be replaced by $PWD if the env variable PWD exists
the "for each" loop
Can you write Python code for it?

for planet in "Mercury 36" "Venus 67" "Earth 93" "Mars 142" "Jupiter 483"

%%bash
#!/bin/bash
# Planets revisited.
# Associate the name of each planet with its distance from the sun.

for planet in "Mercury 36" "Venus 67" "Earth 93" "Mars 142" "Jupiter 483"
do
        set -- $planet  # Parses variable "planet"
                                        #+ and sets positional parameters.
        # The "--" prevents nasty surprises if $planet is null or
        #+ begins with a dash.

# May need to save original positional parameters,
#+ since they get overwritten.
# One way of doing this is to use an array,
# original_params=("$@")

        echo "$1                $2,000,000 miles from the sun"
        #-------two tabs---concatenate zeroes onto parameter $2
done

# (Thanks, S.C., for additional clarification.)

exit 0


I am working on the file "BASH.ipynb".
The first task is:
learn a bit of os.walk():
Can you write Python code for it?

for root,dirs,files in os.walk(os.getcwd()):
    print root,dirs,files
    print

for root, dirs, files in os.walk(os.getcwd()):
    for file in files:
        print os.path.join(root,file)

for root, dirs, files in os.walk(os.getcwd()):
    for file in files:
        if file.endswith('.txt'):
            print file

!whereis regex

!find / -name 'ifconfig'

!find -type f -user chweng -name '*.txt'

!find -type d -user chweng -name '0*'

!sudo fdisk -l | grep -nE "^Disk /dev/[hs]d"

sudo find /etc -type f | wc -l

du -hl /etc/


I am working on the file "BASH.ipynb".
The first task is:
library trace: trace library calls demanded by the specified process.
Can you write Python code for it?


I am working on the file "BASH.ipynb".
The first task is:
list open files:
Can you write Python code for it?

!lsof / |grep "/home/chweng/.ipython"

!strace -c find /etc -name "python*"


I am working on the file "BASH.ipynb".
The first task is:
look for files:
the flag -ld of "ls" means "list directory".
Can you write Python code for it?

!ls -ld /etc/p*

!ls -ld /etc/p* | wc -l

!ls -ld /etc/p????

!ls -ld /etc/p???? | wc -l

!ls -ld /etc/p[aeiou]*

!ls -ld /etc/p[aeiou]* | wc -l

!ls -ld /etc/p[!aeiou]*

!ls -ld /etc/p[^aeiou]*


I am working on the file "BASH.ipynb".
The first task is:
review: when to use the enhanced-test [[ ]]?
A: when && or || operator is used
Can you write Python code for it?

%%bash
#!/bin/bash

#echo $1
#file=$1

cd /home/chweng/code/exercises/
file="hello.sh"

if [[ -f $file && true ]]
then
  echo "the file $file exists"
fi

for((j=1;j<=5;j++))
do
  echo $j, "Hello World"
done


I am working on the file "BASH.ipynb".
The first task is:
the "top" command
http://mugurel.sumanariu.ro/linux/the-difference-among-virt-res-and-shr-in-top-output/

RES stands for the resident size, which is an accurate representation of how much actual physical memory a process is consuming. (This also corresponds directly to the %MEM column.) This will virtually always be less than the VIRT size, since most programs depend on the C library.
SHR indicates how much of the VIRT size is actually sharable (memory or libraries). In the case of libraries, it does not necessarily mean that the entire library is resident. For example, if a program only uses a few functions in a library, the whole library is mapped and will be counted in VIRT and SHR, but only the parts of the library file containing the functions being used will actually be loaded in and be counted under RES.

use "top -o RES" to sort by resident size(RES)
16.11.2016
The following cell is the content of the script testPS.sh:
```BASH
!/bin/bash
ps -f
read
```
Now, we execute it:
```BASH
chweng@chweng-VirtualBox:~/code/exercises$ ./testPS.sh
UID        PID  PPID  C STIME TTY          TIME CMD
chweng    9960  2374  0 16:26 pts/31   00:00:00 bash
chweng    9974  9960  0 16:26 pts/31   00:00:00 /bin/bash ./testPS.sh
chweng    9975  9974  0 16:26 pts/31   00:00:00 ps -f
chweng@chweng-VirtualBox:~/code/exercises$ source testPS.sh
UID        PID  PPID  C STIME TTY          TIME CMD
chweng    9960  2374  0 16:26 pts/31   00:00:00 bash
chweng    9980  9960  0 16:26 pts/31   00:00:00 ps -f
```
```BASH
chweng@chweng-VirtualBox:~$ var1=1111
chweng@chweng-VirtualBox:~$ var2=3333
chweng@chweng-VirtualBox:~$ echo "${var1}222"
1111222
chweng@chweng-VirtualBox:~$ set | grep "var1"
var1=1111
chweng@chweng-VirtualBox:~$ set | grep "var2"
var2=3333
```
BASH
chweng@chweng-VirtualBox:$ export var1
chweng@chweng-VirtualBox:$ bash
chweng@chweng-VirtualBox:$ echo $var1
1111
chweng@chweng-VirtualBox:$ echo $var2
Some examples that uses BASH variables:
Can you write Python code for it?

%%bash
var=12345
echo "The length of var1=$var is ${#var}."

%%bash

set $(eval du -sh ~$user);dir_sz=$1
echo "$1,$2"
echo "${dir_sz}"

echo ""

set $(eval df -h|grep " /$");fs_sz=$2
echo "$1,$2"
echo "${fs_sz}"

%%bash

set $(eval du -sh ~$user);dir_sz=$1
set $(eval df -h|grep " /$");fs_sz=$2
echo "Size of my home directory is ${dir_sz}."
echo "Size of my file system size is ${fs_sz}."


I am working on the file "BASH.ipynb".
The first task is:
the command test can be replaced by its synonym [ ]:
Can you write Python code for it?

%%bash
help [

%%bash

value=123
[ $value == "123" ]
echo "exit status=$?"
echo""
[ $value == "456" ]
echo "exit status=$?"

%%bash

/usr/bin/[ 0 == 1 ]
echo "exit status=$?"


I am working on the file "BASH.ipynb".
The first task is:
therefore, it is arithmatic shift in the above script.
17112016
local variables can be declared as readonly
Can you write Python code for it?

%%bash

#!/bin/bash
# declare constants variables

readonly DATA=/home/sales/data/feb09.dat
echo $DATA
echo

DATA=/tmp/foo
# Error ... readonly variable

echo $DATA

exit 0


I am working on the file "BASH.ipynb".
The first task is:
which is the same as the following command in BASH shell:
Can you write Python code for it?


I am working on the file "BASH.ipynb".
The first task is:
while loop:
Can you write Python code for it?

%%bash

#!/bin/bash
# increment number

# set n to 1
n=1

# continue until $n equals 5
while [ $n -le 5 ]
do
        echo "Welcome $n times."
        n=$(( n+1 )) # increments $n
done

%%bash

#!/bin/bash
# increment number

# set n to 1
n=1

# continue until $n equals 5
while (( n <= 5 ))
do
        echo "Welcome $n times."
        (( n++ )) # increments $n
done

%%bash
#!/bin/bash
# while can read data

ls -al | while read perms links owner group size mon day time file
do
        [[ "$perms" != "total" && $size -gt 100 ]] && echo "$file       $size"
done

exit


I am working on the file "BASS v2.0.ipynb".
The first task is:
Display Summary Results for Bursts
Can you write Python code for it?

#grouped summary for bursts
Results['Bursts-Master'].groupby(level=0).describe()


I am working on the file "BASS v2.0.ipynb".
The first task is:
Display Summary Results for Peaks
Can you write Python code for it?

#grouped summary for peaks
Results['Peaks-Master'].groupby(level=0).describe()


I am working on the file "BASS v2.0.ipynb".
The first task is:
Frequency Plot
Use this block to plot changes of any measurement over time. Does not support 'all'. Example:
event_type = 'Peaks'
meas = 'Intervals'
key = 'Mean1'

Frequency Plot
Can you write Python code for it?

event_type = 'Peaks'
meas = 'Intervals'
key = Settings['Label']
frequency_plot(event_type, meas, key, Data, Settings, Results)


I am working on the file "BASS v2.0.ipynb".
The first task is:
Helpful Stuff
While not completely up to date with some of the new changes, the Wiki can be useful if you have questions about some of the settings: https://github.com/drcgw/SWAN/wiki/Tutorial
More Help?
Stuck on a particular step or function?
Try typing the function name followed by two ??. This will pop up the docstring and source code.
You can also call help() to have the notebook print the doc string.
Example:
analyze??
help(analyze)
Can you write Python code for it?


I am working on the file "BASS v2.0.ipynb".
The first task is:
Instructions
For help, check out the wiki: Protocol
Or the video tutorial: Coming Soon!
1) Load Data File(s)
Use the following block to create a BASS_Dataset object and initialize your settings. All settings are attributes of the dataset instance. Manual initialization of settings in this block is optional and is required only once for a given batch. All BASS_Dataset objects that are initialized are automatically added to the batch.
class BASS_Dataset(inputDir, fileName, outputDir, fileType='plain', timeScale='seconds')

    Attributes:

        Batch: static list
            Contains all instances of the BASS_Dataset object in order to be referenced by the global runBatch function.
        Data: library
            instance data
        Settings: library
            instance settings
        Results: library
            instance results

    Methods:

        run_analysis(settings = self.Settings, analysis_module): BASS_Dataset method
            Highest level of the object-oriented analysis pipeline. First syncs the settings of all BASS_Dataset objects
            (stored in Batch), then runs the specified analysis module on each one.

Analysis must be called after the object is initialize and Settings added if the Settings are to be added manually (not via the interactive check and load settings function). Analysis runs according to batch-oriented protocol and is specific to the analysis module determined by the "analysis_module" parameter.
2) Run Analysis
Run
    BASS_Dataset.run_analysis(analysis_mod, settings, batch)
Runs in either single (batch=False) or batch mode. Assuming batch mode, this function first syncs settings of each dataset within Bass_Dataset.Batch to the entered parameter "settings", then runs analysis on each instance within Batch. Be sure to select the correct module given your desired type of analysis. The current options (as of 9/21/16) are "ekg" and "pleth". Parameters are as follows:
    Parameters:
        analysis_mod: string
            the name of the BASS_Dataset module...
Can you write Python code for it?

#Import and Initialize
#Class BASS_Dataset(inputDir, fileName, outputDir, fileType='plain', timeScale='seconds')
data1 = BASS_Dataset('C:\\Users\\Ryan\\Desktop\\sample_data\\','2016-06-22_C1a_P3_Base.txt','C:\\Users\\Ryan\\Desktop\\bass_output\\pleth')
data2 = BASS_Dataset('C:\\Users\\Ryan\\Desktop\\sample_data\\','2016-06-26_C1a_P7_Base.txt','C:\\Users\\Ryan\\Desktop\\bass_output\\pleth')

#transformation Settings
data1.Settings['Absolute Value'] = False #Must be True if Savitzky-Golay is being used

data1.Settings['Bandpass Highcut'] = 12 #in Hz
data1.Settings['Bandpass Lowcut'] = 1 #in Hz
data1.Settings['Bandpass Polynomial'] = 1 #integer

data1.Settings['Linear Fit'] = False #between 0 and 1 on the whole time series
data1.Settings['Linear Fit-Rolling R'] = 0.5 #between 0 and 1
data1.Settings['Linear Fit-Rolling Window'] = 1000 #window for rolling mean for fit, unit is index not time
data1.Settings['Relative Baseline'] = 0 #default 0, unless data is normalized, then 1.0. Can be any float

data1.Settings['Savitzky-Golay Polynomial'] = 'none' #integer
data1.Settings['Savitzky-Golay Window Size'] = 'none' #must be odd. units are index not time

#Baseline Settings
data1.Settings['Baseline Type'] = r'rolling' #'linear', 'rolling', or 'static'
#For Linear
data1.Settings['Baseline Start'] = None #start time in seconds
data1.Settings['Baseline Stop'] = None #end time in seconds
#For Rolling
data1.Settings['Rolling Baseline Window'] = 5 # in seconds. leave as 'none' if linear or static

#Peaks
data1.Settings['Delta'] = 0.05
data1.Settings['Peak Minimum'] = -0.50 #amplitude value
data1.Settings['Peak Maximum'] = 0.50 #amplitude value

#Bursts
data1.Settings['Apnea Factor'] = 2 #factor to define apneas as a function of expiration
data1.Settings['Burst Area'] = True #calculate burst area
data1.Settings['Exclude Edges'] = True #False to keep edges, True to discard them
data1.Settings['Inter-event interval minimum (time-scale units)'] = 0.0001 #only for bursts, not for peaks
data1.Settings['Maximum Burst Duration...


I am working on the file "BASS v2.0.ipynb".
The first task is:
Interactive Graphs
Line Graphs
One pannel, detected events
Plot one time series by calling it's name
Can you write Python code for it?

#Interactive, single time series by Key
key = Settings['Label']
graph_ts(Data, Settings, Results, key)


I am working on the file "BASS v2.0.ipynb".
The first task is:
OPTIONAL GRAPHS AND ANALYSIS
The following blocks are optional calls to other figures and analysis
Display Event Detection Tables
Display Settings used for analysis
Can you write Python code for it?


I am working on the file "BASS v2.0.ipynb".
The first task is:
Poincare Plots
Create a Poincare Plot of your favorite varible. Choose an event type (Peaks or Bursts), measurement type. Calling meas = 'All' is supported.
Plots and tables are saved automatically
Example:
event_type = 'Bursts'
meas = 'Burst Duration'

More on Poincare Plots
Batch Poincare
Batch Poincare
Can you write Python code for it?

#Batch
event_type = 'Bursts'
meas = 'Total Cycle Time'
Results = poincare_batch(event_type, meas, Data, Settings, Results)
pd.concat({'SD1':Results['Poincare SD1'],'SD2':Results['Poincare SD2']})


I am working on the file "BASS v2.0.ipynb".
The first task is:
Quick Poincare Plot
Quickly call one poincare plot for display. Plot and Table are not saved automatically. Choose an event type (Peaks or Bursts), measurement type, and key. Calling meas = 'All' is not supported.
Quick Poincare
Can you write Python code for it?

#quick
event_type = 'Bursts'
meas = 'Attack'
key = Settings['Label']
poincare_plot(Results[event_type][key][meas])


I am working on the file "BASS v2.0.ipynb".
The first task is:
Raster Plot
Shows the temporal relationship of peaks in each column. Auto scales. Display only. Intended for more than one column of data
Can you write Python code for it?


I am working on the file "BASS v2.0.ipynb".
The first task is:
Sample Entropy
this only runs if you have pyeeg.py in the same folder as this notebook and bass.py. WARNING: THIS FUNCTION RUNS SLOWLY
run the below code to get the sample entropy of any measurement. Returns the entropy of the entire results array (no windowing). I am using the following M and R values:
M = 2
R = 0.2*std(measurement)

these values can be modified in the source code. alternatively, you can call samp_entropy directly.
Supports 'all'
Sample Entropy in BASS
Sample Entropy Source
Events
Can you write Python code for it?

#Sample Entropy
event_type = 'Bursts'
meas = 'Total Cycle Time'
Results = samp_entropy_wrapper(event_type, meas, Data, Settings, Results)
Results['Sample Entropy']

Results['Sample Entropy']['Attack']


I am working on the file "BASS v2.0.ipynb".
The first task is:
Time Series
Can you write Python code for it?

#on raw signal
#takes a VERY long time
version = 'original' #original, trans, shift, or rolling
key = Settings['Label']
start = 0 #seconds, where you want the slice to begin
end = 1 #seconds, where you want the slice to end. The absolute end is -1

samp_entropy(Data[version][key][start:end].tolist(), 2, (0.2*np.std(Data[version][key][start:end])))


#Approximate Entropy on raw signal
#takes a VERY long time
from pyeeg import ap_entropy

version = 'original' #original, trans, shift, or rolling
key = Settings['Label'] #Mean1 default key for one time series
start = 0 #seconds, where you want the slice to begin
end = 1 #seconds, where you want the slice to end. The absolute end is -1

ap_entropy(Data[version][key][start:end].tolist(), 2, (0.2*np.std(Data[version][key][start:end])))


I am working on the file "BASS v2.0.ipynb".
The first task is:
Time Series
Use the settings code block to set your frequency bands to calculate area under the curve. This block is not required. band output is always in raw power, even if the graph scale is dB/Hz.
Power Spectral Density: Signal
Can you write Python code for it?

#optional
Settings['PSD-Signal'] = Series(index = ['ULF', 'VLF', 'LF','HF','dx'])

#Set PSD ranges for power in band
Settings['PSD-Signal']['ULF'] = 25 #max of the range of the ultra low freq band. range is 0:ulf
Settings['PSD-Signal']['VLF'] = 75 #max of the range of the very low freq band. range is ulf:vlf
Settings['PSD-Signal']['LF'] = 150 #max of the range of the low freq band. range is vlf:lf
Settings['PSD-Signal']['HF'] = 300 #max of the range of the high freq band. range is lf:hf. hf can be no more than (hz/2) where hz is the sampling frequency
Settings['PSD-Signal']['dx'] = 2 #segmentation for integration of the area under the curve.


I am working on the file "BASTA_2020_matplotlib_presentation2.ipynb".
The first task is:
Alle defaults kontrollieren
Alle defaults befinden sich in matplotlib.rcParams. Die kann mann verändern und es kontrolliert alle plots. rcParams ist ein dictionnary.
Can you write Python code for it?

import matplotlib as mpl
mpl.rcParams['font.size'] = 18
mpl.rcParams['legend.fontsize'] = 16
mpl.rcParams['xtick.labelsize'] = 20
mpl.rcParams['xtick.top'] = True
mpl.rcParams['xtick.major.size'] = 12
plt.hist([norm_dist1,norm_dist2], bins=20, density=True)
plt.show()
plt.close()


I am working on the file "BASTA_2020_matplotlib_presentation2.ipynb".
The first task is:
Anstatt der Format string kann man auch alles einzel kontrollieren:
* color: Farbe
* linestyle: Linien Typ
* marker: Symbol
Can you write Python code for it?

plt.figure()
plt.plot(x, y1, marker="x", color="g")
plt.plot(x, y2, linestyle="--", color="m")
plt.show()
plt.close()


I am working on the file "BASTA_2020_matplotlib_presentation2.ipynb".
The first task is:
Daten plotten

Die plt.figure Funktion generiert eine neue Figur
Mit der plt.plot Funktion kann man Daten plotten.
plt.show zeigt den plot auf dem Bildschirm
plt.savefig speichert die Figur
plt.close schliesst die Figur
Can you write Python code for it?

plt.figure()
plt.plot(x, y1)
plt.show()
#plt.savefig("path/to/plot.png")
plt.close()


I am working on the file "BASTA_2020_matplotlib_presentation2.ipynb".
The first task is:
Die plt.hist funktion gibt die bin Anfang und Ende, die Werte, und dir Patches zurück. Mann kann das brauchen um einen Normaler Linien Plot zu generieren, der einfacher Lesbar ist
Can you write Python code for it?

hist_y, hist_x, p = plt.hist(norm_dist1, bins=20, density=True)
hist_x = 0.5 * (hist_x[1:] + hist_x[:-1])
plt.close()
plt.figure()
plt.plot(hist_x, hist_y, "rx-")
plt.show()
plt.close()


I am working on the file "BASTA_2020_matplotlib_presentation2.ipynb".
The first task is:
Errorbar plotten
Errorbars werden mit plt.errorbar erstellt
Can you write Python code for it?

y1_err = np.random.normal(0, 1, len(x))
x_err = np.random.normal(0, 1, len(x))
plt.figure()
plt.errorbar(x, y1, yerr=y1_err, marker = "o", capsize = 5)
plt.errorbar(x, y2, xerr=x_err, marker = "x", capsize = 6)
plt.show()
plt.close()


I am working on the file "BASTA_2020_matplotlib_presentation2.ipynb".
The first task is:
Linie und Symbol formattierung

plt.plot nimmt als Optionales Positional Argument eine format string, um Farbe, Linie/Symbol zu kontrollieren
Can you write Python code for it?

plt.figure()
plt.plot(x, y1, "b")
plt.plot(x, y2, "g--")
plt.show()
plt.close()


I am working on the file "BASTA_2020_matplotlib_presentation2.ipynb".
The first task is:
Linien und text

plt.vlines um vertikale Linien hinzufügen
plt.hlines um horizontale Linien hinzufügen

Die nehmen als erstes argument die Position (oder eine Liste von Positionen) wo die Liene sein soll, und dann wo si anfängt und aufhört
Can you write Python code for it?

plt.hist([norm_dist1,norm_dist2], bins=20, density=True)
plt.hlines(0.5, plt.xlim()[0], plt.xlim()[1], linestyle=":", color="r")
plt.vlines([0,1.0],*plt.ylim(),linestyle = "--")
plt.show()
plt.close()


I am working on the file "BASTA_2020_matplotlib_presentation2.ipynb".
The first task is:
Mann kann auch das histogram für mehrere datensätze gleichzeitig rechnen
Can you write Python code for it?

norm_dist1 = np.random.normal(0, 1, 1000)
norm_dist2 = np.random.normal(1, 0.5, 1000)
plt.hist([norm_dist1,norm_dist2], bins=20, density=True)
plt.show()
plt.close()


I am working on the file "BASTA_2020_matplotlib_presentation2.ipynb".
The first task is:
Subplots

Eine Figure mit mehreren Plots wird mit plt.subplots generiert. Gibt die Figur und eine liste von axes zurück
Dan einfach die elementen von axes brauchen um zu plotten
mit plt.subplots_adjust kann man die position von den subplots kontrollieren

Für axes Objekten werden mehrere Methoden von plt umgenennt: set_ wird vor dem Namen eingefügt. Zum Beispiel:
* plt.title -> ax.set_title
* plt.xlabel -> ax.set_xlabel
* plt.xlim -> ax.set_xlim
Can you write Python code for it?

f, axes = plt.subplots(2,2,sharey = "row", figsize = (10,8))
f.suptitle("overall title")
axes[0,0].plot(x,y1)
axes[0,1].errorbar(x, y2, y1_err, capsize = 5)
axes[1,0].hist(norm_dist1, density=True, bins=20)
axes[1,1].hist(norm_dist1, density=True, bins=20, cumulative=True)
for i,axl in enumerate(axes):
    for j,ax in enumerate(axl):
        ax.set_title("subtitle ({},{})".format(i,j))
        ax.set_xlabel("x-label ({},{})".format(i,j))
    axl[0].set_ylabel("y-label")
plt.subplots_adjust(right = 0.98, wspace = 0.02, hspace = 0.25)
plt.show()


I am working on the file "BASTA_2020_matplotlib_presentation2.ipynb".
The first task is:
Title und Achsen Labels
plt hat Funktionen um alle Aspekten von der Figur zu Kontrollieren:
* plt.title um einen Title zu setzen
* plt.xlabel und plt.ylabel für die Achsen Titeln  zu setzen
* plt.xlim und plt.ylim um die Achsen Limiten zu setzen
Can you write Python code for it?

plt.figure()
plt.plot(x, y1)
plt.xlabel("x-label")
plt.ylabel("y-label")
plt.title("My plot title")
plt.show()
plt.close()


I am working on the file "BASTA_2020_matplotlib_presentation2.ipynb".
The first task is:
nbins kontrolliert wieviele bins man brauchen will.
normed kontrolliert ob man die Number von Daten in jedem bin will oder die Probabilität Verteilung
Can you write Python code for it?

plt.hist(norm_dist, density=True, bins=20)
plt.show()
plt.close()


I am working on the file "BASTA_2020_matplotlib_presentation2.ipynb".
The first task is:
plt.text(xpos, ypos, text) um text hinzufügen an der Position (xpos, ypos)
plt.annotate(text, xy=(xpos, ypos), xytext=(text_x, text_y)) um text hinzufügen an der Position (text_x, text_y) mit einem Pfeil der auf (xpos, ypos) zeigt.
Can you write Python code for it?

plt.figure()
plt.plot(x, y1,"x")
plt.text(0,0, "text at position 0,0", fontsize=16)
plt.annotate("point to 1,1",xy=(1, 1), xytext=(3, 2),arrowprops=dict(facecolor='black'))
plt.show()
plt.close()


I am working on the file "BDFEX_mixing.ipynb".
The first task is:
And plot the stability constraint in the neighborhood of the minimum:
Can you write Python code for it?

n = 200
alphas = np.linspace(0., 1., num=n)
betas = np.linspace(-2.0, 1.0, num=2*n)
cfls = np.zeros((n,2*n))
for i in range(n):
    for j in range(2*n):
        cfls[i,j] = mix_bdfex_34(alphas[i], betas[j])
plt.imshow(cfls.transpose(), origin='lower')
plt.colorbar()
foo = np.unravel_index(np.argmax(cfls), cfls.shape)

mix_bdfex_34(0,-1.4682977182989303, plot=True)

def mix_bdfex_24(alpha, beta, plot=False):
    res = stab_region((1.-beta)*bdf2+beta*bdf4,
                      (1.-alpha)*ex2+alpha*ex4,
                     plot=plot);
    if plot:
        print("The eigenvalue for BDF/EX-2,4({},{}) is {}".format(alpha, beta, res))
    return res

def wrap(x):
    return -mix_bdfex_24(x[0], x[1])
def accept(f_new, x_new, f_old, x_old):
    if x_new[0] < 0:
        return False
#    if x_new[1] < 0:
#        return False
#    if x_new[0] + x_new[1] > 1.0:
#        return False
#    if x_new[2] < 0:
#        return False
#    if x_new[3] < 0:
#        return False
#    if x_new[2] + x_new[3] > 1.0:
#        return False
#    if x_new[1] < 0:
#        return False
    return True

res = basinhopping(wrap,
                   [1.0, 1.0],
                   accept_test=accept,
                   minimizer_kwargs={'method': 'SLSQP', 'bounds': [(0, None), (None, None)]},
                   niter=1000,
                   niter_success=500,
                   T=0.01)
mix_bdfex_24(res.x[0], res.x[1], plot=True);

def mix_bdfex_234(alpha1, alpha2, beta1, beta2, plot=False):
    res = stab_region((1.-beta1 - beta2)*bdf2+beta1*bdf3 + beta2*bdf4,
                      (1.-alpha1 - alpha2)*ex2+alpha1*ex3 + alpha2*ex4,
                     plot=plot);
    if plot:
        print("The eigenvalue for BDF/EX-2,3,4({}-{},{}-{}) is {}".format(alpha1, alpha2, beta1, beta2, res))
    return res

interact(mix_bdfex_234, alpha1=.667, alpha2=0., beta1=.48, beta2 = 0., plot=True);

plt.figure(figsize=(15,15))
mix_bdfex_234(.633, .4, -.48, 0.0, plot=True)

mix_bdfex_234(0,-0.250000001,-2.5,-3, plot=True)

def wrap(x):
    return...


n = 200
alphas = np.linspace(0., 1., num=n)
betas = np.linspace(-2.0, 1.0, num=2*n)
cfls = np.zeros((n,2*n))
for i in range(n):
    for j in range(2*n):
        cfls[i,j] = mix_bdfex_23(alphas[i], betas[j])
plt.imshow(cfls.transpose(), origin='lower')
plt.colorbar()
foo = np.unravel_index(np.argmax(cfls), cfls.shape)


I am working on the file "BDFEX_mixing.ipynb".
The first task is:
For example, the stability regions of BDF2/EX2 and BDF3/EX3 are below:
Can you write Python code for it?

print("The maximum convection eigenvalue for BDF2/EX3 is {}".format(
        stab_region(bdf1, ex3, label="BDF2/EX2")))
print("The maximum convection eigenvalue for BDF3/EX3 is {}".format(
        stab_region(bdf3, ex3, label="BDF3/EX3")))
print("The maximum convection eigenvalue for BDF4/EX4 is {}".format(
        stab_region(bdf4, ex4, label="BDF4/EX4")))


I am working on the file "BDFEX_mixing.ipynb".
The first task is:
From this, it is clear to see that BDF2/EX2 is not suitable for the convection operator but BDF3/EX3 is.
Interestingly, BDF2/EX3 is stable, so this is the 2nd order scheme implemented in codes like Nek5000.
Can you write Python code for it?

print("The maximum convection eigenvalue for BDF2/EX2 is {}".format(
        stab_region(bdf2, 2./3.*ex3+1./3.*ex2, label="BDF2/EX2")))


I am working on the file "BDFEX_mixing.ipynb".
The first task is:
Introduction
Consider the PDE:
$$ \frac{d \phi}{d t} = L(\phi) $$
where $L$ is a linear operator.
First, we discretize the LHS with a backwards difference formula (BDF):
$$ \sum \beta_i \phi^{n-i} = L(\phi^n) $$
where $\beta$ are the BDF coefficients.
Next, we discretize the RHS with explicit extrapolation (EX), likely motivated by discrete versions of $L$ having a high condition number:
\begin{equation}
\sum_{i=0} \beta_i \phi^{n-i} = \sum_{i=1} \chi_i L \phi^{n-i}
\end{equation}
where $\chi$ are the extrapolation coefficients.
The standard coefficients, those which achieve maximal convergence order, are found below.
Can you write Python code for it?


I am working on the file "BDFEX_mixing.ipynb".
The first task is:
Mixing BDF/EX-N and BDF/EX-N+1
The same procedure can be used to mix BDF/EX3 and BDF/EX4.
Can you write Python code for it?

def mix_bdfex_34(alpha, beta, plot=False):
    res = stab_region((1.-beta)*bdf3+beta*bdf4,
                      (1.-alpha)*ex3+alpha*ex4,
                     plot=plot);
    if plot:
        print("The eigenvalue for BDF/EX-3,4({},{}) is {}".format(alpha, beta, res))
    return res

interact(mix_bdfex_34, alpha=.5, beta=.5, plot=True);


I am working on the file "BDFEX_mixing.ipynb".
The first task is:
We can maximize the stability constraint.  Again, we must impose $\alpha \ge 0$.
Can you write Python code for it?

def wrap(x):
    return -mix_bdfex_34(x[0], x[1])
def accept(f_new, x_new, f_old, x_old):
    if x_new[0] < 0:
        return False
#    if x_new[1] < 0:
#        return False
    return True
res = basinhopping(wrap,
                   [0.5, 0.5],
                   accept_test=accept,
                   minimizer_kwargs={'method': 'SLSQP', 'bounds': [(0., None), (None, None)]},
                   niter=1000,
                   niter_success=500,
                   T=0.01)
mix_bdfex_34(res.x[0], res.x[1], plot=True);