I am working on the file "SA_and_PSO.ipynb".
The first task is:
何回か繰り返したが大体48を返すxの組み合わせで更新が止まってしまう（以下で再検討）。
②次にPSOで検討する。以下は提出したコードと同じ、PSOのコードである。
Can you write Python code for it?

#粒子を表す。
class particle:
    #引数の意味はx:粒子の位置、v:粒子の速度、alpha,beta,ganmaは速度更新で重みの調整に使われる。
    def __init__(self, x, v, omega, alpha, beta, ganma):
        self.x = x
        self.v = v
        self.omega = omega
        self.alpha = alpha
        self.beta = beta
        self.ganma = ganma
        self.lbest_x = x
        self.lbest = 10000

    #その粒子のローカルベストを返す関数。
    def local_best(self):
        cal = rosenbrock(self.x)
        if self.lbest > cal:
            self.lbest = cal
            self.lbest_x = self.x

    #vの値の更新。
    def renew_v(self, gbest_x):
        r_1 = np.random.uniform()
        r_2 = np.random.uniform()
        self.v = self.ganma*(self.omega*self.v + (self.lbest_x - self.x)*self.alpha*r_1 + (gbest_x - self.x)*self.beta*r_2)
    #xの値の更新。
    def renew_x(self):
        self.x = self.x + self.v

#PSO全体のクラス。
class PSO:
    #引数の意味はN:変数の数、pN:粒子の数、limit_x:変数（粒子の位置）の下限と上限、残りはそれぞれ対応する変数の下限と上限（値は経験的に決めた）。
    def __init__(self, N=50, pN=100, limit_x=[-2.048, 2.048], limit_v=[0.0, 0.01], limit_omega=[0.8, 1.0], limit_alpha=[0.0, 1.0], limit_beta=[0.0, 1.0], limit_ganma=[0.8, 1.0]):
        #Alpha is Attenuation coefficient.
        #Omega is Convergence factor.
        #N is the number of iteration. Exit condition is given by the number of iteration.
        self.N = N
        self.pN = pN
        self.limit_x = limit_x
        self.limit_v = limit_v
        self.limit_omega = limit_omega
        self.limit_alpha = limit_alpha
        self.limit_ganma = limit_ganma
        self.limit_beta = limit_beta
        self.GBest = 10000
        self.GBest_x = np.array([np.random.uniform(limit_x[0], limit_x[1]) for i in range(N)])

    #以下、initialize_?関数は?の初期化を行う。
    def initialize_x(self):
        x = np.array([np.random.uniform(self.limit_x[0], self.limit_x[1]) for i in range(self.N)])
        return x

    def initialize_v(self):
        v = np.array([np.random.uniform(self.limit_v[0], self.limit_v[1]) for i in range(self.N)])
        return v

    def initialize_omega(self):
  ...


I am working on the file "SA_and_PSO.ipynb".
The first task is:
大体、48.9ぐらいの値を返す時点で更新をやめてしまっている。
2 PSOの収束を確認する。rosenbrock関数が返す数がある値になって、xの更新が行われなくなったら試行をとめるテストをしたかったが、収束が終わらなかったので、100回試行するごとにグローバルベストを出力させる。
Can you write Python code for it?

pso = PSO()
for i in range(100):
    pso.simulate(100)
    print pso.GBest


I am working on the file "SA_and_PSO.ipynb".
The first task is:
大体、すべての値が0.52の近くにいる。これはrosenbrock関数の形的にすべての値が等しいときに、局所最適解になるからだと考えられる。
ここで10000回追加で試行すると、
Can you write Python code for it?


I am working on the file "SA_and_PSO.ipynb".
The first task is:
平均的な推移をみるために30回繰り返す実験を10回行っている。
Can you write Python code for it?


I am working on the file "SA_and_PSO.ipynb".
The first task is:
続いて100回の試行を100回繰り返した（上と合わせて計20000回）が収束は続いている。
Can you write Python code for it?

for i in range(20):
    pso.simulate(500)
    print pso.GBest


I am working on the file "SA_and_PSO.ipynb".
The first task is:
続いて500回の試行を20回繰り返した（上と合わせて計40000回）が収束は続いている。
キリがないので、ここでやめる。この時点でのグローバルベストを返す粒子の位置を確認する。
Can you write Python code for it?


I am working on the file "SBARepayEDA.ipynb".
The first task is:

4. Adding Macroeconomic Indicators
As a an indicator of economic state at the time a loan is awarded, we will use S&P 1500 composite index.
Source: http://www.econ.yale.edu/~shiller/data.htm
Can you write Python code for it?

# Load monthly consumer price index and composite S&P index data
cpi_sp = pd.read_excel('ie_data.xls', sep = ' ', skiprows=7)
# Use only values starting in 1990
cpi_sp  = cpi_sp[cpi_sp['Date']>1989]

# Keep only the relevant columns: date (year+month), S&P 1500 index and Consumer Price Index (CPI)
cpi_sp = cpi_sp[['Date', 'P', 'CPI']]

# Rename the columns
cpi_sp.columns = ['Date', 'SP', 'CPI']
cpi_sp.head()

# Change the format of the date from year.month to datetime (and deal with issues of .10 getting shortented to .1)
def split_date(x):
    tmp = str(x).split('.')
    if tmp[1] == '1':
        tmp[1] = '10'
    return tmp
cpi_sp['Date_dt'] = cpi_sp['Date'].apply(split_date)
s = '-'
cpi_sp['Date_dt'] = cpi_sp['Date_dt'].apply(lambda x: s.join(x))
cpi_sp['Date_dt'] = cpi_sp['Date_dt'].apply(lambda x: x + '-01')
cpi_sp.reset_index()
cpi_sp.head(10)

cpi_sp.drop('Date', axis = 1, inplace = True)
# Rename the columns
cpi_sp.columns = ['SP', 'CPI', 'Date']

# Adjust S&P based on CPI, relative to January 2016 US Dollar value
cpi2016 = float(cpi_sp[cpi_sp['Date'] == '2016-01-01']['CPI'])
cpi_sp['CPI_to2016'] = cpi2016/cpi_sp['CPI']
cpi_sp['SP_to2016'] = cpi_sp['SP'].multiply(cpi_sp['CPI_to2016'])
cpi_sp.drop(['CPI', 'SP'], axis = 1, inplace = True)
cpi_sp.reset_index(inplace = True, drop = True)
cpi_sp.head()

# Plot monhtly S&P (adjusted to 2016)
fig = plt.figure(figsize = (10,5))
axes = fig.add_subplot(1, 1, 1, facecolor ='gainsboro')
axes.plot(pd.to_datetime(cpi_sp['Date']), cpi_sp['SP_to2016'], c = 'k', linewidth = 3)
plt.xlabel('Year', fontsize = 20)
plt.ylabel('S&P 1500', fontsize = 20)
plt.xticks(fontsize = 20)
plt.yticks(fontsize = 20)
plt.tight_layout()
plt.show()

fig.savefig('1_SP1500.png', dpi = 300)

# Use S&P value relative to loan's approval date
cpi_sp.columns = [['ApprovalDate', 'CPI_to2016', 'SP_to2016']]

# Merge loan data with CPI + S&P data
# Keep only the date and set it to first of the month in order to use CPI and SP data
loans_7a['ApprovalDate'] = loans_7a['ApprovalDate'].apply(lambda...


I am working on the file "SBARepayEDA.ipynb".
The first task is:

5. Grouping Businesses into Industry Sectors
NAICS codes may be grouped into 20 broad business sectors.
Can you write Python code for it?

code_groups = {'Agriculture': 11, 'Mining' : 21, 'Utilities' : 22,
'Construction' : 23, 'Manufacturing' : [31, 32, 33], 'Wholesale Trade' : 42,
'Retail Trade' : [44, 45], 'Transportation & Warehousing' : [48, 49],
'Information' : 51, 'Finance & Insurance' : 52,
'Real Estate Rental & Leasing' : 53,
'Professional, Scientific, & Technical Services' : 54,
'Management of Companies & Enterprises': 55,
'Administrative/Support/Waste Management' : 56,
'Educational Services' : 61,
'Health Care & Social Assistance' : 62,
'Arts, Entertainment, & Recreation' : 71,
'Accommodation & Food Services' : 72,
'Other Services' : 81,
'Public Administration' : 92}

# Get the "Sector" variable from NAICS codes

# Only the first two digits of NAICS code are relevant to group into 20 sectors
loans_7a_matured['Sector'] = (loans_7a_matured['NaicsCode'].apply(lambda x:x/10000))
loans_7a_matured['Sector'].fillna(0, inplace = True)
loans_7a_matured['Sector'] = (loans_7a_matured['Sector'].apply(int))

code_groups.values()

tmp = 0
list_codes = []
list_cat = []
for i in range(len(code_groups.values())):
    val = list(code_groups.values())[i]
    if type(val) == int:
        list_codes.append(val)
        list_cat.append(list(code_groups.keys())[i])

    else:
        for j in range(len(val)):
            list_codes.append(val[j])
            list_cat.append(list(code_groups.keys())[i])

code_cat = pd.DataFrame(list_codes, columns = ['Code'])
code_cat['Sector'] = list_cat
code_cat

cols = list(loans_7a_matured.columns)
cols[-1] = 'Code'

loans_7a_matured.columns = cols
loans_7a_matured = pd.merge(loans_7a_matured, code_cat, on = 'Code', how = 'left')
loans_7a_matured.head()

loans_7a_matured['Sector'].isnull().sum()

loans_7a_matured.dropna(subset = ['Sector'], inplace = True)

loans_7a_matured.reset_index(inplace = True, drop=True)

# Keep only paid in full or defaulted loans
loans_7a_matured = loans_7a_matured[(loans_7a_matured['LoanStatus']=='PIF') |
                                                            ...


I am working on the file "SBARepayEDA.ipynb".
The first task is:

6. Examining State Information
Can you write Python code for it?

# Examine state information (borrower location)

borr_states = set(loans_7a_matured['BorrState'].unique())

states = ["AL", "AK", "AZ", "AR", "CA", "CO", "CT", "DC", "DE", "FL", "GA",
          "HI", "ID", "IL", "IN", "IA", "KS", "KY", "LA", "ME", "MD",
          "MA", "MI", "MN", "MS", "MO", "MT", "NE", "NV", "NH", "NJ",
          "NM", "NY", "NC", "ND", "OH", "OK", "OR", "PA", "RI", "SC",
          "SD", "TN", "TX", "UT", "VT", "VA", "WA", "WV", "WI", "WY"]

#US teritories
borr_states.difference(states)

#Count all the "odd" ones (not one of 50 states)
for s in borr_states.difference(states):
   print((loans_7a_matured['BorrState'] == s).sum())

#PR has 9834 loans, all others < 1000

print((loans_7a_matured['BorrState']).isnull().sum())

# Number of missing values for states
print((loans_7a_matured['BorrState']).isnull().sum())

# Drop the territories except for PR

for s in {'FM', 'MP', 'GU', 'MH', 'PW', 'AE', 'VI', 'AS', 'TQ'}:
     loans_7a_matured = loans_7a_matured[loans_7a_matured['BorrState'] != s]

# Drop missing values
loans_7a_matured = loans_7a_matured[loans_7a_matured['BorrState'].notnull()]

loans_7a_matured.info()


I am working on the file "SBARepayEDA.ipynb".
The first task is:

7. Encoding Categorical Variables
Can you write Python code for it?

#One-hot encoding: state, year approved, category, business type, loan tatus
loans_7a_matured = pd.concat([loans_7a_matured, pd.get_dummies(loans_7a_matured['BorrState'], drop_first=True)], axis  = 1)
loans_7a_matured = pd.concat([loans_7a_matured, pd.get_dummies(loans_7a_matured['Sector'], drop_first=True)], axis  = 1)
loans_7a_matured = pd.concat([loans_7a_matured, pd.get_dummies(loans_7a_matured['LoanStatus'], drop_first = True)], axis  = 1)
loans_7a_matured = pd.concat([loans_7a_matured, pd.get_dummies(loans_7a_matured['BusinessType'], drop_first = True)], axis  = 1)
loans_7a_matured = pd.concat([loans_7a_matured, pd.get_dummies(loans_7a_matured['ApprovalFiscalYear'], drop_first = True)], axis  = 1)
loans_7a_matured.reset_index(inplace = True, drop = True)

list(loans_7a_matured.columns)

# Drop those columns that we won't use in the (binary classification) model
loans_7a_matured.drop(['BorrState', 'SBAGuaranteedApproval', 'ApprovalDate', 'FirstDisbursementDate',
 'NaicsCode', 'BusinessType', 'LoanStatus', 'ChargeOffDate', 'GrossChargeOffAmount',
 'CPI_to2016','GrossChargeOffAmountAdj', 'MaturityDate','MaturityYear', 'Code', 'Sector', ], axis = 1, inplace = True)

#Save for later use
pd.to_pickle(loans_7a_matured, 'loans_7a_matured')


I am working on the file "SBARepayEDA.ipynb".
The first task is:
About 36% of businesses are non-employers!
Jobs Supported = "Total Jobs Created + Jobs Retained as reported by lender on SBA Loan Application.  SBA does not review, audit, or validate these numbers - they are simply self-reported, good faith estimates by the lender."
Can you write Python code for it?


I am working on the file "SBARepayEDA.ipynb".
The first task is:
Adjust all dollar amounts to 2016 US Dollar value
Can you write Python code for it?

# Adjust all dollar amounts to 2016 US Dollar value
loans_7a['SBAGuaranteedApprovalAdj'] = loans_7a['SBAGuaranteedApproval'].multiply(loans_7a['CPI_to2016']).astype('float64')
loans_7a['GrossChargeOffAmountAdj'] = loans_7a['GrossChargeOffAmount'].multiply(loans_7a['CPI_to2016']).astype('float64')

# Get a sense of number of each Loan Status per approval year
clrs    = ["#4878CF", "#6ACC65", "#D65F5F", "#B47CC7", "#C4AD66", "#77BEDB"]
newPal   = dict(PIF = clrs[0], DEFAULT = clrs[2], CANCLD = clrs[1], COMMIT = clrs[3], EXEMPT = clrs[4])

g = sns.factorplot(x ='ApprovalFiscalYear', y = 'SBAGuaranteedApprovalAdj',
            hue = 'LoanStatus', hue_order = ['PIF', 'DEFAULT', 'CANCLD', 'COMMIT', 'EXEMPT'],
                   kind = 'point', data = loans_7a, estimator = np.count_nonzero,
               ci = None, size = 5, aspect = 3, palette = newPal, legend = False)

g.set_xticklabels(rotation = 30)

sns.plt.title('7A Loans')
sns.plt.xlabel('Approval Fiscal Year')
sns.plt.ylabel('Number of Loans')
sns.plt.legend(['Paid in Full', 'Defaulted', 'Cancelled', 'Committed', 'Exempt'], loc = 'best', fontsize = 20)

g.savefig('2_AllLoans_Counts_FY.png', dpi = 300)


I am working on the file "SBARepayEDA.ipynb".
The first task is:
Franchise code is 0 or 1 for non-franchises and a 5-digit code otherwise!
Can you write Python code for it?

# Percent of franchises
(loans_7a['FranchiseCode'] > 1).sum()/len(loans_7a)*100


I am working on the file "SBARepayEDA.ipynb".
The first task is:
Let's check the correlation beetween the total approval amount and guaranteed amount.
Can you write Python code for it?


I am working on the file "SBARepayEDA.ipynb".
The first task is:
Most businesses receiving SBA loans are corporations, but a large portion are individuals, too.
Let's examine how many jobs these businesses support.
Can you write Python code for it?


I am working on the file "SBARepayEDA.ipynb".
The first task is:
Our target variable could be Loan Status (for binary classification) or  Charged Off Amount (for continuos prediction/regression).
We may choose to limit the data to only:

Loans that have been completed (paid in full or defaulted/charged off), or
Completed loans that have matured?
Can you write Python code for it?

# See how many loans there are in the data set per each status
loans_7a['LoanStatus'].value_counts()
sns.countplot(x = 'LoanStatus', data = loans_7a)


I am working on the file "SBARepayEDA.ipynb".
The first task is:
Since these two quantitites are highly correlated we will drop the gross approval amount.
Can you write Python code for it?

loans_7a.drop('GrossApproval', axis = 1, inplace = True)

loans_7a['BusinessType'].value_counts()
sns.countplot(x = 'BusinessType', data = loans_7a)


I am working on the file "SBARepayEDA.ipynb".
The first task is:
The plot above may be too optimistic since it accounts for loans that were paid in full before they were due, and does not account for loans that have been awarded but may default in the future.
Can you write Python code for it?

# Look at Maturity Date and only use loans whose maturity data has pased
loans_7a['MaturityDate'] = loans_7a['ApprovalDate'] + loans_7a['TermInMonths'].values.astype("timedelta64[M]")
loans_7a['MaturityYear'] = loans_7a['MaturityDate'].dt.year
loans_7a_matured = loans_7a[loans_7a['MaturityDate'] < '2017-07-01']
loans_7a_matured.reset_index(inplace = True, drop = True)

loans_7a_matured['MaturityYear'].isnull().sum()

loans_7a_matured.info()

# Plot amount paid in full and amount charged off per approval year when accounting for matured loans only

n_groups = loans_7a_matured['ApprovalFiscalYear'].nunique()

loans_7a_by_AY_by_LoanStatus = loans_7a_matured.groupby(by = ['ApprovalFiscalYear', 'LoanStatus'], as_index = False).sum()

sum_pif = loans_7a_by_AY_by_LoanStatus[loans_7a_by_AY_by_LoanStatus['LoanStatus']=='PIF']['SBAGuaranteedApprovalAdj']

sum_def = loans_7a_by_AY_by_LoanStatus[loans_7a_by_AY_by_LoanStatus['LoanStatus']=='DEFAULT']['GrossChargeOffAmountAdj']

fig = plt.figure(figsize = (16, 6))
ax = fig.add_subplot(1, 1, 1, facecolor = "gainsboro")
sns.set_context('poster', font_scale = 1.2)

x = np.arange(1991, 2017)

y = np.array(sum_pif)[:-1]
z = np.array(sum_def)

ax.bar(x-0.4, y,width=0.5,color='b',align='center')
ax.bar(x, z,width=0.5,color='r',align='center')
plt.xlabel('Approval Fiscal Year')
plt.xticks(loans_7a_matured['ApprovalFiscalYear'].unique(), rotation = 30)
plt.ylabel('2016 US Dollars', fontsize = 20)
plt.title('SBA 7A Loans (Matured)', fontsize = 20)
plt.legend(labels = ['Paid in Full', 'Defaulted'], fontsize = 20)
plt.tight_layout()
plt.show()

fig.savefig('4_Matured_per_FY.png', dpi = 300)

loans_7a_matured['MaturityYear'].unique()

loans_7a_matured['NaicsCode'].nunique()

# Missing NAICS codes?
loans_7a_matured['NaicsCode'].isnull().sum()/len(loans_7a_matured)


I am working on the file "SBARepayEDA.ipynb".
The first task is:
The plot bellow shows the actual amounts paid in full and charged off each year.
Can you write Python code for it?

# Group by year and loan status
loans_7a_by_FY_by_LoanStatus = loans_7a.groupby(by = ['ApprovalFiscalYear', 'LoanStatus'], as_index = False).sum()
loans_7a_by_FY_by_LoanStatus.head()

# Plot amount paid in full and amount charged off per approval year
fig = plt.figure(figsize = (16, 6))
ax = fig.add_subplot(1, 1, 1, facecolor ='gainsboro')

n_groups = loans_7a['ApprovalFiscalYear'].nunique()

sum_pif = loans_7a_by_FY_by_LoanStatus[loans_7a_by_FY_by_LoanStatus['LoanStatus']=='PIF']['SBAGuaranteedApprovalAdj']

sum_choff = loans_7a_by_FY_by_LoanStatus[loans_7a_by_FY_by_LoanStatus['LoanStatus']== 'DEFAULT']['GrossChargeOffAmountAdj']

x = np.arange(1991, 2017)

y = np.array(sum_pif)[:-1]
z = np.array(sum_choff)

ax.bar(x-0.4, y,width=0.5,color='b',align='center')
ax.bar(x, z,width=0.5,color='r',align='center')
plt.xlabel('Approval Fiscal Year', fontsize = 20)
plt.xticks(loans_7a['ApprovalFiscalYear'].unique(), rotation = 30)
plt.ylabel('2016 US Dollars', fontsize = 20)
plt.title('7A Loans', fontsize = 20)
plt.legend(labels = ['Paid in Full', 'Defaulted'], fontsize = 20)
plt.tight_layout()
plt.show()

fig.savefig('3_Completed_per_FY.png', dpi = 300)


I am working on the file "SBARepayEDA.ipynb".
The first task is:
There are more than 1300 different NAICS values in this dataset!
Can you write Python code for it?

loans_7a['NaicsCode'].isnull().sum()

# Remove rows with missing NAICS Codes
loans_7a.dropna(subset = ['NaicsCode'], inplace = True)

loans_7a.reset_index(drop = True, inplace = True)

# Replace CHGOFF with DEFAULT for simplicity
def repCHGOFF_DEFAULT(status):
    if status == 'CHGOFF':
        return 'DEFAULT'
    else:
        return status
loans_7a['LoanStatus'] = loans_7a['LoanStatus'].apply(repCHGOFF_DEFAULT)


I am working on the file "SBARepayEDA.ipynb".
The first task is:
We may be tempted to incorporate changes in S&P index during the loan pay-off period into the model; however,
1. we do not have information about this period for paid in full loans (end data is only indicated for charged off loans), and
2. doing so would make the model unusable for future loans (at any given point in time, we cannot predict future changes in S&P Index)
Can you write Python code for it?

loans_7a = pd.merge(loans_7a, cpi_sp, on = 'ApprovalDate', how = 'left')

loans_7a.head()

# Check for NaNs
loans_7a['SP_to2016'].isnull().sum()


I am working on the file "SBARepayEDA.ipynb".
The first task is:
We see a clear  spike in defaults for loans approved immediately before and during the U.S. financial crisis.
This indicates that macroeconomic factors may be important.
Can you write Python code for it?

# Group data by approval fiscal year
loans_7a_by_FY = loans_7a.groupby('ApprovalFiscalYear')

# Get a sense of the number of each Loan Status per approval year

g = sns.factorplot(x ='ApprovalFiscalYear', y = 'SBAGuaranteedApprovalAdj',
            hue = 'LoanStatus', hue_order = ['PIF', 'DEFAULT', 'CANCLD', 'COMMIT', 'EXEMPT'],
                   kind = 'bar', data = loans_7a, estimator = np.count_nonzero,
               ci = None, size = 5, aspect = 3, palette = newPal, legend = False)

g.set_xticklabels(rotation = 30)

sns.plt.title('7A Loans')
sns.plt.xlabel('Approval Fiscal Year')
sns.plt.ylabel('Number of Loans')
sns.plt.legend(['Paid in Full', 'Defaulted', 'Cancelled', 'Committed', 'Exempt'], loc = 'best', fontsize = 20)


I am working on the file "SBARepayML.ipynb".
The first task is:

4. Random Forest Feature Importances
Let's examine which features are contributing most in the optimal Random Forest model.
Can you write Python code for it?

#Feature ranking for random forest
fig = plt.figure(figsize = (10,5))
importances = opt_rf.feature_importances_
std = np.std([tree.feature_importances_ for tree in opt_rf.estimators_],
            axis=0)
indices = np.argsort(importances)[::-1]
fts = list(X_train.columns)
# Print the feature ranking
print("Feature ranking:")

for f in range(X_train.shape[1]):
   print("%d. feature %s (%f)" % (f + 1, fts[indices[f]], importances[indices[f]]))

# Plot the top ten feature importances of the optimized random forest method
importances = opt_rf.feature_importances_
std = np.std([tree.feature_importances_ for tree in opt_rf.estimators_],
             axis=0)
indices = np.argsort(importances)[::-1]

# Plot the feature importances of the forest
fig = plt.figure(figsize = (10,5))

plt.title("Feature Importances")
plt.barh(range(10), importances[indices][0:10][::-1],
       color="r", xerr = std[indices][0:10][::-1], align="center")
# If you want to define your own labels,
# change indices to a list of labels on the following line.
plt.yticks(range(10), ['Term in Months', 'Amount', 'S&P 1500', 'Jobs Supported', 'Revolver Status',
                      'Individual', 'CA', 'Retail Trade', 'Construction', 'FL'][::-1])

plt.yticks(range(10))
plt.ylim([-1,10])
plt.tight_layout()

fig.savefig('OPTIMAL_RF_FImportance.png', dpi = 300)


I am working on the file "SBARepayML.ipynb".
The first task is:
Again, Logistic Regression does not seem to be sufficient.
We will again tune hyperparamters for Random Forest using grid search with cross validation.
Can you write Python code for it?

# GridSearch for Random Forest
param_grid = {'max_features': [0.2, 'auto', 'log2'],
              'n_estimators': [50, 100], 'min_samples_leaf': [1, 5, 10, 50, 100]}
grid_rf_l84 = GridSearchCV(RandomForestClassifier(class_weight = 'balanced', n_jobs = 4),
                param_grid, cv = 10, refit = True, verbose = 3, scoring = 'f1')

grid_rf_l84.fit(X_train_l84, y_train_l84)

print(grid_rf_l84.best_params_)

# Run RF with optimal paramters
opt_rf_l84 = RandomForestClassifier(class_weight = 'balanced', max_features = 0.2, min_samples_leaf = 5, n_estimators = 100)
opt_rf_l84.fit(X_train_l84, y_train_l84)
opt_rf_l84_pred = opt_rf_l84.predict(X_test_l84)

print(classification_report(y_test_l84, opt_rf_l84_pred))


I am working on the file "SBARepayML.ipynb".
The first task is:
Average amount is lower for defaulted loans.
Can you write Python code for it?

# Check if there is a difference in mean number of jobs
t, prob = scipy.stats.ttest_ind(data[data['PIF']==0]['JobsSupported'], data[data['PIF']==1]['JobsSupported'] )
print(t, prob)

#Examine loans with TermDuration = 84
data_84 = data[data['TermInMonths'] == 84]
data_84.reset_index(inplace=True, drop = True)

data_84['PIF'].value_counts()


I am working on the file "SBARepayML.ipynb".
The first task is:
For loans of duration = 84 months, < 0.5% default!

5. Focusing on Shorter Duration Loans
Let's examine loans with duration < 84 months and repeat all the steps from above without TermInMonths.
Can you write Python code for it?

# Consider only loans with TermDuration < 84
data_l84 = data[data['TermInMonths'] < 84]
data.reset_index(inplace = True, drop = True)

target_l84 = y = (1 - data[data['TermInMonths'] < 84]['PIF'])

# Split into training and testing data as before
X_train_l84, X_test_l84, y_train_l84, y_test_l84 = train_test_split(data_l84.drop(['TermInMonths',  'PIF'],axis = 1),
                                                                target_l84,
                                                                stratify = target_l84,
                                                                test_size = 0.30, random_state = 101)

# Logistic Regression Classification
logmodel_l84 = LogisticRegression(class_weight = 'balanced')
logmodel_l84.fit(X_train_l84, y_train_l84)
lm_l84_pred = logmodel_l84.predict(X_test_l84)
print(classification_report(y_test_l84, lm_l84_pred))

# GridSearch for Logistic Regression
param_grid = {'C': [0.001, 0.005, 0.1, 0.5, 1, 5, 10, 50, 100, 500, 1000, 5000, 10000]}
grid_l84 = GridSearchCV(LogisticRegression(class_weight = 'balanced'), param_grid, refit=True,verbose=1)
grid_l84.fit(X_train_l84, y_train_l84)
print(grid_l84.best_params_)

grid_l84_pred = grid_l84.predict(X_test_l84)

print(classification_report(y_test_l84, grid_l84_pred))


I am working on the file "SBARepayML.ipynb".
The first task is:
From here on:

     Class 0 = Paid in Full

     Class 1 = Defaulted
We will first split our data into training and testing sets. We will use a stratified split since classes are somewhat imbalanced. We will always run model optimization/selection on the training data (by introducing an additional cross validation portion/k-fold cross validation) and testing on the untouched test set.
Can you write Python code for it?

# Set aside a test set
# Random stratified 70-30 split: preserves the original proportion of positive and negative class
# examples in train and test
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify = y,
                                                        test_size = 0.30, random_state = 101)


I am working on the file "SBARepayML.ipynb".
The first task is:
From the tree, it looks like the most important features are:
    * Term in Months (most decisions/splits are made based on this feature, and it looks like loans < 83 months are more likely to default)
    * S&P 1500 Index
    * Guaranteed Amount
Let's examine Term in Months in more detail.
Can you write Python code for it?

# Term in months seems to be the most important from the Decision Tree
fig = plt.figure(figsize = (5,3), facecolor = 'gainsboro')
sns.set_context('poster', font_scale = 1.2)

g = sns.factorplot(x = 'PIF', y = 'TermInMonths',
                   kind = 'bar', data = data, estimator = np.mean, palette = 'Set1' )

g.set_xticklabels(['Defaulted', 'Repaid'])
sns.plt.xlabel('')
sns.plt.title('7A Matured Loans')
sns.plt.ylabel('Mean Term in Months')
g.savefig('Term.png', dpi = 300)


I am working on the file "SBARepayML.ipynb".
The first task is:
Looks like duration is significantly longer for paid in full loans.
Can you write Python code for it?

# Check if there is a difference in mean amount
t, prob = scipy.stats.ttest_ind(data[data['PIF']==0]['SBAGuaranteedApprovalAdj'], data[data['PIF']==1]['SBAGuaranteedApprovalAdj'] )
print(t, prob)


I am working on the file "SBARepayML.ipynb".
The first task is:
Random (control) model has much lower precision and recall than random forest for loans with duration < 84 months.
Let's again examine feature importances.
Can you write Python code for it?

# Feature ranking for random forest
fig = plt.figure(figsize = (15,5))
importances_rf_l84 = opt_rf_l84.feature_importances_
std = np.std([tree.feature_importances_ for tree in opt_rf_l84.estimators_],
            axis=0)
indices_rf_l84 = np.argsort(importances_rf_l84)[::-1]
fts_rf_l84 = list(X_train_l84.columns)
# Print the feature ranking
print("Feature ranking:")

for f in range(X_train_l84.shape[1]):
   print("%d. feature %s (%f)" % (f + 1, fts_rf_l84[indices_rf_l84[f]], importances_rf_l84[indices_rf_l84[f]]))

# Plot the feature importances of the forest
importances = opt_rf_l84.feature_importances_
std = np.std([tree.feature_importances_ for tree in opt_rf_l84.estimators_],
             axis=0)
indices = np.argsort(importances)[::-1]

# Plot the feature importances of the forest
fig = plt.figure(figsize = (10,5))

plt.title("Feature importances")
plt.barh(range(10), importances[indices][0:10][::-1],
       color="r", xerr=std[indices][0:10][::-1], align="center")
# If you want to define your own labels,
# change indices to a list of labels on the following line.
plt.yticks(range(10), ['S&P 1500', 'Amount', 'Jobs Supported', 'Revolver Status',
                       'CA',  'FL', 'Individual', 'Retail Trade', 'GA', 'Franchise'][::-1])

plt.ylim([-1,10])
plt.tight_layout()

fig.savefig('L84_RF_FImportance.png', dpi = 300)


I am working on the file "SBARepayML.ipynb".
The first task is:
Target variable is Loan Status - Paid in Full (PIF) versus Defaulted. Right now:
* Negative class = defaulted.
* Postive class = repaid (paid in full).
Can you write Python code for it?

data['PIF'].value_counts()

# Percent of loans that defaulted out of paid in full + defaulted loans
print(str(np.round(100*(data['PIF']==0).sum() / len(data), 2)) + '% of matured loans defaulted')

# Select the features
X = data.drop(['PIF'], axis = 1)
# Select the target variable: switch class labels so that "defaulted" is the postive class
# since this is what we really care about
y = (1 - data['PIF'])

class_names = ['Paid in Full', 'Defaulted']


I am working on the file "SBARepayML.ipynb".
The first task is:
We can see that, relative to logistic regression, this model is overall doing better - precision of the positive class has significantly improved (from 0.75 to 0.94). However, this is only  slightly better in terms of recall (0.50 to 0.57).  We would like to further improve recall as well. Let's see what an individual tree looks like.
Can you write Python code for it?

# Visualize the tree
dot_data = tree.export_graphviz(dtree, out_file = None,
                         feature_names = X_train.columns,
                         class_names = ['DEF', 'PIF'],
                         filled = True, rounded = True,
                         special_characters = True)
graph = graphviz.Source(dot_data)
graph.render("dec_tree_simple")
graph


I am working on the file "SBARepayML.ipynb".
The first task is:
We see that many defaulted loans got labeled as paid in full (actually, as many as were classified correctly). We will try to improve the perofrmance by optimizing hyperparamters using a Grid Search with 10-fold Cross Validation (stratified split) on training data, picking the best model (optimal hyperparameters), and then applying it to the test data. One hyperparameter that is of importance to logistic regression is C - the amount of regularization used.
Can you write Python code for it?

# Tune the hyperparameters: vary the regularization paramter
# Try an exhaustive range of values

param_grid = {'C': [0.0001, 0.0005, 0.001, 0.005, 0.1, 0.5, 1, 5, 10, 50, 100, 500, 1000, 5000]}

# 10-fold cross validation on training data to tune C
grid_logmodel = GridSearchCV(LogisticRegression(class_weight = 'balanced'),
                             param_grid, refit = True, verbose = 1,
                             cv = StratifiedShuffleSplit(n_splits = 10, test_size = 0.3, random_state = 101))
grid_logmodel.fit(X_train_std, y_train)

# See the chosen optimal parameter
grid_logmodel.best_params_


I am working on the file "SBARepayML.ipynb".
The first task is:
We should aim to avoid predicting that a loan will be paid in full, when in fact it will default, i.e., we want to detect all of defaults (positive class). False negatives should be important. Therefore, we'll pay particular attention to recall  (of the positive/Default class).
This is exactly the metric that the above classifier struggles with - it is 0.50.
We can gain more insight as to what went wrong by examining the confusion matrix.
Can you write Python code for it?

# Function to display the confusion matrix - original or normalized

import itertools
def plot_confusion_matrix(cm, classes, title = 'Confusion matrix', cmap = plt.cm.Blues, normalize = False):
    """
    This function prints and plots the confusion matrix.
    """

    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
        print("Normalized confusion matrix")
    else:
      print('Confusion matrix, without normalization')

    print(cm)

    plt.imshow(cm, interpolation='nearest', cmap = cmap)
    plt.title(title, fontsize = 20)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, fontsize = 20)
    plt.yticks(tick_marks, classes, fontsize = 20)
    fmt = '.2f' if normalize else 'd'
    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, format(cm[i, j], fmt),
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")

    plt.tight_layout()
    plt.ylabel('True Label', fontsize = 20)
    plt.xlabel('Predicted Label', fontsize = 20)
    plt.grid(False)

# Plot confusion matrix without normalization

def_logreg_cm = confusion_matrix(def_logreg_pred, y_test)

plt.figure(figsize = (8,4))
plot_confusion_matrix(def_logreg_cm, classes = class_names, normalize = False,
                      title = 'Confusion Matrix')# Plot normalized confusion matrix
plt.figure(figsize = (8,4))
plt.figure(figsize = (8,4))
plot_confusion_matrix(def_logreg_cm, classes = class_names, normalize = True,
                      title = 'Confusion Matrix with Normalization')


I am working on the file "SBARepayML.ipynb".
The first task is:
We will first run logistic regression with default hyperparamters. We will use class_weight = 'balanced' to introduce higher penalty for missclassifying the minority class.
Can you write Python code for it?

# Define the model
def_logreg_model = LogisticRegression(class_weight = 'balanced', random_state = 101)
# Train the model on scaled training data
def_logreg_model.fit(X_train_std, y_train)
# Test the model: make predictions on testing data
def_logreg_pred = def_logreg_model.predict(X_test_std)
# Compare model outputs with actual outputs
print(classification_report(def_logreg_pred, y_test))


I am working on the file "SBARepayML.ipynb".
The first task is:
We will remove the approval year information in order to make the model useful for future years (not included in the data).
Can you write Python code for it?

data.drop(list(data.columns)[80:], axis = 1, inplace = True)
data.drop('ApprovalFiscalYear', axis = 1, inplace = True)


I am working on the file "SBARepayML.ipynb".
The first task is:
We will start with (L2-regularized) logistic regression with default hyperparameters. We will scale all numeric features prior to model training and testing.
Can you write Python code for it?

# Scale numerical features for logistic regression (with regularization)

from sklearn.preprocessing import StandardScaler

# Get scaling parameters from training data, then apply the scaler to testing data as well
std_scale = StandardScaler().fit(X_train[['TermInMonths', 'JobsSupported', 'SP_to2016', 'SBAGuaranteedApprovalAdj']])
X_train_std = std_scale.transform(X_train[['TermInMonths', 'JobsSupported', 'SP_to2016', 'SBAGuaranteedApprovalAdj']])
X_test_std = std_scale.transform(X_test[['TermInMonths', 'JobsSupported', 'SP_to2016', 'SBAGuaranteedApprovalAdj']])


I am working on the file "SC1/03_floating_point_arithmetic.ipynb".
The first task is:
<div id='func' />

Loss of significance in function evaluation
Loss of Significance is present too in the representation of functions. A classical example (which you can see in the guide book), is the next function:
\begin{equation}f(x)= \frac{1 - \cos x}{\sin^{2}x} \end{equation}
Applying trigonometric identities, we can obtain the 'equivalent' function:
\begin{equation}f(x)= \frac{1}{1 + \cos x} \end{equation}
Both of these functions are apparently equal. Nevertheless, its graphics say to us another thing when $x$ is equal to zero.
Can you write Python code for it?

x = np.arange(-10,10,0.1)
y = (1-np.cos(x))/(np.sin(x)**2)
plt.figure()
plt.plot(x,y,'.')
plt.grid(True)
plt.show()

x = np.arange(-10,10,0.1)
y = 1/(1+np.cos(x))
plt.figure()
plt.plot(x,y,'.')
plt.grid(True)
plt.show()

x = np.arange(-1,1,0.01)
y = (1-np.cos(x))/np.sin(x)**2
plt.figure()
plt.plot(x,y,'.',markersize=10)
plt.grid(True)
plt.show()

y = 1/(1+np.cos(x))
plt.figure()
plt.plot(x,y,'.',markersize=10)
plt.grid(True)
plt.show()


I am working on the file "SC1/03_floating_point_arithmetic.ipynb".
The first task is:
<div id='intro' />

Introduction
Hello! This notebook is an introduction to how our computers handle the representation of real numbers using double-precision floating-point format. To understand the contents of this notebook you should at least have a basic notion of how binary numbers work.
The aforementioned format occupies 64 bits which are divided as follows:

1 bit for the sign
11 bits for the exponent
52 bits for the mantissa

This means that the very next representable number after $1$ is $1 + 2^{-52}$, and their difference, $2^{-52}$, is $\epsilon_{mach}$.
Additionally, if you'd like to quickly go from a base-2 integer to a base-10 integer and viceversa, Python has some functions that can help you.
Can you write Python code for it?


I am working on the file "SC1/03_floating_point_arithmetic.ipynb".
The first task is:
As you can see, if you try to save a number between $1$ and $1 + \epsilon _{mach}$, it will have to be rounded (according to some criteria) to a representable number before being stored, thus creating a difference between the real number and the stored number. This is an example of loss of significance.
Does that mean that the "leap" between representable numbers is always going to be $\epsilon _{mach}$? Of course not! Some numbers will require smaller leaps, and some others will require bigger leaps.
In any interval of the form $[2^n,2^{n+1}]$ for a representable $n\in \mathbb{Z}$, the leap is constant. For example, all the numbers between $2^{-1}$ and $2^0$ (but excluding $2^0$) have a distance of $\epsilon {mach}/2$ between them. All the numbers between $2^0$ and $2^1$ (excluding $2^1$) have a distance of $\epsilon {mach}$ between them. Those between $2^1$ and $2^2$ (not including $2^2$) have a distance of $2\,\epsilon _{mach}$ between them, and so on and so forth.
Can you write Python code for it?


I am working on the file "SC1/03_floating_point_arithmetic.ipynb".
The first task is:
As you can see, the hypothesis was right. In other words: Floating point numbers are not linearly distributed across the real numbers, and the distance between them is proportional to their magnitude. Tiny numbers (~ 0) are closer to each other than big numbers are.
<div id='visualization' />

Visualization of floating point numbers
With the help of bitstring library we could write a function to visualize floating point numbers in their binary representation
Can you write Python code for it?

def to_binary(f):
    b = bs.pack('>d', f)
    b = b.bin
    #show sign + exponent + mantisa
    print(b[0]+' '+b[1:12]+ ' '+b[12:])


I am working on the file "SC1/03_floating_point_arithmetic.ipynb".
The first task is:
In order to prove our hypotesis, we will create an array of values: [1e-32, 1e-31, ..., 1e31, 1e32] and compute their corresponding epsilon.
Can you write Python code for it?

#values between 10**-32 and 10**+32
values = np.array([10**i for i in range(-32,32)]).astype(float)

#corresponding epsilons
vepsilon = np.vectorize(epsilon)
eps = vepsilon(values)


I am working on the file "SC1/03_floating_point_arithmetic.ipynb".
The first task is:
Let's see some intereseting examples
Can you write Python code for it?

to_binary(1.)

int('0b01111111111', 2)

to_binary(1.+epsilon(1.))

to_binary(+0.)

to_binary(-0.)

to_binary(np.inf)

to_binary(-np.inf)

to_binary(np.nan)

to_binary(-np.nan)

to_binary(2.**-1074)

print(2.**-1074)

to_binary(2.**-1075)

print(2.**-1075)

to_binary(9.4)


I am working on the file "SC1/03_floating_point_arithmetic.ipynb".
The first task is:
The distance from $2^{52}$ and the following number representable is $1$ !!!! So, what would happen if we were to store $2^{53}+1$?
Can you write Python code for it?


I am working on the file "SC1/03_floating_point_arithmetic.ipynb".
The first task is:
This means that if we want to store any number in the interval $[1,1+\epsilon_{\text{mach}}]$, only the numbers $1$ and $1+\epsilon_{\text{mach}}$ will be stored. For example, compare the exponent and the mantissa in the previous cell with the following outputs:
Can you write Python code for it?

for i in np.arange(11):
    to_binary(1+i*2**-55)


I am working on the file "SC1/03_floating_point_arithmetic.ipynb".
The first task is:
We can now scale this difference such that the scaling factor multiplied by $\epsilon_{\text{mach}}$ is one. The factor will be $2^{52}$. This means $2^{52}\,\epsilon_{\text{mach}}=1$. Repeating the same example as before, but with the scaling factor, we obtain:
Can you write Python code for it?


I am working on the file "SC1/03_floating_point_arithmetic.ipynb".
The first task is:
We now include a comparison between a linear scale plot and a loglog scale plot. Which one is more useful here?
Can you write Python code for it?

fig = plt.figure(figsize=(10,5))

plt.subplot(121)
plt.plot(values, eps,'.',markersize=20)
plt.xlabel('Values')
plt.ylabel('Corresponding Epsilons')
plt.title('Epsilons v/s Values')
plt.grid(True)

plt.subplot(122)
plt.loglog(values, eps,'.')
plt.xlabel('Values')
plt.ylabel('Corresponding Epsilons')
plt.title('Epsilons v/s Values')
plt.grid(True)

fig.tight_layout()
plt.show()


I am working on the file "SC1/03_floating_point_arithmetic.ipynb".
The first task is:
Which means we can only exactly store the numbers:
Can you write Python code for it?


I am working on the file "SC1/04_roots_of_1D_equations.ipynb".
The first task is:
<div id='fpi' />

Fixed Point Iteration
To learn about the Fixed-Point Iteration we will first learn about the concept of Fixed Point.
A Fixed Point of a function $g$ is a real number $r$, where $g(r) = r$
The Fixed-Point Iteration is based in the Fixed Point concept and works like this to find the root of a function:
\begin{equation} x_{0} = initial_guess \ x_{i+1} = g(x_{i})\end{equation}
To find an equation's solution using this method you'll have to move around some things to rearrange the equation in the form $x = g(x)$. That way, you'll be iterating over the funcion $g(x)$, but you will not find $g$'s root, but $f(x) = g(x) - x$ (or $f(x) = x - g(x)$)'s root. In our following example, we'll find the solution to $f(x) = x - \cos(x)$ by iterating over the funcion $g(x) = \cos(x)$.
Can you write Python code for it?


I am working on the file "SC1/04_roots_of_1D_equations.ipynb".
The first task is:

Introduction
Hello again! In this document we're going to learn how to find a 1D equation's solution using numerical methods. First, let's start with the definition of a root:
Definition: The function $f(x)$ has a root in $x = r$ if $f(r) = 0$.
An example: Let's say we want to solve the equation $x + \log(x) = 3$. We can rearrange the equation: $x + \log(x) - 3 = 0$. That way, to find its solution we can find the root of $f(x) = x + \log(x) - 3$. Now let's study some numerical methods to solve these kinds of problems.
Defining a function $f(x)$
Can you write Python code for it?


I am working on the file "SC1/04_roots_of_1D_equations.ipynb".
The first task is:
<div id='wilkinson' />

Wilkinson Polynomial
https://en.wikipedia.org/wiki/Wilkinson%27s_polynomial
Final question: Why is the root far far away from $16$?
Can you write Python code for it?

x = sym.symbols('x', reals=True)
W=1
for i in np.arange(1,21):
        W*=(x-i)
W # Printing W nicely

# Expanding the Wilkinson polynomial
We=sym.expand(W)
We

# Just computiong the derivative
Wep=sym.diff(We,x)
Wep

# Lamdifying the polynomial to be used with sympy
P=sym.lambdify(x,We)
Pp=sym.lambdify(x,Wep)

# Using scipy function to compute a root
root = optimize.newton(P,16)
print(root)


I am working on the file "SC1/04_roots_of_1D_equations.ipynb".
The first task is:
Building a FPI to compute the cubic root of 7
Can you write Python code for it?

# What is 'a'? Can we find another 'a'?
a = -3*(1.7**2)
print(a)

f = lambda x: x**3-7
g =  lambda x: f(x)/a+x
r=fpi(g, 1.7, 14, True)
print(f(r))


I am working on the file "SC1/04_roots_of_1D_equations.ipynb".
The first task is:
Finding $r$ using sympy
Can you write Python code for it?

y = sym.Symbol('y')
fsym = lambda y: y+sym.log(y)-3
r_all=sym.solve(sym.Eq(fsym(y), 0), y)
r=r_all[0].evalf()
print(r)
print(r_all)

def find_root_manually(r=2.0):
    x = np.linspace(1,3,1000)
    plt.figure(figsize=(8,8))
    plt.plot(x,f(x),'b-')
    plt.grid()
    plt.ylabel('$f(x)$',fontsize=16)
    plt.xlabel('$x$',fontsize=16)
    plt.title('What is r such that $f(r)='+str(f(r))+'$? $r='+str(r)+'$',fontsize=16)
    plt.plot(r,f(r),'k.',markersize=20)
    plt.show()
interact(find_root_manually,r=(1e-5,3,1e-3))


I am working on the file "SC1/04_roots_of_1D_equations.ipynb".
The first task is:
In class
From the textbook
Can you write Python code for it?


I am working on the file "SC1/04_roots_of_1D_equations.ipynb".
The first task is:
In-Class 20200423
Can you write Python code for it?

g1 = lambda x: np.cos(x)
g2 = lambda x: np.cos(x)**2
g3 = lambda x: np.sin(x)
g4 = lambda x: 1-5*x+(15/2)*x**2-(5/2)*x**3 # (0,1) y (1,2)

interact(lambda x0, n: fpi2(g4, x0, n, True),x0=widgets.FloatSlider(min=0, max=3, step=0.01, value=0), n=widgets.IntSlider(min=10, max=100, step=1, value=10))


I am working on the file "SC1/04_roots_of_1D_equations.ipynb".
The first task is:
It's very important to define a concept called convergence rate.
This rate shows how fast the convergence of a method is at a specified point.
The convergence rate for the bisection is always 0.5 because this method uses the half of the interval for each iteration.
<div id='cobweb' />

Cobweb Plot
Can you write Python code for it?

def cobweb(x,g=None):
    min_x = np.amin(x)
    max_x = np.amax(x)

    plt.figure(figsize=(10,10))
    ax = plt.axes()
    plt.plot(np.array([min_x,max_x]),np.array([min_x,max_x]),'b-')
    for i in np.arange(x.size-1):
        delta_x = x[i+1]-x[i]
        head_length =  np.abs(delta_x)*0.04
        arrow_length = delta_x-np.sign(delta_x)*head_length
        ax.arrow(x[i], x[i], 0, arrow_length, head_width=1.5*head_length, head_length=head_length, fc='k', ec='k')
        ax.arrow(x[i], x[i+1], arrow_length, 0, head_width=1.5*head_length, head_length=head_length, fc='k', ec='k')

    if g!=None:
        y = np.linspace(min_x,max_x,1000)
        plt.plot(y,g(y),'r')

    plt.title('Cobweb diagram')
    plt.grid(True)
    plt.show()


I am working on the file "SC1/04_roots_of_1D_equations.ipynb".
The first task is:
Let's quickly explain the Cobweb Diagram we have here. The blue line is the function $x$ and the red is the function $g(x)$. The point in which they meet is $g$'s fixed point. In this particular example, we start at $y = x = 1.5$ (the top right corner) and then we "jump" vertically to $y = \cos(1.5) \approx 0.07$. After this, we jump horizontally to $x = \cos(1.5) \approx 0.07$. Then, we jump again vertically to $y = \cos\left(\cos(1.5)\right) \approx 0.997$ and so on. See the pattern here? We're just iterating over $x = \cos(x)$, getting closer to the center of the diagram where the fixed point resides, in $x \approx 0.739$.
It's very important to mention that the algorithm will converge only if the rate of convergence $S < 1$, where $S = \left| g'(r) \right|$. If you want to use this method, you'll have to construct $g(x)$ starting from $f(x)$ accordingly. In this example, $g(x) = \cos(x) \Rightarrow g'(x) = -\sin(x)$ and $|-\sin(0.739)| \approx 0.67$.
Another example. Source: https://divisbyzero.com/2008/12/18/sharkovskys-theorem/amp/?__twitter_impression=true
Can you write Python code for it?


I am working on the file "SC1/04_roots_of_1D_equations.ipynb".
The first task is:
Playing with some roots
Can you write Python code for it?

f = lambda x: 8*x**4-12*x**3+6*x**2-x
fp = lambda x: 32*x**3-36*x**2+12*x-1

x = np.linspace(-1,1,1000)
plt.figure(figsize=(10,10))

plt.title('What are we seeing with the semiloigy plot? Is this function differentiable?')
plt.semilogy(x,np.abs(f(x)),'b-')
plt.semilogy(x,np.abs(fp(x)),'r-')
plt.grid()
plt.ylabel('$f(x)$',fontsize=16)
plt.xlabel('$x$',fontsize=16)
plt.show()

r=newton_method(f, fp, 0.3, rel_error=1e-8, m=1)
print([r,f(r)])
# Is this showing quadratic convergence? If not, can you fix it?


I am working on the file "SC1/04_roots_of_1D_equations.ipynb".
The first task is:
Solutions
<div id='sol1' />
Problem: Build a FPI such that given $x$ computes $\displaystyle \frac{1}{x}$
Can you write Python code for it?

# We are finding the 1/a
# Solution code:
a = 2.1
g = lambda x: 2*x-a*x**2
gp = lambda x: 2-2*a*x
r=fpi2(g, 0.7, 7, flag_cobweb=True)
print([r,1/a])
# Are we seeing quadratic convergence?


I am working on the file "SC1/04_roots_of_1D_equations.ipynb".
The first task is:
Which function shows quadratic convergence? Why?
Can you write Python code for it?


I am working on the file "SC1/05_linear_systems_of_equations.ipynb".
The first task is:
<div id='acknowledgements' />

Acknowledgements

Material created by professor Claudio Torres (ctorres@inf.utfsm.cl) and assistants: Laura Bermeo, Alvaro Salinas, Axel Simonsen and Martín Villanueva. DI UTFSM. April 2016.
Update May 2020 - v1.11 - C.Torres : Fixing formatting issues.
Can you write Python code for it?


I am working on the file "SC1/05_linear_systems_of_equations.ipynb".
The first task is:
<div id='ex' />

Exercises
Now that you know how to solve systems of linear equations problem with these methods, let's try to answer a few questions!
$a)$ Find the values of $\alpha$ that make possible to do a LU descomposition of the following matrix:
$$ \begin{bmatrix}
   \alpha & 2  \[0.3em]
   1 & \alpha  \end{bmatrix} $$
$b)$- Let $A$ be the following matrix:
$$ A = \begin{bmatrix}
        2 &  4 &  2  \[0.3em]
       -1 &  1 &  2  \[0.3em]
       -1 & -3 & -1  \end{bmatrix} $$

Find the PALU descomposition of the matrix $A$.

Solve the system of equations $Ax = [1 , \frac{1}{2}, \frac{1}{3}]^T$.

$c)$ Considering this matrix:
$$  \begin{bmatrix}
    1 & 1 & 0  \[0.3em]
    1 & 5 & 2  \[0.3em]
    0 & 2 & 3  \end{bmatrix} $$

Find the LU descomposition.

Find the Cholesky descomposition.

Compare the efficiency of both methods.

$d)$ Use Jacobi, Gauss Seidel, and SOR to solve the following system of equations (number of iterations = 2):
$$2x + y = 3$$
$$x + 2y = 2$$

Which is the best method to solve this problem (with better results)?

$e)$ Explain the pros and cons of using iterative methods instead of the direct ones.
Extra: Hilbert Matrix
Can you write Python code for it?

from scipy.linalg import hilbert

N=20
errors=np.zeros(N+1)
kappas=np.zeros(N+1)
my_range=np.arange(5,N+1)
for n in my_range:
    A=hilbert(n)
    x_exact=np.ones(n)
    b=np.dot(A,x_exact)
    x=np.linalg.solve(A,b)
    errors[n]=np.linalg.norm(x-x_exact,ord=np.inf)
    kappas[n]=np.linalg.cond(A,2)

plt.figure(figsize=(12,6))
plt.semilogy(my_range, 10.**(-16+np.log10(kappas[my_range])), marker='o', linestyle='--', color='b',label='Estimated forward error')
plt.semilogy(my_range, errors[my_range], marker='o', linestyle='--', color='r',label='Forward error')
plt.grid(True)
plt.xlabel('n')
#plt.ylabel('$\kappa$ and errors')
plt.title('')
plt.legend(loc=0)
plt.show()


I am working on the file "SC1/05_linear_systems_of_equations.ipynb".
The first task is:
<div id='im' />

Iterative Methods
Can you write Python code for it?

"""
Randomly generates an nxn strictly diagonally
dominant matrix A.
"""
def generate_dd_matrix(n):
    A = np.random.random((n,n))
    deltas = 0.1*np.random.random(n)
    row_sum = A.sum(axis=1)-np.diag(A)
    np.fill_diagonal(A, row_sum+deltas)
    return A

"""
Computes relative error between each row on
X matrix and y vector.
"""
def error(X, y):
    D = X-y
    err = np.linalg.norm(D, axis=1, ord=np.inf)
    return err


I am working on the file "SC1/05_linear_systems_of_equations.ipynb".
The first task is:
As before we will create a linear system $A x = b$, with $A$ as a diagonally dominant matrix, which is a sufficient condition for the methods we will study in this section converge
Can you write Python code for it?


I am working on the file "SC1/05_linear_systems_of_equations.ipynb".
The first task is:
Gauss Seidel
Can you write Python code for it?

def gauss_seidel(A, b, n_iter=50):
    n = A.shape[0]
    #array with solutions
    X = np.empty((n_iter, n))
    #initial guess
    X[0] = np.zeros(n)
    #submatrices
    R = np.tril(A) #R=(L+D)
    U = A-R
    for i in range(1, n_iter):
        #X[i] = solve_triangular(R, b-np.dot(U, X[i-1]), upper=False)
        # v1.11
        X[i] = X[i-1]+solve_triangular(R, b-np.dot(A, X[i-1]), upper=False)
    return X
def gauss_seidel_M(A):
    L = np.tril(A,-1)
    U = np.triu(A,1)
    D = np.diag(np.diag(A))
    M = -np.dot(np.linalg.inv(L+D),U)
    return M


I am working on the file "SC1/05_linear_systems_of_equations.ipynb".
The first task is:
Given a symmetric positive-definite matrix $A \in \mathbb{R}^{n \times n}$, the Cholesky decomposition is of the form $A =R^T R$, with $R$ being an upper triangular matrix. This method takes advantage of the properties of symmetric matrices, reaching approximately twice the efficiency of LU.
Can you write Python code for it?

def cholesky_decomp(A, show=False):
    N,_ = A.shape
    A = np.copy(A)
    R = np.zeros((N,N))
    if show:
        print('Initial matrix')
        print('A = '); print(np.array_str(A, precision=2, suppress_small=True))
        print('R = '); print(np.array_str(R, precision=2, suppress_small=True))
        print('----------------------------------------')
    for i in range(N):
        R[i,i] = np.sqrt(A[i,i])
        u = (1./R[i,i])*A[i,i+1:]
        R[i,i+1:] = u
        A[i+1:,i+1:] -= np.outer(u,u)
        if show:
            print('A = '); print(np.array_str(A, precision=2, suppress_small=True))
            print('R = '); print(np.array_str(R, precision=2, suppress_small=True))
            print('----------------------------------------')
    return R


I am working on the file "SC1/05_linear_systems_of_equations.ipynb".
The first task is:
Here are some questions about SOR:
- Why can averaging the current solution with the Gauss-Seidel solution improve convergence?
- Why do we use $\omega > 1$ and not $\omega < 1$?
- Could you describe a method to find the best value of $\omega$ (the one which optimizes convergence)?
- Would it be a better option to re-compute $\omega$ at each iteration?
<div id='ca' />

Convergence Analysis
Let's see convergence plots all together
Can you write Python code for it?

plt.figure(figsize=(12,6))
plt.semilogy(it, jac_err, marker='o', linestyle='--', color='b', label='Jacobi')
plt.semilogy(it, gauss_err, marker='o', linestyle='--', color='r', label='Gauss-Seidel')
plt.semilogy(it, sor_err, marker='o', linestyle='--', color='g', label='SOR')
plt.grid(True)
plt.xlabel('Iterations')
plt.ylabel('Error')
plt.title('Infinity norm error for all methods')
plt.legend(loc=0)
plt.show()


I am working on the file "SC1/05_linear_systems_of_equations.ipynb".
The first task is:
How can we choose a good value of $\omega$? Well there are  some methods you could search, but for now we will try a naive way, i.e, computing the solution for a range $\omega \in [1,1.3]$ as follows:
Can you write Python code for it?

n = 30 #width of subdivisions
sor_solutions = list()
for w in np.linspace(1., 1.3, n):
    sor_solutions.append(sor(A, b, w, n_iter=5)[-1])
np.asarray(sor_solutions)

#now compute error solutions with each w
sor_errors = error(sor_solutions, np_sol)
w = np.linspace(1., 1.3, n)


I am working on the file "SC1/05_linear_systems_of_equations.ipynb".
The first task is:
Jacobi
Can you write Python code for it?

"""
Iterative methods implementations returns an array X
with the the solutions at each iteration
"""
def jacobi(A, b, n_iter=50):
    n = A.shape[0]
    #array with solutions
    X = np.empty((n_iter, n))
    #initial guess
    X[0] = np.zeros(n)
    #submatrices
    D = np.diag(A)
    Dinv = D**-1
    R = A - np.diag(D) # R = (L+U)
    for i in range(1, n_iter):
        # X[i] = Dinv*(b - np.dot(R, X[i-1]))
        # v1.12
        ri = b - np.dot(A, X[i-1])
        X[i] = X[i-1]+Dinv*ri
    return X
def jacobi_M(A):
    L = np.tril(A,-1)
    U = np.triu(A,1)
    D = np.diag(np.diag(A))
    M = -np.dot(np.linalg.inv(D),L+U)
    return M

# \mathbf{x}_{n+1}=M\,\mathbf{x}_{n}+\mathbf{c}


I am working on the file "SC1/05_linear_systems_of_equations.ipynb".
The first task is:
Let's test this new method against the LU and NumPy solvers
Can you write Python code for it?

palu_sol = solve_palu(A, b, show=True)

np.linalg.norm(palu_sol - lu_sol)

np.linalg.norm(palu_sol - np_sol)


I am working on the file "SC1/05_linear_systems_of_equations.ipynb".
The first task is:
Now let's resolve the same linear system with Jacobi method!
Can you write Python code for it?

jac_sol = jacobi(A,b, n_iter=50)

jac_err = error(jac_sol, np_sol)
it = np.linspace(1, 50, 50)

plt.figure(figsize=(12,6))
plt.semilogy(it, jac_err, marker='o', linestyle='--', color='b')
plt.grid(True)
plt.xlabel('Iterations')
plt.ylabel('Error')
plt.title('Infinity norm error for Jacobi method')
plt.show()

Mj = jacobi_M(A)
print(np.linalg.norm(Mj))

np.linalg.eigvals(Mj)

np.abs(np.linalg.eigvals(Mj))

np.max(np.abs(np.linalg.eigvals(Mj)))


I am working on the file "SC1/05_linear_systems_of_equations.ipynb".
The first task is:
Now we test our implementation, comparing time execution with LU and PALU on two different linear systems
Can you write Python code for it?

A = generate_spd_matrix(3)
b = np.ones(3)

b=np.array([4,2,0])
solve_cholesky(A, b, show=True)

A = generate_spd_matrix(100)
b = np.ones(100)

%timeit solve_cholesky(A, b)
%timeit solve_lu(A, b)
%timeit solve_palu(A, b)

A = generate_spd_matrix(1000)
b = np.ones(1000)

%timeit solve_cholesky(A, b)
%timeit solve_lu(A, b)
%timeit solve_palu(A, b)


I am working on the file "SC1/05_linear_systems_of_equations.ipynb".
The first task is:
SOR
Can you write Python code for it?

def sor(A, b, w=1.05, n_iter=50):
    n = A.shape[0]
    #array with solutions
    X = np.empty((n_iter, n))
    #initial guess
    X[0] = np.zeros(n)
    #submatrices
    R = np.tril(A) #R=(L+D)
    U = A-R
    # v1.11
    L = np.tril(A,-1)
    D = np.diag(np.diag(A))
    M = L+D/w
    for i in range(1, n_iter):
        #X_i = solve_triangular(R, b-np.dot(U, X[i-1]), upper=False)
        #X[i] = w*X_i + (1-w)*X[i-1]
        # v1.11
        X[i] = X[i-1]+solve_triangular(M, b-np.dot(A, X[i-1]), upper=False)
    return X
def sor_M(A,w=1.05):
    L = np.tril(A,-1)
    U = np.triu(A,1)
    D = np.diag(np.diag(A))
    M = np.dot(np.linalg.inv(w*L + D),((1-w)*D -w*U))
    return M


I am working on the file "SC1/05_linear_systems_of_equations.ipynb".
The first task is:
and find the solution $x$ through np.linalg.solve to use it as the reference solution-
Can you write Python code for it?


I am working on the file "SC1/05_linear_systems_of_equations.ipynb".
The first task is:
in order to compare these huge vectors, we use the Euclidean metric as follows:
Can you write Python code for it?


I am working on the file "SC1/06_conjugate_gradient_method.ipynb".
The first task is:
The science behind this algorithm is in the classnotes and in the textbook (Numerical Analysis, 2nd Edition, Timothy Sauer). Now let's try it!
Here are some questions to think about:
* What are the advantages and disadvantages of each method: gradient_descent and conjugate_gradient?
* In which cases can the Conjugate Gradient Method converge in less than $n$ iterations?
* What will happen if you use the Gradient Descent or Conjugate Gradient Method with non-symmetric, non-positive-definite matrices?
<div id='LP' />

Let's Play: Practical Exercises and Profiling
First of all, define a function to calculate the progress of the relative error for a given method, that is, input the array of approximate solutions X and the real solution provided by Numpy's solver r_sol and return an array with the relative error for each step.
Can you write Python code for it?

def relative_error(X, r_sol):
    n_steps = X.shape[0]
    n_r_sol = np.linalg.norm(r_sol)
    E = np.zeros(n_steps)
    for i in range(n_steps):
        E[i] = np.linalg.norm(X[i] - r_sol) / n_r_sol
    return E


I am working on the file "SC1/06_conjugate_gradient_method.ipynb".
The first task is:
Trying the two methods with a small non-symmetric, non-positive-definite matrix and plotting the forward error for all the methods.
Can you write Python code for it?

def show_output_for_non_symmetric_and_npd(np_seed=0):
    np.random.seed(np_seed)
    n = 10
    A = 10 * np.random.random((n,n))
    b = 10 * np.random.random(n)
    x0 = np.zeros(n)

    X1 = gradient_descent(A, b, x0, n)
    X2, D, R, alphas, betas, n_residuals = conjugate_gradient(A, b, x0, True)
    r_sol = np.linalg.solve(A, b)

    E1 = relative_error(X1, r_sol)
    E2 = relative_error(X2, r_sol)
    iterations1 = np.linspace(1, n, n)
    iterations2 = np.linspace(1, X2.shape[0], X2.shape[0])

    plt.figure(figsize=(10,5))
    plt.xlabel('Iteration')
    plt.ylabel('Relative Error')
    plt.title('Evolution of the Relative Forward Error for each method')
    plt.semilogy(iterations1, E1, 'rd', markersize=8, label='GD') # Red diamonds are for Gradient Descent
    plt.semilogy(iterations2, E2, 'b.', markersize=8, label='CG') # Blue dots are for Conjugate Gradient
    plt.grid(True)
    plt.legend(loc='best')
    plt.show()
interact(show_output_for_non_symmetric_and_npd,np_seed=(0,100,1))


I am working on the file "SC1/06_conjugate_gradient_method.ipynb".
The first task is:
We'll try our algorithm with some matrices of different sizes, and we'll compare it with the solution given by Numpy's solver.
Can you write Python code for it?

def show_small_example_GD(n_size=3, n_iter=10):
    np.random.seed(0)
    A = generate_spd_matrix(n_size)
    b = np.ones(n_size)
    x0 = np.zeros(n_size)

    X = gradient_descent(A, b, x0, n_iter)
    sol = np.linalg.solve(A, b)
    print('Gradiente descent : ',X[-1])
    print('np solver         : ',sol)
    print('norm(difference): \t',np.linalg.norm(X[-1] - sol)) # difference between gradient_descent's solution and Numpy's solver solution
interact(show_small_example_GD,n_size=(3,50,1),n_iter=(5,50,1))


I am working on the file "SC1/07_Polynomial_Interpolation_1D.ipynb".
The first task is:
<div id='cheby' />

Chebyshev Interpolation
With the objective of reducing the error of the polynomial interpolation, we need to find the values of $x_1,x_2,...,x_n$ that minimize $(x-x_1) \cdot (x-x_2) \cdot ... \cdot (x-x_n)$.
To choose these values of $-1 \leq x_1,x_2,...,x_n \leq 1$ (to use another interval we just need to do a change of variables) that minimize the error, we will use the roots of the Chebyshev polynomials, also called Chebyshev nodes (of the first kind), which are defined by:
$$ x_i = \cos\left(\frac{(2i-1)\pi}{2n}\right), i = 1,...,n $$
Can you write Python code for it?

def Chebyshev(xmin,xmax,n=5):
    # This function calculates the n Chebyshev points and plots or returns them depending on ax
    ns = np.arange(1,n+1)
    x = np.cos((2*ns-1)*np.pi/(2*n))
    y = np.sin((2*ns-1)*np.pi/(2*n))
    plt.figure(figsize=(10,5))
    plt.ylim(-0.1,1.1)
    plt.xlim(-1.1,1.1)
    plt.plot(np.cos(np.linspace(0,np.pi)),np.sin(np.linspace(0,np.pi)),'k-')
    plt.plot([-2,2],[0,0],'k-')
    plt.plot([0,0],[-1,2],'k-')
    for i in range(len(y)):
        plt.plot([x[i],x[i]],[0,y[i]],'r-')
        plt.plot([0,x[i]],[0,y[i]],'r-')
    plt.plot(x,[0]*len(x),'bo',label='Chebyshev points')
    plt.plot(x,y,'ro')
    plt.xlabel('$x$')
    plt.title('n = '+str(n))
    plt.grid(True)
    plt.legend(loc='best')
    plt.show()
def Chebyshev_points(xmin,xmax,n):
    ns = np.arange(1,n+1)
    x = np.cos((2*ns-1)*np.pi/(2*n))
    #y = np.sin((2*ns-1)*np.pi/(2*n))
    return (xmin+xmax)/2 + (xmax-xmin)*x/2
def Chebyshev_points_histogram(n=50,nbins=20):
    xCheb=Chebyshev_points(-1,1,n)
    plt.figure()
    plt.hist(xCheb,bins=nbins,density=True)
    plt.grid(True)
    plt.show()

interact(Chebyshev,xmin=fixed(-1),xmax=fixed(1),n=(2,50))

interact(Chebyshev_points_histogram,n=(20,10000),nbins=(20,200))


I am working on the file "SC1/07_Polynomial_Interpolation_1D.ipynb".
The first task is:
<div id='runge' />

Runge's Phenomenon: It is a problem of oscillation of polynomials at the edges of the interval.
We are interpolating a data that is 0 almost everywhere and 1 at the middle point, notice that when $n$ increases the oscilations increase and the red dots seems to be at 0 everywhere but it is just an artifact, there must be a 1 at the middle. The oscillations you see at the end of the interval is the Runge phenomenon.
Can you write Python code for it?

def Runge(n=9):
    x = np.linspace(0,1,n)
    y = np.zeros(n)
    y[int((n-1.0)/2.)]=1
    D = Newton(x,y,False)
    Interpolation_Plot(D)
interact(Runge,n=(5,25,2))


I am working on the file "SC1/07_Polynomial_Interpolation_1D.ipynb".
The first task is:
By using these points, we reduce the numerator of the interpolation error formula:
$$ (x-x_1) \cdot (x-x_2) \cdot ... \cdot (x-x_n) = \dfrac{1}{2^{n-1}} \cdot T_n(x), $$
where $T(x) = \cos (n \cdot \arccos (x))$ is the n-th Chebyshev polynomial.
$$ T_0(x) = 1 $$
$$ T_1(x) = x $$
$$ T_2(x) = 2x^2 -1 $$
$$...$$
$$ T_{n+1}(x) = 2 \cdot x \cdot T_n(x) - T_{n-1}(x) $$
Can you write Python code for it?

def T(n,x):
    # Recursive function that returns the n-th Chebyshev polynomial evaluated at x
    if n == 0:
        return x**0
    elif n == 1:
        return x
    else:
        return 2*x*T(n-1,x)-T(n-2,x)

def Chebyshev_Polynomials(n=2, Flag_All_Tn=False):
    # This function plots the first n Chebyshev polynomials
    x = np.linspace(-1,1,1000)
    plt.figure(figsize=(10,5))
    plt.xlim(-1, 1)
    plt.ylim(-1.1, 1.1)
    if Flag_All_Tn:
        for i in np.arange(n+1):
            y = T(i,x)
            plt.plot(x,y,label='$T_{'+str(i)+'}(x)$')
    else:
        y = T(n,x)
        plt.plot(x,y,label='$T_{'+str(n)+'}(x)$')
    #    plt.title('$T_${:}$(x)$'.format(n))
    plt.legend(loc='right')
    plt.grid(True)
    plt.xlabel('$x$')
    plt.show()

interact(Chebyshev_Polynomials,n=(0,12),Flag_All_Tn=True)

n=9
xmin=1
xmax=9

mee = Error(lambda x: np.sin(x)**3, n, xmin, xmax, method=Lagrange)
mec = Error(lambda x: np.sin(x)**3, n, xmin, xmax, method=Lagrange, points=Chebyshev_points)

print("Max error (equidistants points):", mee)
print("Max error (Chebyshev nodes):", mec)

def test_error_chebyshev(n=5):
    mee = Error(lambda x: (1/(1+12*x**2)), n, -1, 1, Lagrange)
    mec = Error(lambda x: (1/(1+12*x**2)), n, -1, 1, method=Lagrange, points=Chebyshev_points)
    print("Max error (equidistants points):", mee)
    print("Max error (Chebyshev nodes):", mec)

interact(test_error_chebyshev,n=(5,100,2))


I am working on the file "SC1/07_Polynomial_Interpolation_1D.ipynb".
The first task is:
Here you get some questions about Lagrange Interpolation:
- Explain what happens with the interpolator polynomial when you add a new point to the set of points to interpolate. Answer: We need to recalculate the polynomial
- Why it is not a good idea to use Lagrange interpolation for a set of points which is constantly changing? A: Because we need to compute the whole interpolation again
- What is the operation count of obtaining the interpolator polynomial using Lagrange? What happens with the error?
<div id='DDN' />

Newton's Divided Difference
In this interpolation method we will use divided differences to calculate the coefficients of our interpolation polynomial. Given a set of $n$ data points $(x_1,y_1),...,(x_n,y_n)$, the Newton polynomial is:
$$ P(x) = \sum^n_{i=1} (f[x_1 ... x_i] \cdot \prod^{i-1}_{j=1} (x-x_j)) ,$$
where $ \prod^{0}_{j=1} (x-x_j) = 0 $, and:
$$ f[x_i] = y_i $$
$$ f[x_j...x_i] = \frac{f[x_{j+1}...x_i]-f[x_j...x_{i-1}]}{x_i-x_j}$$
Can you write Python code for it?

def Divided_Differences(x, y):
    dd = np.array([y])
    for i in range(len(x)-1):
        ddi = []
        for a in range(len(x)-i-1):
            ddi.append((dd[i][a+1]-dd[i][a])/(x[a+i+1]-x[a]))
        ddi = np.append(ddi,np.full((len(x)-len(ddi),),0.0))
        dd = np.append(dd,[ddi],axis=0)
    return np.array(dd)

def Newton(x, y, show=False):
    # We calculate the divided differences and store them in a data structure
    dd = Divided_Differences(x,y)

    # The function shows the data if the flag is true
    if show:
        print('Data Points: '); print([(x[i],y[i]) for i in range(len(x))])
        xS = sp.Symbol('x')
        N = np.dot(dd[:,0],np.append([1],[np.prod(xS-x[:i]) for i in range(1,len(dd))]))
        print('Interpolation Function: ');
        print(N)

    # Finally, we return a data structure with our interpolating polynomial
    D = {'M':'Newton',
         'P':dd[:,0],
         'x':x,
         'y':y}
    return D

def show_time_N(epsilon=0):
    x = np.array([0.0,2.0,3.0+epsilon,4.0,5.0,6.0])
    y = np.array([1.0,3.0,0.0,6.0,8.0,4.0])
    D = Newton(x,y,True)
    Interpolation_Plot(D)
interact(show_time_N,epsilon=(-1,1,0.1))


I am working on the file "SC1/07_Polynomial_Interpolation_1D.ipynb".
The first task is:
Questions about Newton's DD:
- What is the main problem using this method (and Lagrange)? How can you fix it? A: A problem with polynomial interpolation with equispaced date is the Runge phenomenon and can be handle with Chebyshev points
- What to do when a new point is added? A: Pro, is not necessary re-calculate the whole polynomial only a small piece
<div id='Error' />

Polynomial Interpolation Error
The interpolation error is given by:
$$ f(x)-P(x) = \frac{(x-x_1) \cdot (x-x_2) \cdot ... \cdot (x-x_n)}{n!} \cdot f^{(n)}(c) ,$$
where $c$ is within the interval from the minimun value of $x$ and the maximum one.
Can you write Python code for it?

def Error(f, n, xmin, xmax, method=Lagrange, points=np.linspace, plot_flag=True):
    # This function plots f(x), the interpolating polynomial, and the associated error
    # points can be np.linspace to equidistant points or Chebyshev to get Chebyshev points
    x = points(xmin,xmax,n)
    y = f(x)

    xe = np.linspace(xmin,xmax,100)
    ye = f(xe)

    D = method(x,y)
    yi = Y(D, xe)

    if plot_flag:
        plt.figure(figsize=(5,10))
        f, (ax1, ax2) = plt.subplots(1, 2, figsize=(15,5), sharey = False)

        ax1.plot(xe, ye,'r-', label='f(x)')
        ax1.plot(x, y,'ro', label='Interpolation points')
        ax1.plot(xe, yi,'b-', label='Interpolation')
        ax1.set_xlim(xmin-0.5,xmax+0.5)
        ax1.set_ylim(min(yi)-0.5,max(yi)+0.5)
        ax1.set_title('Interpolation')
        ax1.grid(True)
        ax1.set_xlabel('$x$')
        ax1.legend(loc='best')

        ax2.semilogy(xe, abs(ye-yi),'b-', label='Absolute Error')
        ax2.set_xlim(xmin-0.5,xmax+0.5)
        ax2.set_title('Absolute Error')
        ax2.set_xlabel('$x$')
        ax2.grid(True)
        #ax2.legend(loc='best')
        plt.show()

    return max(abs(ye-yi))

def test_error_Newton(n=5):
    #me = Error(lambda x: np.sin(x)**3, n, 1, 7, Newton)
    me = Error(lambda x: (1/(1+12*x**2)), n, -1, 1, Newton)
    print("Max Error:", me)

interact(test_error_Newton,n=(5,25))


I am working on the file "SC1/08_Interpolation_Splines.ipynb".
The first task is:

Introduction
Previously in our jupyter notebooks, we learn about interpolation. Methods like Newton's Divided Difference, Lagrange, among others. Other alternative for interpolate a set of data points is using  Cubic Splines.
This technique, avoids the Runge's Phenomenon and creates a 3-degree polynomial easily.

Splines
The most common spline is the linear spline. Given a set of points $(x_{1},y_{1}), (x_{2},y_{2}),...,(x_{n},y_{n}) $, this spline connects each point creating a non-smooth curve. However, this polynomial haves a problem. It's no smooth curve! For to avoid this problem, the cubic splines creates a set of 3-degree polynomial (specifically n-1 polynomials)... a much better curve.
Can you write Python code for it?

# Code based on Example from: https://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.CubicSpline.html#scipy.interpolate.CubicSpline

# The data
x = np.linspace(0,2*np.pi,12)
y = np.sin(x)*x

# Building interpolation object
cs = CubicSpline(x, y)

# Defining a finer mesh to plot the function
xx = np.linspace(0,2*np.pi,1000)
yyo = np.sin(xx)*xx
yyo1 = np.cos(xx)*xx+np.sin(xx)
yyo2 = -np.sin(xx)*xx+2*np.cos(xx)
yyo3 = -np.cos(xx)*xx-3*np.sin(xx)
yyo4 = np.sin(xx)*xx-4*np.cos(xx)

#Interpolating the date with the spline
yy = cs(xx)
yy1 = cs(xx, 1)
yy2 = cs(xx, 2)
yy3 = cs(xx, 3)
yy4 = cs(xx, 4)

# Plotting the splines and its derivatives
plt.figure(figsize=(M,M))
plt.plot(x,y,'k.',markersize=20,label=r'Data Points')
plt.plot(xx,yy, linewidth=4, label=r'S$(x)$')
plt.plot(xx,yy1, linewidth=4, label=r'$\frac{d}{dx}$S$(x)$')
plt.plot(xx,yy2, linewidth=4, label=r'$\frac{d^2}{dx^2}$S$(x)$')
plt.plot(xx,yy3, linewidth=4, label=r'$\frac{d^3}{dx^3}$S$(x)$')
plt.plot(xx,yy4, linewidth=4, label=r'$\frac{d^4}{dx^4}$S$(x)$')

plt.plot(xx,yyo4,'k--',linewidth=4, label='test', alpha=0.4)

plt.plot(x,y,'k.',markersize=20)

plt.title(r'Cubic Spline is defined as S$(x)$')
plt.axis('tight')
plt.xlabel(r'$x$')
plt.ylabel(r'$y$')
plt.grid(True)
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))
plt.show()

plt.figure()
plt.semilogy(np.abs(yy-yyo))
plt.grid(True)


I am working on the file "SC1/08_Interpolation_Splines.ipynb".
The first task is:
Example 1 - Hand made interpolation
Can you write Python code for it?


I am working on the file "SC1/08_Interpolation_Splines.ipynb".
The first task is:
Example 2
Can you write Python code for it?

def show_spline(type_ending='Natural',k1=0, k2=0, p1=0, p2=0):
    x = np.array([1,2,4,5,7,9])
    y = np.array([2,1,4,3,3,4])
    xx=np.linspace(np.min(x),np.max(x),1000)
    S = cubic_spline(x,y,end=type_ending, k1=k1, k2=k2, p1=p1, p2=p2)
    plt.figure(figsize=(M,M))
    plt.plot(xx,cubic_spline_eval(xx,S),'-',linewidth=5,label=r'S$(x)$')
    plt.plot(x,y,'k.',markersize=20,label='Data Points')
    plt.xlabel('$x$')
    plt.ylabel('$y$')
    plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))
    plt.grid(True)
    plt.show()
widgets.interact(show_spline, type_ending=['Natural','Adjusted','Clamped','Parabolically','NaK'],
                k1=(-20,20,1),k2=(-20,20,1),p1=(-2,2,0.2),p2=(-2,2,0.2))


I am working on the file "SC1/08_Interpolation_Splines.ipynb".
The first task is:
The orange curve is generated with cubic splines (using the scipy implementation). The other colors are the derivatives of the Cubic Spline as indicated in the legend.
However, if we thinks about this curve, we can say that exists an infinitely many quantity of polynomials such that meets all the points. Our goal is to create an unique polynomial. Given this condition, there is 4 properties that defines the cubic spline we are looking for.

Properties of Splines
When we want creates a spline of n data points, we obtains a set of n-1 3-degree polynomials. For example:
Given a set of points $(x_{1},y_{1}), (x_{2},y_{2}),...,(x_{n},y_{n})$, the splines is:
\begin{equation} S_{1}(x) = y_{1} + b_{1}(x-x_{1}) + c_{1}(x-x_{1})^{2} + d_{1}(x-x_{1})^{3} \
                 S_{2}(x) = y_{2} + b_{2}(x-x_{2}) + c_{2}(x-x_{2})^{2} + d_{2}(x-x_{2})^{3} \
                 ... \
                 ... \
                 ... \
                 S_{n-1}(x) = y_{n-1} + b_{n-1}(x-x_{n-1}) + c_{n-1}(x-x_{n-1})^{2} + d_{n-1}(x-x_{n-1})^{3}
\end{equation}
Thus, our goal is obtains the $y, b, c$ and $d$ coefficients. With this values, we are creating the spline $S(x)$ that meets all the data points. This spline have the next properties:
Property 1 (Are the points connected?)
The first property checks if each x-coordinate reachs the correspondant y-coordinate in the spline $S(x)$. Thus, checks if the spline meets the data points.
$$S_{i}(x_{i}) = y_{i}$$ $$ S_{i}(x_{i+1}) = y_{i+1}$$
$$i \in [1,n-1]$$
Property 2 (Slope Continuity)
The second property ensures that the slopes of the splines at the points' neighborhood to be equals. This guarantees the smoothly of $S(x)$
$$S'{i-1}(x{i}) = S'{i}(x{i})$$
$$i \in [2,n-1]$$
Property 3 (Curvature Continuity)
This property ensure that the curvature between differents polynomials will be equal at the neighborhood of points, avoiding abrupt changes of the curve at the data points.
$$S''{i-1}(x{i}) = S''{i}(x{i})$$
$$i \in [2,n-1]$$

Solving the system
If we have n...
Can you write Python code for it?

def cubic_spline(x, y, end=None, k1=0, k2=0, p1=0, p2=0):
    #x: x-coordinates of points
    #y: y-coordinates of points
    #end: Natural, Adjusted, Clamped, Parabolically, NaK

    n = len(x)
    A = np.zeros((3*n-3, 3*n-3))
    b = np.zeros(3*n-3)

    delta_x=np.diff(x)

    #Building the linear system of equations

    #1st property
    for i in np.arange(n-1):
        b[i]= y[i+1]-y[i]
        A[i,3*i:3*(i+1)] = [delta_x[i],delta_x[i]**2,delta_x[i]**3]
    #2nd property
    for i in np.arange(n-2):
        A[(n-1)+i,3*i:3*(i+1)+1]=[1, 2*delta_x[i], 3*delta_x[i]**2, -1]
    #3rd property
    for i in np.arange(n-2):
        A[(n-1)+(n-2)+i,3*i:3*(i+1)+2] = [0, 2, 6*delta_x[i], 0, -2]

    #Ending conditions (4th property)
    if end =='Natural':
        A[-2,1]= 2
        A[-1,-2] = 2
        A[-1,-1] = 6*delta_x[-1]

    elif end == 'Adjusted':
        A[-2,1]= 2
        A[-1,-2] = 2
        A[-1,-1] = 6*delta_x[-1]
        b[-2:] = [k1,k2]
        print('Adjusted',b[-2:])

    elif end == 'Clamped':
        A[-2,0]=1
        A[-1,-3:] = [1,2*delta_x[-1],3*delta_x[-1]**2]
        b[-2:] = [p1,p2]

    elif end == 'Parabolically':
        A[-2,2]=1
        A[-1,-1]=1

    elif end == 'NaK':
        A[-2,2:6]=[6,0,0,-6]
        A[-1,-4:]=[6,0,0,-6]

    #Solving the system
    sol = np.linalg.solve(A,b)
    S = {'b':sol[::3],
         'c':sol[1::3],
         'd':sol[2::3],
         'x':x,
         'y':y
        }
    return S

# 'der' computes the 'der'-derivative of the Spline,
# but it has not been implemented. Can you do it? Please do it!
def cubic_spline_eval(xx,S,der=0):
    x=S['x']
    y=S['y']
    b=S['b']
    c=S['c']
    d=S['d']
    n=len(x)
    yy=np.zeros_like(xx)
    for i in np.arange(n-1):
        jj = np.where(np.logical_and(x[i]<=xx,xx<=x[i+1]))
        yy[jj]=y[i]+b[i]*(xx[jj]-x[i])+c[i]*(xx[jj]-x[i])**2+d[i]*(xx[jj]-x[i])**3
    return yy


I am working on the file "SC1/09_Least_Squares.ipynb".
The first task is:
<div id='in' />

Overdetermined Linear Systems of Equations
There is cases where the number of equations is greater than variables. Many times, those systems don't have an exact solution (inconsistent system). Then, in this case we needs an approximation closest to the data. Based in orthogonality, the shortest distance from a point to plane. The orthogonal distance represents the error which would be minimum.
\begin{equation}
                 \mathbf{b} - A\,\mathbf{x} = \mathbf{r}\
                 \mathbf{b} - A\,\mathbf{x} \perp {A\,\mathbf{x}\, |\, \mathbf{x} \in \mathbb{R}^m}
\end{equation}
The idea is that $\mathbf{r}$ would be closest to zero. We need to apply orthogonality to find the vector that satisfied this condition.
\begin{equation}
                 (A\,\mathbf{x})^\,(\mathbf{b}-A\,\overline{\mathbf{x}})=0, \hspace{1cm} \text{for all } \mathbf{x} \in \mathbb{R}^n\
                 \mathbf{x}^\, A^\,(\mathbf{b}-A\,\overline{\mathbf{x}})=0, \hspace{1cm} \text{for all } \mathbf{x} \in \mathbb{R}^n\
                 A^\,(\mathbf{b}-A\,\overline{\mathbf{x}})=\mathbf{0} \
                 A^\,A\,\overline{\mathbf{x}}= A^\,\mathbf{b} \
\end{equation}
This last equation gives us a new square $n\times n$ matrix, which let us resolve the equation system.
This linear system of equations is known as the Normal Equations.
Can you write Python code for it?

def least_squares(A,b):
    Q,R = QR(A,type_gram_schmidt='modified')
    return spla.solve_triangular(R,np.dot(Q.T,b))

def solve_model(M):
    A=M['A']
    b=M['b']
    M['x_bar']=least_squares(A,b)
    return M

def create_model(data, type_model='linear'):
    if type_model == 'linear': # f(x)=a0+a1*x
        A = np.ones((data.shape[0],2))
        A[:,1] = data[:,0]
        b = data[:,1]
    if type_model == 'parabollic': # f(x)=a0+a1*x+a_2*x^2
        A = np.ones((data.shape[0],3))
        A[:,1] = data[:,0]
        A[:,2] = data[:,0]**2
        b = data[:,1]
    if type_model == 'exponential': #f(x)=a0 \exp(a1*x) = \exp(\log(a0)+a1*x) -> log(f(x))=log(a0)+a1*x = A0+a1+x (it is linear now!)
        A = np.ones((data.shape[0],2))
        A[:,1] = data[:,0]
        b = np.log(data[:,1])
    M = {'A':A,
         'b':b,
         'type_model':type_model}
    M=solve_model(M)
    return M

def evaluate_model(M,x):
    x_bar=M['x_bar']
    if M['type_model'] == 'linear':
        return x_bar[0] + x_bar[1]*x
    if M['type_model'] == 'parabollic':
        return x_bar[0] + x_bar[1]*x + x_bar[2]*x**2
    if M['type_model'] == 'exponential':
        return np.exp(x_bar[0]+x_bar[1]*x)


I am working on the file "SC1/09_Least_Squares.ipynb".
The first task is:

Introduction
We have learned about square linear system of equations.
However, How can we solve a non-square system? (More equations that unknowns!) Well, we need to find a least squares approximation.

QR Factorization
Gram-Schmidt Orthogonalization
This algorithm orthogonalize a set of input vectors, returning an orthogonal set that spans the same column space.
We will only consider now that the input set of vectors are linearly independent.
Let $A=[\mathbf{a}_1\, ...., \mathbf{a}_n]$ a matrix with linearly independent column vectors $\in\mathbb{R}^m$ and $n \le m$.
We know the following for the orthogonal set:
\begin{align}
    \mathbf{q}_i^\,\mathbf{q}_i & = \|\mathbf{q}_i\|_2^2= 1\
    \mathbf{q}_i^\,\mathbf{q}_j & = 0, \, \text{ for } i\neq j
\end{align}
Then the Gram-Schmidt orthogonalization finds the $\mathbf{q}i$ and $r{ij}$ from the following set of equations and considering the previous constraints:
\begin{align}
    \mathbf{a}1 &= r{11}\,\mathbf{q}1\
    r{11} &= \|\mathbf{a}1\|_2\
    \mathbf{q}_1 &= \dfrac{\mathbf{a}_1}{r{11}}\
    \mathbf{a}2 &= r{12}\,\mathbf{q}1+r{22}\,\mathbf{q}2\
    r{12} &= \mathbf{q}_1^\,\mathbf{a}2\
    r{22} &= \|\mathbf{a}2-r{12}\,\mathbf{q}1\|_2\
    \mathbf{q}_2 &= \dfrac{\mathbf{a}_2-r{12}\,\mathbf{q}1}{r{22}}\
    \vdots &=  \vdots\
    \mathbf{a}j &= \sum{i=1}^j r_{ij}\,\mathbf{q}i\
    r{ij} &= \mathbf{q}i^\,\mathbf{a}j, \, \text{ for } i<j\
    r{jj} &= \left\|\mathbf{a}j-\sum{i=1}^{j-1} r_{ij}\,\mathbf{q}i\right\|_2\
    \mathbf{q}_j &= \dfrac{\mathbf{a}_j-\sum{i=1}^{j-1} r_{ij}\,\mathbf{q}i}{r_jj}\
    \vdots &=  \vdots\
    \mathbf{a}_n &= \sum{i=1}^n r_{in}\,\mathbf{q}i\
    r{in} &= \mathbf{q}_i^\,\mathbf{a}_n, \, \text{ for } i<n\
    r{nn} &= \left\|\mathbf{a}n-\sum{i=1}^{n-1} r_{in}\,\mathbf{q}i\right\|_2\
    \mathbf{q}_n &= \dfrac{\mathbf{a}_n-\sum{i=1}^{n-1} r_{in}\,\mathbf{q}i}{r{nn}}
\end{align*}
Thus, we obtain the QR decomposition as follows:
\begin{equation}
                 \mathbf{a}{m\times n} =...
Can you write Python code for it?

# Inputs:
# A: A set of linearly independent columns
# type_factorization: reduced or full
# type_gram_schmidt: classic or modified
def QR(A, type_factorization = 'reduced', type_gram_schmidt='classic'):
    A.astype('float')
    if type_factorization == 'reduced':
        Q = np.zeros(A.shape)
        R = np.zeros((A.shape[1],A.shape[1]))
    elif type_factorization == 'full':
        Q = np.zeros((A.shape[0],A.shape[0]))
        R = np.zeros(A.shape)
    for j in np.arange(A.shape[1]):
        y = A[:,j]
        for i in np.arange(j):
            if type_gram_schmidt == 'classic':
                R[i,j] = np.dot(Q[:,i],A[:,j])
            elif type_gram_schmidt == 'modified':
                R[i,j] = np.dot(Q[:,i],y)
            y=y-R[i,j]*Q[:,i]
        R[j,j] = np.linalg.norm(y)
        Q[:,j] = y/np.linalg.norm(R[j,j])
    # The following lines must be completed by you!
    #if type_factorization == 'full':
        # (1) We need to add 0's to the R matrix so it is of the same shape as the matrix A,
        # fortunately this was already done!
        # (2) We need to add orthogonal vectors to Q so it is square,
        # how do we do this?
    return Q,R

A = np.array([[1,-4],[2,3],[2,2]])
Qa, Ra = QR(A, type_factorization ='reduced', type_gram_schmidt='classic')
print(A)
print(np.dot(Qa,Ra))
print(Qa)
print(Ra)


I am working on the file "SC1/09_Least_Squares.ipynb".
The first task is:
Adjusting some models
Can you write Python code for it?

def generate_data(type_of_data='linear'):
    n=40
    np.random.seed(0)
    x = np.linspace(0,10,n)
    y = np.random.rand(n)
    x = np.concatenate((x,x,y),axis=0)
    n = 3*n
    if type_of_data=='linear':
        y = x+0.1*np.random.normal(0,1,n)+1.5
    elif type_of_data=='parabollic':
        y = 4*x**2+0.1*x*np.random.normal(0,1,n)+1.5
    elif type_of_data=='exponential':
        y = np.exp(x+0.1*np.random.normal(0,1,n)+1.5)
    elif type_of_data=='sinusoidal':
        y = np.sin(2*np.pi*x/10)+0.1*np.random.normal(0,1,n)+1.5
    elif type_of_data=='random':
        y = 0.1*np.random.normal(0,1,n)+1.5
    elif type_of_data=='boston house-prices':
        x,y=datasets.load_boston(return_X_y=True)
        x=x[:,5]
    elif type_of_data=='diabetes':
        x,y=datasets.load_diabetes(return_X_y=True)
        x=x[:,2]
    data = np.stack((x, y)).T
    return data

def looking_at_data(type_of_data='diabetes'):
    data=generate_data(type_of_data)
    Ml = create_model(data, type_model='linear')
    Mp = create_model(data, type_model='parabollic')
    Me = create_model(data, type_model='exponential')
    xx=np.linspace(np.min(data[:,0])-0.1,np.max(data[:,0])+0.1,1000)
    yyl=evaluate_model(Ml,xx)
    yyp=evaluate_model(Mp,xx)
    yye=evaluate_model(Me,xx)

    error_l=data[:,1]-evaluate_model(Ml,data[:,0])
    error_p=data[:,1]-evaluate_model(Mp,data[:,0])
    error_e=data[:,1]-evaluate_model(Me,data[:,0])

    plt.figure(figsize=(2*M,M))

    plt.subplot(1, 2, 1)
    plt.plot(xx,yyl,'k-',linewidth=5,label='linear model')
    plt.plot(xx,yyp,'y-',linewidth=20,label='parabollic model',alpha=0.4)
    plt.plot(xx,yye,'g-',linewidth=5,label='exponential model')
    plt.plot(data[:,0],data[:,1],'.b',markersize=20,label='original data',alpha=0.3)
    plt.grid(True)
    plt.xlabel(r'$x$')
    plt.ylabel(r'$y$')
    plt.legend(loc='best')
    #plt.ylim(0,10)

    plt.subplot(1, 2, 2)
    plt.title('What does this histogram tell us?')
    three_errors=np.vstack((error_l, error_p, error_e)).T
   ...


I am working on the file "SC1/09_Least_Squares.ipynb".
The first task is:
This method let us resolve a system of equations. However, exists a Full QR Factorization, creating the next system:
\begin{equation}
                 A_{m\times n} = Q_{m\times m}R_{m\times n}\
\end{equation}
Q is a square orthogonal matrix, adding $m-n$ columns and R grows adding $m-n$ zero rows.
Theorem
A square matrix $Q$ is orthogonal if $Q^*\, = Q^{-1}$
<div id='ex' />

Examples
Normal vs Modified Gram-Schmidt
Can you write Python code for it?

d = 1e-10
A = np.array([[1,1,1],[d,0,0],[0,d,0],[0,0,d]])
print(A)

Q1,R1 = QR(A, type_gram_schmidt = 'classic')
Q2,R2 = QR(A, type_gram_schmidt = 'modified')

# What are the Q's?
print('What are the Q\'s?')
print(Q1)
print(Q2)
# Are truly orthogonal the Q's?
print('Are truly orthogonal the Q\'s?')
print(np.dot(Q1.transpose(),Q1)) # Warning: We are just using the transpose since the matrices are real!
print(np.dot(Q2.transpose(),Q2))
# Do we recover A?
print('Showing Q1 and Q2')
print(np.dot(Q1,R1)-A)
print(np.dot(Q2,R2)-A)


I am working on the file "SC1/10_GMRes.ipynb".
The first task is:

Introduction
Welcome to another edition of our Jupyter Notebooks. A few notebooks back, we saw that the Conjugate Gradient Method, an iterative method, was very useful to solve $A\,\mathbf{x}=\mathbf{b}$ but it only worked when $A$ was positive definite and symmetric. So now we need an iterative method that works with nonsymmetric linear system of equations, and for that we have the Generalized Minimum Residual Method (GMRes). It works really well for finding the solution of large, sparse (and dense as well), nonsymmetric linear systems of equations. Of course, it will also have trouble for ill-conditioned linear system of equations. But it is really easy to add a left or right or both preconditioners!

A quick review on Least Squares
Least Squares is used to solve overdetemined linear systems of equations $A\,\mathbf{x} = \mathbf{b}$. That is, for example, a linear system of equations where there are more equations than unknowns. It finds the best $\overline{\mathbf{x}}$ so that it minimizes the euclidean length of $\mathbf{r} = \mathbf{b} - A\,\mathbf{x}$.
So, you might be wondering, what does Least Squares have to do with GMRes? WELL, since you're dying to know, I'll tell you: the backward error of the system in GMRes is minimized at each iteration step using a Least Squares formulation.

GMRes
GMRes is a member of the family of Krylov methods. It finds an approximation of $\mathbf{x}$ restricted to live on the Krylov sub-space $\mathcal{K_k}$,  where $\mathcal{K_k}={\mathbf{r}_0, A\,\mathbf{r}_0, A^2\,\mathbf{r}_0, \cdots, A^{k-1}\,\mathbf{r}_0}$ and $\mathbf{r}_0 = \mathbf{b} - A\,\mathbf{x}_0$ is the residual vector of the initial guess.
The idea behind this method is to look for improvements to the initial guess $\mathbf{x}_0$ in the Krylov space. At the $k$-th iteration, we enlarge the Krylov space by adding $A^k\,\mathbf{r}_0$, reorthogonalize the basis, and then use least squares to find the best improvement to add to $\mathbf{x}_0$.
The algorithm...
Can you write Python code for it?

# This is a very instructive implementation of GMRes.
def GMRes(A, b, x0=np.array([0.0]), m=10, flag_display=True, threshold=1e-12):
    n = len(b)
    if len(x0)==1:
        x0=np.zeros(n)
    r0 = b - np.dot(A, x0)
    nr0=np.linalg.norm(r0)
    out_res=np.array(nr0)
    Q = np.zeros((n,n))
    H = np.zeros((n,n))
    Q[:,0] = r0 / nr0
    flag_break=False
    for k in np.arange(np.min((m,n))):
        y = np.dot(A, Q[:,k])
        if flag_display:
            print('||y||=',np.linalg.norm(y))
        for j in np.arange(k+1):
            H[j][k] = np.dot(Q[:,j], y)
            if flag_display:
                print('H[',j,'][',k,']=',H[j][k])
            y = y - np.dot(H[j][k],Q[:,j])
            if flag_display:
                print('||y||=',np.linalg.norm(y))
        # All but the last equation are treated equally. Why?
        if k+1<n:
            H[k+1][k] = np.linalg.norm(y)
            if flag_display:
                print('H[',k+1,'][',k,']=',H[k+1][k])
            if (np.abs(H[k+1][k]) > 1e-16):
                Q[:,k+1] = y/H[k+1][k]
            else:
                print('flag_break has been activated')
                flag_break=True
            # Do you remember e_1? The canonical vector.
            e1 = np.zeros((k+1)+1)
            e1[0]=1
            H_tilde=H[0:(k+1)+1,0:k+1]
        else:
            H_tilde=H[0:k+1,0:k+1]
        # Solving the 'SMALL' least square problem.
        # This could be improved with Givens rotations!
        ck = np.linalg.lstsq(H_tilde, nr0*e1)[0]
        if k+1<n:
            x = x0 + np.dot(Q[:,0:(k+1)], ck)
        else:
            x = x0 + np.dot(Q, ck)
        # Why is 'norm_small' equal to 'norm_full'?
        norm_small=np.linalg.norm(np.dot(H_tilde,ck)-nr0*e1)
        out_res = np.append(out_res,norm_small)
        if flag_display:
            norm_full=np.linalg.norm(b-np.dot(A,x))
            print('..........||b-A\,x_k||=',norm_full)
            print('..........||H_k\,c_k-nr0*e1||',norm_small);
        if flag_break:
            if...


I am working on the file "SC1/10_GMRes.ipynb".
The first task is:
Another example, how may iteration does it need to converge?
Can you write Python code for it?

A = np.array([[0,0,0,1],[1,0,0,0],[0,1,0,0],[0,0,1,0]])
b = np.array([1,0,1,0])
x_our, _ = GMRes(A, b, m=10)
norm_full=np.linalg.norm(b-np.dot(A,x_our))
print(norm_full)

A = np.random.rand(10,10)+10*np.eye(10)
b = np.random.rand(10)
x_our, out_res = GMRes(A, b, m=10,flag_display=True)
norm_full=np.linalg.norm(b-np.dot(A,x_our))
print(norm_full)


I am working on the file "SC1/10_GMRes.ipynb".
The first task is:
Plotting the residual over the iterations
Can you write Python code for it?

plt.figure(figsize=(M,M))
plt.semilogy(out_res,'.k',markersize=20,label='residual')
plt.grid(True)
plt.xlabel(r'$k$')
plt.ylabel(r'$\|\mathbf{b}-A\,\mathbf{x}_k\|_2$')
plt.grid(True)
plt.show()


I am working on the file "SC1/Bonus - Sylvester Equation with GMRes .ipynb".
The first task is:

Sylvester Equation
https://en.wikipedia.org/wiki/Sylvester_equation
$$A\,X+X\,B=C$$
where $A\in\mathbb{R}^{n\times n}$, $B\in\mathbb{R}^{n\times n}$, $C\in\mathbb{R}^{n\times n}$ and $X\in\mathbb{R}^{n\times n}$.
$A$, $B$ and $C$ are given, the problem is to find $X$.

GMRes
GMRes is a member of the family of Krylov methods. It finds an approximation of $\mathbf{x}$ restricted to live on the Krylov sub-space $\mathcal{K_k}$,  where $\mathcal{K_k}={\mathbf{r}_0, A\,\mathbf{r}_0, A^2\,\mathbf{r}_0, \cdots, A^{k-1}\,\mathbf{r}_0}$ and $\mathbf{r}_0 = \mathbf{b} - A\,\mathbf{x}_0$ is the residual vector of the initial guess.
The idea behind this method is to look for improvements to the initial guess $\mathbf{x}_0$ in the Krylov space. At the $k$-th iteration, we enlarge the Krylov space by adding $A^k\,\mathbf{r}_0$, reorthogonalize the basis, and then use least squares to find the best improvement to add to $\mathbf{x}_0$.
The algorithm is as follows:
Generalized Minimum Residual Method
$\mathbf{x}0$ = initial guess

$\mathbf{r}$ = $\mathbf{b} - A\,\mathbf{x}_0$ = $\mathbf{b} - $afun$(\mathbf{x}_0)$

$\mathbf{q}_1$ = $\mathbf{r} / \|\mathbf{r}\|_2$

for $k = 1, ..., m$

$\qquad \ \ \mathbf{y} = A\,\mathbf{q}_k$ = afun$(\mathbf{q}_k)$

$\qquad$ for $j = 1,2,...,k$

$\qquad \qquad$ $h{jk} = \mathbf{q}j^*\,\mathbf{y}$

$\qquad \qquad$ $\mathbf{y} = \mathbf{y} - h{jk}\, \mathbf{q}j$

$\qquad$ end

$\qquad \ h{k+1,k} = \|y\|2 \qquad$ (If $h{k+1,k} = 0$ , skip next line and terminate at bottom.)

$\qquad \ \mathbf{q}{k+1} = \mathbf{y}/h{k+1,k}$

$\qquad$ Minimize $\left\|\widehat{H}_k\, \mathbf{c}_k - [\|\mathbf{r}\|_2 \ 0 \ 0 \ ... \ 0]^T \right\|_2$ for $\mathbf{c}_k$

$\qquad$ $\mathbf{x}_k = Q_k \, \mathbf{c}_k + \mathbf{x}_0$

end
Generating an instance of the Sylvester Equation
Can you write Python code for it?

n = 10
np.random.seed(0)
A = np.random.rand(n,n)+2*np.eye(n)
#print(np.linalg.eigvals(A))
B = np.random.rand(n,n)
#print(np.linalg.eigvals(B))
C = np.random.rand(n,n)


I am working on the file "SC1/Bonus - Sylvester Equation with GMRes .ipynb".
The first task is:
Computing the relative residues
Can you write Python code for it?

Ax_JGS = Ax(X_JGS.flatten())
Ax_GMRes = Ax(X_GMRes.flatten())
c =  C.flatten()
print(np.linalg.norm(Ax_JGS-c)/np.linalg.norm(c))
print(np.linalg.norm(Ax_GMRes-c)/np.linalg.norm(c))


I am working on the file "SC1/Bonus - Sylvester Equation with GMRes .ipynb".
The first task is:
Using GMRes of SciPy
Can you write Python code for it?

def compute_matrix_vector_product(x,A,B,n):
    X = np.reshape(x,(n,n))
    out = np.dot(A,X)+np.dot(X,B)
    return out.flatten()
Ax = lambda x: compute_matrix_vector_product(x,A,B,n)
afun = spla.LinearOperator((n**2, n**2), matvec=Ax)
x, exitCode = spla.gmres(afun, C.flatten(), tol=1e-10)
X_GMRes = np.reshape(x,(n,n))
print(X_GMRes)

print(np.linalg.norm(X_JGS-X_GMRes))


I am working on the file "SC1/Bonus - Sylvester Equation with GMRes .ipynb".
The first task is:
Using a Jacobi/Gauss-Seidel iterative solver
Can you write Python code for it?

def solve_JGS_iterative_Sylvester(A,B,C,m,alg=1):
    if alg==1:
        # Algorithm 1
        # AX+XB=C
        # X=A^{-1}(C-XB)
        # X^{(i+1)}=A^{-1}(C-X^{(i)} B)
        X0 = np.zeros_like(A)
        X1 = np.zeros_like(A)
        for i in range(m):
            X1=np.linalg.solve(A,C-np.dot(X0,B))
            X0=X1
            print(np.linalg.norm(np.dot(A,X1)+np.dot(X1,B)-C))
        return X1
#    elif algo==2: # TO DO 1!!!!!!!!!!!!!
        # Algorithm 2
        # AX+XB=C
        # X = (C-AX)B^{-1}
        # X^{(i+1)}=(C-A X^{(i)})B^{-1}
        # How do we implement this? Hint: You only need to use np.linalg.solve in a convenient way.

X_JGS=solve_JGS_iterative_Sylvester(A,B,C,10)

print(X_JGS)


I am working on the file "SC1/Bonus_Finding_2_Chebyshev_Points_Graphically.ipynb".
The first task is:
Finally, we have included the min in the plot and we see the agreement between the min of $\omega(x_1,x_2)$ and the Chebyshev points found.
Can you write Python code for it?

plt.figure(figsize=(8,8))
plt.contourf(X, Y, W,100, cmap=cm.nipy_spectral, antialiased=False)
plt.plot(c1,c2,'k.',markersize=16)
plt.plot(c2,c1,'k.',markersize=16)
plt.colorbar()
plt.xlabel(r'$x_1$')
plt.ylabel(r'$x_2$')
plt.show()


I am working on the file "SC1/Bonus_Finding_2_Chebyshev_Points_Graphically.ipynb".
The first task is:
Now we need to evaluate $\omega(x_1,x_2)$ over the domain $\Omega=[0,1]^2$.
Can you write Python code for it?


I am working on the file "SC1/Bonus_Finding_2_Chebyshev_Points_Graphically.ipynb".
The first task is:
Recall that the Chebyshev points are points that minimize the following expression:
$$
\displaystyle{\omega(x_1,x_2,\dots,x_n)=\max_{x} |(x-x_1)\,(x-x_2)\,\cdots\,(x-x_n)|}.
$$
This comes from the Interpolation Error Formula (I hope you remember it, otherwise see the textbook or the classnotes!).
In this notebook, we will find the $\min$ for 2 points,
this means:
$$
[x_1,x_2]= \displaystyle{\mathop{\mathrm{argmin}}{x_1,x_2\in [-1,1]}} \,\omega(x_1,x_2)=\displaystyle{\mathop{\mathrm{argmin}}{x_1,x_2\in [-1,1]}}\,
\max_{x\in [-1,1]} |(x-x_1)\,(x-x_2)|.
$$
For doing this, we first need to build $\omega(x_1,x_2)$,
Can you write Python code for it?

N=50
x=np.linspace(-1,1,N)
w = lambda x1,x2: np.max(np.abs((x-x1)*(x-x2)))
wv=np.vectorize(w)


I am working on the file "SC1/Bonus_Finding_2_Chebyshev_Points_Graphically.ipynb".
The first task is:
With this data, we can now plot the function $\omega(x_1,x_2)$ on $\Omega$.
The minimun value of is shown by the color at the bottom of the colorbar.
By visual inspection, we see that we have two mins.
They are located at the bottom right and top left.
Can you write Python code for it?

plt.figure(figsize=(8,8))
#plt.contourf(X, Y, W,100, cmap=cm.hsv, antialiased=False)
plt.contourf(X, Y, W,100, cmap=cm.nipy_spectral, antialiased=False)
plt.xlabel(r'$x_1$')
plt.ylabel(r'$x_2$')
plt.colorbar()
plt.show()


I am working on the file "SC1/Bonus_Newton_Rn.ipynb".
The first task is:
<div id='newton' />

Newton's method
Can you write Python code for it?

f1= lambda x,y: x**2+y**2-1
f2= lambda x,y: y-x**2
J = lambda x,y: np.array([[2*x, 2*y],[-2*x, 1]])

Newton = lambda x,y: np.array([[x],[y]])-np.linalg.solve(J(x,y),np.array([[f1(x,y)],[f2(x,y)]]))

delta = 0.025
x = np.arange(-1.5, 1.5, delta)
y = np.arange(-1.5, 1.5, delta)
X, Y = np.meshgrid(x, y)
Z1 = f1(X,Y)
Z2 = f2(X,Y)

plt.figure()
CS1 = plt.contour(X, Y, Z1,levels=[0])
CS2 = plt.contour(X, Y, Z2,levels=[0])
#plt.clabel(CS1, inline=1, fontsize=10)
#plt.clabel(CS2, inline=1, fontsize=10)
plt.grid()
plt.axis('equal')
plt.title(r'Newton $\mathbb{R}^n$')
plt.show()

def Show_Newton(x0=1.2,y0=0.3,n=0):
    plt.figure()
    CS1 = plt.contour(X, Y, Z1,levels=[0])
    CS2 = plt.contour(X, Y, Z2,levels=[0])
    plt.grid()
    plt.axis('equal')
    plt.title(r'Newton $\mathbb{R}^n$')
    plt.plot(x0,y0,'rx')
    print(x0,y0)
    for i in np.arange(n):
        xout=Newton(x0,y0)
        x1=float(xout[0])
        y1=float(xout[1])
        plt.plot(x1,y1,'rx')
        plt.plot([x0, x1],[y0, y1],'r')
        x0=x1
        y0=y1
        print(x0,y0)
    plt.show()

interact(Show_Newton,x0=(-1.4,1.4,0.1),y0=(-1.4,1.4,0.1), n=(0,100,1))


I am working on the file "SC1v2/02_floating_point_arithmetic.ipynb".
The first task is:
<div id='firstinteger' />

What is the first integer that is not representable in double precision?
Back to TOC
Recall that $\epsilon_{\text{mach}}=2^{-52}$ in double precision.
Can you write Python code for it?