I am working on the file "07-Control-Flow-Statements.ipynb".
The first task is:
Note especially the use of colons (:) and whitespace to denote separate blocks of code.
Python adopts the if and else often used in other languages; its more unique keyword is elif, a contraction of "else if".
In these conditional clauses, elif and else blocks are optional; additionally, you can optinally include as few or as many elif statements as you would like.
for loops
Loops in Python are a way to repeatedly execute some code statement.
So, for example, if we'd like to print each of the items in a list, we can use a for loop:
Can you write Python code for it?

for N in [2, 3, 5, 7]:
    print(N, end=' ') # print all on same line


I am working on the file "07-Control-Flow-Statements.ipynb".
The first task is:
Note that the range starts at zero by default, and that by convention the top of the range is not included in the output.
Range objects can also have more complicated values:
Can you write Python code for it?

# range from 5 to 10
list(range(5, 10))

# range from 0 to 10 by 2
list(range(0, 10, 2))


I am working on the file "07-Control-Flow-Statements.ipynb".
The first task is:
Notice that we use a while True loop, which will loop forever unless we have a break statement!
Loops with an else Block
One rarely used pattern available in Python is the else statement as part of a for or while loop.
We discussed the else block earlier: it executes if all the if and elif statements evaluate to False.
The loop-else is perhaps one of the more confusingly-named statements in Python; I prefer to think of it as a nobreak statement: that is, the else block is executed only if the loop ends naturally, without encountering a break statement.
As an example of where this might be useful, consider the following (non-optimized) implementation of the Sieve of Eratosthenes, a well-known algorithm for finding prime numbers:
Can you write Python code for it?

L = []
nmax = 30

for n in range(2, nmax):
    for factor in L:
        if n % factor == 0:
            break
    else: # no break
        L.append(n)
print(L)


I am working on the file "07-Control-Flow-Statements.ipynb".
The first task is:
Notice the simplicity of the for loop: we specify the variable we want to use, the sequence we want to loop over, and use the "in" operator to link them together in an intuitive and readable way.
More precisely, the object to the right of the "in" can be any Python iterator.
An iterator can be thought of as a generalized sequence, and we'll discuss them in Iterators.
For example, one of the most commonly-used iterators in Python is the range object, which generates a sequence of numbers:
Can you write Python code for it?

for i in range(10):
    print(i, end=' ')


I am working on the file "07-Control-Flow-Statements.ipynb".
The first task is:
You might notice that the meaning of range arguments is very similar to the slicing syntax that we covered in Lists.
Note that the behavior of range() is one of the differences between Python 2 and Python 3: in Python 2, range() produces a list, while in Python 3, range() produces an iterable object.
while loops
The other type of loop in Python is a while loop, which iterates until some condition is met:
Can you write Python code for it?

i = 0
while i < 10:
    print(i, end=' ')
    i += 1


I am working on the file "07-Data-Visualization/CrimeData_descriptive_statistics_and_visualization.ipynb".
The first task is:
Convert Time Columns
Please refer to the following resources for working with time series data in pandas:
- https://pandas.pydata.org/pandas-docs/stable/timeseries.html
- https://pandas.pydata.org/pandas-docs/stable/api.html#id10
Can you write Python code for it?

# function currying
def fixdatetime(fld):
    def _fix(s):
        date_col = '%s_date' % fld # "rpt_date"
        time_col = '%s_time' % fld # "rpt_time"
        if time_col in s.index:
            return str(s[date_col])+' '+str(s[time_col])
        else:
            return str(s[date_col])+' 00:00:00'
    return _fix

for col in ['rpt', 'occur', 'poss']:
    datser = df.apply(fixdatetime(col), axis=1)
    df['%s_dt'%col] = pd.to_datetime(datser, format="%m/%d/%Y %H:%M:%S", errors='coerce')

df[["MI_PRINX", "offense_id", "beat", "UC2 Literal", "neighborhood", "rpt_dt", "occur_dt", "poss_dt"]].head()


I am working on the file "07-Data-Visualization/CrimeData_descriptive_statistics_and_visualization.ipynb".
The first task is:
Correlation In Number of Crimes Over Time
You can use the "corr" method in Pandas to find the correlation between columns of a dataframe.
Can you write Python code for it?


I am working on the file "07-Data-Visualization/CrimeData_descriptive_statistics_and_visualization.ipynb".
The first task is:
Crime Over Time
Has the number of crimes in Atlanta changed over time?
Are some areas more affected by crime than others?
Do different types of crime correlate with each other?
Number of Crimes Over Time, with Pivot Tables
Can you write Python code for it?

df[["occur_dt", "UC2 Literal", "offense_id"]].head()

# Pivoting the table:
# index = nolumn that the new table will be indexed by
# columns = column whose unique values will form the new column names
# values = values used to fill the table (default = all columns other than those given in index and columns)
df_ct = df.pivot_table(index="occur_dt", columns="UC2 Literal", values="offense_id")

df_ct.head()


I am working on the file "07-Data-Visualization/CrimeData_descriptive_statistics_and_visualization.ipynb".
The first task is:
Looks like most of the data is actually from 2009-2017! Let's throw the rest away...
Can you write Python code for it?


I am working on the file "07-Data-Visualization/CrimeData_descriptive_statistics_and_visualization.ipynb".
The first task is:
Number of crimes reported each year:
Can you write Python code for it?

# resample is like "groupby" for time
df.resample('A-DEC', closed='right', on='occur_dt').offense_id.count()
# df['Year'] = df.occur_dt.map(lambda d: d.year)
# df2 = df[(df.Year>=2010) & (df.Year<=2017)]
# df2.shape, df.shape


I am working on the file "07-Data-Visualization/CrimeData_descriptive_statistics_and_visualization.ipynb".
The first task is:
Review: Creating a data key
Let's look at the structure of this table. We're actually creating some text output that can be used to create a data dictionary.
Can you write Python code for it?

dataDict = pd.DataFrame({'DataType': df.dtypes.values, 'Description': '', }, index=df.columns.values)


I am working on the file "07-Data-Visualization/CrimeData_descriptive_statistics_and_visualization.ipynb".
The first task is:
This gives us a timeline for different types of crime reported in Atlanta.
By itself, this can be useful, but we are more interested in aggregate statistics. Let's get the number of crimes by month...
Can you write Python code for it?

df_ct = df_ct.resample("1M", closed="right").count()

df_ct.head()


I am working on the file "07-Data-Visualization/CrimeData_descriptive_statistics_and_visualization.ipynb".
The first task is:
We need to enter the descriptions for each entry in our dictionary manually...
Can you write Python code for it?

dataDict.loc['MI_PRINX'].Description = '' # type: int64
dataDict.loc['offense_id'].Description = 'Unique ID in the format YYDDDNNNN with the year YY, the day of the year DDD and a counter NNNN' # type: int64
dataDict.loc['rpt_date'].Description = 'Date the crime was reported' # type: object
dataDict.loc['occur_date'].Description = 'Estimated date when the crime occured' # type: object
dataDict.loc['occur_time'].Description = 'Estimated time when the crime occured' # type: object
dataDict.loc['poss_date'].Description = '' # type: object
dataDict.loc['poss_time'].Description = '' # type: object
dataDict.loc['beat'].Description = '' # type: int64
dataDict.loc['apt_office_prefix'].Description = '' # type: object
dataDict.loc['apt_office_num'].Description = '' # type: object
dataDict.loc['location'].Description = '' # type: object
dataDict.loc['MinOfucr'].Description = '' # type: int64
dataDict.loc['MinOfibr_code'].Description = '' # type: object
dataDict.loc['dispo_code'].Description = '' # type: object
dataDict.loc['MaxOfnum_victims'].Description = '' # type: float64
dataDict.loc['Shift'].Description = 'Zones have 8 or 10 hour shifts' # type: object
dataDict.loc['Avg Day'].Description = '' # type: object
dataDict.loc['loc_type'].Description = '' # type: float64
dataDict.loc['UC2 Literal'].Description = '' # type: object
dataDict.loc['neighborhood'].Description = '' # type: object
dataDict.loc['npu'].Description = '' # type: object
dataDict.loc['x'].Description = '' # type: float64
dataDict.loc['y'].Description = '' # type: float64
dataDict.to_csv("COBRA_Data_Dictionary.csv")

dataDict


I am working on the file "07-Data-Visualization/CrimeData_descriptive_statistics_and_visualization.ipynb".
The first task is:
What's the date range of the data?
Can you write Python code for it?

print df.occur_dt.min(), '---', df.occur_dt.max()


I am working on the file "07-Data-Visualization/MoreAPD_class.ipynb".
The first task is:
Beats and Zones
The City of Atlanta is divided into 6 zones. Each with 12 to 14 beats.

Let's create a separate column for the zones:
Can you write Python code for it?

df.beat[:10]

df['Zone'] = df['beat']//100

df.Zone[:4]

df['UC2 Literal'].unique()

##df[df['UC2 Literal']=='LARCENY-FROM VEHICLE']

df.occur_date.min(), df.occur_date.max()

df['Year'] = df.rpt_dt.map(lambda d: d.year)

df.groupby('Year').offense_id.count()

brdf = df[df['UC2 Literal']=='BURGLARY-RESIDENCE'].copy()

brdf.shape, df.shape

def gethour(d):
    return d.hour

brdf.occur_dt.map(gethour)
##brdf['occur_dt'].map(gethour)

##brdf.occur_dt.map(lambda d: d.hour)

print type(brdf.occur_dt)
brdf['Hour'] = brdf.occur_dt.apply(gethour)

brdf.head()

br_hr = brdf.groupby(['Hour']).offense_id.count()

plt.step(br_hr.index, br_hr.values)

plt.figure(figsize=(20,8))
for z in range(1,7):
    plt.subplot(3,2,z)
    plt.title("Zone %d" % z)
    #brdf[brdf.Zone==z].hist(column='Hour', bins=24)
    plt.hist(brdf[brdf.Zone==z].Hour, bins=24)
plt.show()

plt.figure(figsize=(30,15))
for h in range(24):
    plt.subplot(4,6,h+1)
    plt.title("Hour %d" % h)
    #brdf[brdf.Zone==z].hist(column='Hour', bins=24)
    plt.hist(brdf[brdf.Hour==h].Zone, bins=6)
    plt.ylim(0,40) ## sets limit on Y-axis
plt.show()

df['UC2 Literal'].unique()

df.groupby(['UC2 Literal', 'Zone']).offense_id.count()

df['dayofweek'] = df.occur_dt.map(lambda d: d.dayofweek)

df.groupby(['UC2 Literal','dayofweek']).offense_id.count()

brdf.apply(lambda r: str(r.location)+', '+str(r.npu), axis=1)

brdf.apply(np.min, axis=0)

df.occur_dt.map(lambda d: d.year).unique()

df['Year'] = df.occur_dt.map(lambda d: d.year)
df2 = df[(df.Year>=2010) & (df.Year<=2017)]
df2.shape, df.shape


I am working on the file "07-Data-Visualization/MoreAPD_class.ipynb".
The first task is:
Convert Columns
Can you write Python code for it?

for col in ['rpt', 'occur', 'poss']:
    datser = df.apply(fixdatetime(col), axis=1)
    df['%s_dt'%col] = pd.to_datetime(datser, format="%m/%d/%Y %H:%M:%S", errors='coerce')

df.head()

df.dtypes


I am working on the file "07-Data-Visualization/MoreAPD_class.ipynb".
The first task is:
Descriptive Statistics
https://pandas.pydata.org/pandas-docs/stable/basics.html#descriptive-statistics
Time Series

https://pandas.pydata.org/pandas-docs/stable/timeseries.html
https://pandas.pydata.org/pandas-docs/stable/api.html#id10
Can you write Python code for it?

df_LarcenyFromVehicle = df2[(df2['UC2 Literal']=='LARCENY-FROM VEHICLE')&(df2.Year==2017)].copy()
agr_LarcenyFromVehicle = df_LarcenyFromVehicle.set_index('occur_dt').resample('W').offense_id.count()
agr_LarcenyFromVehicle

df_LarcenyFromVehicle["Hour"] = df_LarcenyFromVehicle.occur_dt.map(lambda d: d.hour)

df_LarcenyFromVehicle.groupby("Hour").offense_id.count()

hourly = df_LarcenyFromVehicle.resample('H', on='occur_dt').offense_id.count()

hourly.reset_index().occur_dt.map(lambda d: d.week)

df3 = pd.DataFrame({"N": hourly})
##df3['Day'] = df3.reset_index().occur_dt ##.map(lambda d: d.day)
df3

ls


I am working on the file "07-Data-Visualization/MoreAPD_class.ipynb".
The first task is:
Fixing Data Types
Date and Time

Working with dates can be tricky. Often dates and times are coded as strings and need to be converted to a date and time data format.
Python provides a module datetime to deal with converting parsing and formatting dates and times. See https://docs.python.org/2/library/datetime.html
The pandas package provides functionality to convert text fields into date/time fields...given the values adhere to a given format. See http://pandas.pydata.org/pandas-docs/version/0.20/generated/pandas.to_datetime.html

Create a proper text field
In order to use the text to date/time converter our text columns need to have the appropriate format.
Can you write Python code for it?

df[['occur_date', 'occur_time']][0:5]

# function currying
def fixdatetime(fld):
    def _fix(s):
        date_col = '%s_date' % fld
        time_col = '%s_time' % fld
        if time_col in s.index:
            return str(s[date_col])+' '+str(s[time_col])
        else:
            return str(s[date_col])+' 00:00:00'
    return _fix

##df.apply(fixdatetime('occur'), axis=1)[:10]


I am working on the file "07-Data-Visualization/MoreAPD_class.ipynb".
The first task is:
We need to enter the descriptions for each entry in our dictionary manually. However, why not just create a the Python code automatically...
Run the code below only if you haven't edited the datadict.py file in a different way, since it will overwrite what you have so far. (That's why the code is commented-out.)
Can you write Python code for it?

dataDict

with open("datadict2.py", "w") as io:
     for i in dataDict.index:
         io.write("dataDict.loc['%s'].Description = '' # type: %s\n" % (i, str(dataDict.loc[i].DataType)))

ls -l datadict2.py

# %load datadict.py
dataDict.loc['MI_PRINX'].Description = '' # type: int64
dataDict.loc['offense_id'].Description = 'Unique ID in the format YYDDDNNNN with the year YY, the day of the year DDD and a counter NNNN' # type: int64
dataDict.loc['rpt_date'].Description = 'Date the crime was reported' # type: object
dataDict.loc['occur_date'].Description = 'Estimated date when the crime occured' # type: object
dataDict.loc['occur_time'].Description = 'Estimated time when the crime occured' # type: object
dataDict.loc['poss_date'].Description = '' # type: object
dataDict.loc['poss_time'].Description = '' # type: object
dataDict.loc['beat'].Description = '' # type: int64
dataDict.loc['apt_office_prefix'].Description = '' # type: object
dataDict.loc['apt_office_num'].Description = '' # type: object
dataDict.loc['location'].Description = '' # type: object
dataDict.loc['MinOfucr'].Description = '' # type: int64
dataDict.loc['MinOfibr_code'].Description = '' # type: object
dataDict.loc['dispo_code'].Description = '' # type: object
dataDict.loc['MaxOfnum_victims'].Description = '' # type: float64
dataDict.loc['Shift'].Description = 'Zones have 8 or 10 hour shifts' # type: object
dataDict.loc['Avg Day'].Description = '' # type: object
dataDict.loc['loc_type'].Description = '' # type: float64
dataDict.loc['UC2 Literal'].Description = '' # type: object
dataDict.loc['neighborhood'].Description = '' # type: object
dataDict.loc['npu'].Description = '' # type: object
dataDict.loc['x'].Description = '' # type: float64
dataDict.loc['y'].Description = '' # type: float64
dataDict.to_csv("COBRA_Data_Dictionary.csv")

sorted(df.npu.unique())

len(df.neighborhood.unique())


I am working on the file "07-Data-Visualization/MoreAPD_orig.ipynb".
The first task is:
Beats and Zones
The City of Atlanta is divided into 6 zones. Each with 12 to 14 beats.

Let's create a separate column for the zones:
Can you write Python code for it?


I am working on the file "07-Data-Visualization/MoreAPD_orig.ipynb".
The first task is:
Date and Time

Working with dates can be tricky. Often dates and times are coded as strings and need to be converted to a date and time data format.
Python provides a module datetime to deal with converting parsing and formatting dates and times. See https://docs.python.org/2/library/datetime.html
The pandas package provides functionality to convert text fields into date/time fields...given the values adhere to a given format. See http://pandas.pydata.org/pandas-docs/version/0.20/generated/pandas.to_datetime.html

Create a proper text field
In order to use the text to date/time converter our text columns need to have the appropriate format.
Can you write Python code for it?

df[['occur_date', 'occur_time']]

# function currying
def fixdatetime(fld):
    def _fix(s):
        date_col = '%s_date' % fld
        time_col = '%s_time' % fld
        if time_col in s.index:
            return str(s[date_col])+' '+str(s[time_col])
        else:
            return str(s[date_col])+' 00:00:00'
    return _fix

df.apply(fixdatetime('occur'), axis=1)[:10]


I am working on the file "07-Data-Visualization/MoreAPD_orig.ipynb".
The first task is:
Descriptive Statistics
https://pandas.pydata.org/pandas-docs/stable/basics.html#descriptive-statistics
Time Series

https://pandas.pydata.org/pandas-docs/stable/timeseries.html
https://pandas.pydata.org/pandas-docs/stable/api.html#id10
Can you write Python code for it?


I am working on the file "07-Data-Visualization/MoreAPD_orig.ipynb".
The first task is:
Fixing Data Types
Can you write Python code for it?


I am working on the file "07-Data-Visualization/MoreAPD_orig.ipynb".
The first task is:
Seasonal Model
Can you write Python code for it?

fig = plt.figure(figsize=(40,30))
crime_types = crime_year.index.levels[0]
years = crime_year.index.levels[1]
for c in range(len(crime_types)):
    y_max = max(crime_year.loc[crime_types[c]])

    plt.subplot(4,3,c+1)
    plt.hlines(crime_year.loc[crime_types[c]].iloc[-1]*100/y_max, years[0], years[-1], linestyles="dashed", color="r")
    plt.bar(crime_year.loc[crime_types[c]].index, crime_year.loc[crime_types[c]]*100/y_max, label=crime_types[c], alpha=0.5)
    ##plt.legend()
    plt.ylim(0, 100)
    plt.xticks(years+0.4, [str(int(y)) for y in years], rotation=0, fontsize=24)
    plt.yticks([0,20,40,60,80,100], ['0%','20%','40%','60%','80%','100%'], fontsize=24)
    plt.title(crime_types[c], fontsize=30)
    None

c = 3 ## 'BURGLARY-RESIDENCE'
resburglaries = crime_year_month.loc[crime_types[c]]
fig = plt.figure(figsize=(20,10))
for y in years:
    plt.plot(resburglaries.loc[y].index, resburglaries.loc[y], label=("%4.0f"%y))
plt.legend()
plt.title("Seasonal Trends - %s"%crime_types[c], fontsize=20)
plt.xticks(range(13), ['', 'Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'])
plt.xlim(0,13)
None

c = 3 ## 'BURGLARY-RESIDENCE'
fig = plt.figure(figsize=(20,10))
for y in years:
    avg = resburglaries.loc[y].mean()
    std = resburglaries.loc[y].std()
    ##plt.hlines(avg, 1, 13, linestyle='dashed')
    plt.plot(resburglaries.loc[y].index, (resburglaries.loc[y]-avg)/std, label=("%4.0f"%y))
plt.legend()
plt.title("Seasonal Trends - %s (normalized)"%crime_types[c], fontsize=20)
plt.xticks(list(range(1,13)), ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'])
plt.xlim(0,13)
plt.ylabel("Standard deviations $\sigma_y$")
None


I am working on the file "07-Data-Visualization/MoreAPD_orig.ipynb".
The first task is:
We need to enter the descriptions for each entry in our dictionary manually. However, why not just create a the Python code automatically...
Run the code below only if you haven't edited the datadict.py file in a different way, since it will overwrite what you have so far. (That's why the code is commented-out.)
Can you write Python code for it?

dataDict

with open("datadict2.py", "w") as io:
     for i in dataDict.index:
         io.write("dataDict.loc['%s'].Description = '' # type: %s\n" % (i, str(dataDict.loc[i].DataType)))

ls -l datadict2.py

# %load datadict2.py
dataDict.loc['MI_PRINX'].Description = '' # type: int64
dataDict.loc['offense_id'].Description = '' # type: int64
dataDict.loc['rpt_date'].Description = '' # type: object
dataDict.loc['occur_date'].Description = '' # type: object
dataDict.loc['occur_time'].Description = '' # type: object
dataDict.loc['poss_date'].Description = '' # type: object
dataDict.loc['poss_time'].Description = '' # type: object
dataDict.loc['beat'].Description = '' # type: int64
dataDict.loc['apt_office_prefix'].Description = '' # type: object
dataDict.loc['apt_office_num'].Description = '' # type: object
dataDict.loc['location'].Description = '' # type: object
dataDict.loc['MinOfucr'].Description = '' # type: int64
dataDict.loc['MinOfibr_code'].Description = '' # type: object
dataDict.loc['dispo_code'].Description = '' # type: object
dataDict.loc['MaxOfnum_victims'].Description = '' # type: float64
dataDict.loc['Shift'].Description = '' # type: object
dataDict.loc['Avg Day'].Description = '' # type: object
dataDict.loc['loc_type'].Description = '' # type: float64
dataDict.loc['UC2 Literal'].Description = '' # type: object
dataDict.loc['neighborhood'].Description = '' # type: object
dataDict.loc['npu'].Description = '' # type: object
dataDict.loc['x'].Description = '' # type: float64
dataDict.loc['y'].Description = '' # type: float64

# %load datadict.py
dataDict.loc['MI_PRINX'].Description = '' # type: int64
dataDict.loc['offense_id'].Description = 'Unique ID in the format YYDDDNNNN with the year YY, the day of the year DDD and a counter NNNN' # type: int64
dataDict.loc['rpt_date'].Description = 'Date the crime was reported' # type: object
dataDict.loc['occur_date'].Description = 'Estimated date when the crime occured' # type: object
dataDict.loc['occur_time'].Description = 'Estimated time when the crime...


I am working on the file "07-Tables/reading_tables.ipynb".
The first task is:
Astropy can read a variety of formats easily.
The following example uses a quite
Can you write Python code for it?

# Read the data from the source
table = ascii.read("ftp://cdsarc.u-strasbg.fr/pub/cats/VII/253/snrs.dat",
                   readme="ftp://cdsarc.u-strasbg.fr/pub/cats/VII/253/ReadMe")

# See the stats of the table

# If we want to see the first 10 entries

# the units are also stored, we can extract them too

# Adding values of different columns

# adding values of different columns but being aware of column units

# Create a new column in the table

# Show table's new column

# add a description to the new column

# Now it does show the values

# Using numpy to calculate the sin of the RA

# Let's change the units...

# does the sin now works?


I am working on the file "07-Tables/reading_tables.ipynb".
The first task is:
Automatically, read has identified the header and the format of each column. The result is a Table object, and that brings some additional properties.
Can you write Python code for it?

# Show the info of the data read

# Get the name of the columns

# Get just the values of a particular column

# get the first element


I am working on the file "07-Tables/reading_tables.ipynb".
The first task is:
When the values are not empty, then the keyword fill_values on read has to be used.
Reading VOTables
VOTables are an special type of tables which should be self-consistent and can be tied to a particular scheme.
This mean the file will contain where the data comes from (and which query produced it) and the properties for each field, making it easier to ingest by a machine.
Can you write Python code for it?

from astropy.io.votable import parse_single_table

# Read the example table from HELIO (hfc_ar.xml)

# See the fields of the table

# extract one  (NOAA_NUMBER) or all of the columns

# Show the data

# See the mask

# Shee the whole array.

# Convert the table to an astropy table

# See the table

# Different results because quantities are not

# And it can also be converted to other units


I am working on the file "07-Tables/reading_tables_Instructor.ipynb".
The first task is:
Properties when reading
the reading of the table has many properties, let's imagine the following easy example:
Can you write Python code for it?

weather_data = """
# Country = Finland
# City = Helsinki
# Longitud = 24.9375
# Latitud = 60.170833
# Week = 32
# Year = 2015
day, precip, type
Mon,1.5,rain
Tues,,
Wed,1.1,snow
Thur,2.3,rain
Fri,0.2,
Sat,1.1,snow
Sun,5.4,snow
"""

# Read the table
weather = ascii.read(weather_data)

# Blank values are interpreted by default as bad/missing values
weather.info('stats')

# Let's define missing values for the columns we want:
weather['type'].fill_value = 'N/A'
weather['precip'].fill_value = -999

# Use filled to show the value filled.
weather.filled()

# We can see the meta as a dictionary, but not as key, value pairs
weather.meta

# To get it the header as a table
header = ascii.read(weather.meta['comments'], delimiter='=',
                    format='no_header', names=['key', 'val'])
print(header)


I am working on the file "07-Tables/reading_tables_Instructor.ipynb".
The first task is:
When the values are not empty, then the keyword fill_values on read has to be used.
Reading VOTables
VOTables are an special type of tables which should be self-consistent and can be tied to a particular scheme.
This mean the file will contain where the data comes from (and which query produced it) and the properties for each field, making it easier to ingest by a machine.
Can you write Python code for it?

from astropy.io.votable import parse_single_table

# Read the example table from HELIO (hfc_ar.xml)
table = parse_single_table("hfc_ar.xml")

# See the fields of the table
table.fields

# extract one  (NOAA_NUMBER) or all of the columns
NOAA = table.array['NOAA_NUMBER']

# Show the data
NOAA.data

# See the mask
NOAA.mask

# Shee the whole array.
NOAA

# Convert the table to an astropy table
asttable = table.to_table()

# See the table
asttable

# Different results because quantities are not
print(np.sin(asttable['FEAT_HG_LAT_DEG'][0:5]))
print(np.sin(asttable['FEAT_HG_LAT_DEG'][0:5].quantity))

# And it can also be converted to other units
print(asttable[0:5]['FEAT_AREA_DEG2'].quantity.to('arcmin2'))


I am working on the file "07-Textual-Similarity/01-Textual-Similarity.ipynb".
The first task is:
Another measure of two vectors, more common for text analysis, is called cosine similarity. According to Wikipedia:

Cosine similarity is a measure of similarity between two non-zero vectors of an inner product space that measures the cosine of the angle between them. The cosine of 0° is 1, and it is less than 1 for any other angle. It is thus a judgment of orientation and not magnitude: two vectors with the same orientation have a cosine similarity of 1, two vectors at 90° have a similarity of 0, and two vectors diametrically opposed have a similarity of -1, independent of their magnitude.
$\text{similarity} = \cos(\theta) = {\mathbf{A} \cdot \mathbf{B} \over \|\mathbf{A}\|2 \|\mathbf{B}\|_2} = \frac{ \sum\limits{i=1}^{n}{A_i  B_i} }{ \sqrt{\sum\limits_{i=1}^{n}{A_i^2}}  \sqrt{\sum\limits_{i=1}^{n}{B_i^2}} }$

Essentially we want to take the cosine of the angle formed between two vectors (documents). We start the vector at the origin and measure the angle between the two vectors we're interested in.
Can you write Python code for it?

mpl.rcParams['legend.fontsize'] = 10

origin = (0,0,0)

fig = plt.figure()
ax = fig.gca(projection='3d')
ax.scatter([a[0], b[0], origin[0]], [a[1], b[1], origin[1]], [a[2], b[2], origin[2]])
ax.plot([origin[0], a[0]], [origin[1], a[1]], [origin[2], a[2]])
ax.plot([origin[0], b[0]], [origin[1], b[1]], [origin[2], b[2]])
plt.show()


I am working on the file "07-Textual-Similarity/01-Textual-Similarity.ipynb".
The first task is:
Bag of Words (BoW) language model
Today we'll see our first, admittedly primitive, computational model of language called "Bag of Words". This model was very popular in early text analysis, and continues to be used today. In fact, the models that have replaced it are still very difficult to actually interpret, giving the BoW approach a slight advantage if we want to understand why the model makes certain decisions.
Getting into the model we'll have to revisit Term Frequency (think Counter). We'll then see the Document-Term Matrix (DTM), which we've discusssed briefly before. We'll have to normalize these counts if we want to compare. Then we'll look at the available Python libraries to streamline this process.
Once we have our BoW model we can analyze it in a high-dimensional vector space, which gives us more insights into the similarities and clustering of different texts. We'll then get into Piper's analysis.
We'll build our model  from scratch with numpy and datascience libraries before we get into the higher level libraries:
Can you write Python code for it?

%matplotlib inline
import numpy as np
from datascience import *


I am working on the file "07-Textual-Similarity/01-Textual-Similarity.ipynb".
The first task is:
Challenge
Try passing different values into both the euclidean and cosine distance functions. What is your intuition about these different measurements? Remember that all values in the Term-Frequency Matrix are positive, between [0,1], and that most are very small.
Visualizing Texts in Vector Space
Let's walk through this now. Say we have 3 texts, a, b, and c. The whole corpus, again, only has 2 words (dimensions)!
Can you write Python code for it?

a = (2,6)
b = (5,10)
c = (14,11)

print(distance.euclidean(a,b))
print(distance.euclidean(a,c))
print(distance.euclidean(b,c))


I am working on the file "07-Textual-Similarity/01-Textual-Similarity.ipynb".
The first task is:
Each vector has a number of coordinates equal to the number of unique words in the corpus. Let's just take Book 1:
Can you write Python code for it?


I am working on the file "07-Textual-Similarity/01-Textual-Similarity.ipynb".
The first task is:
For the 3-d model:
Can you write Python code for it?


I am working on the file "07-Textual-Similarity/01-Textual-Similarity.ipynb".
The first task is:
Homework
Try visualizing the textual similarities again using the Cosine distance. How does that change the result? Why?
Brief Aside: K-Means Clustering
Tries to find natural groupings among points, once we tell it how many groups to look for.
Can you write Python code for it?

from sklearn.cluster import KMeans

kmeans = KMeans(n_clusters=2)
kmeans.fit_predict(dist_matrix)


I am working on the file "07-Textual-Similarity/01-Textual-Similarity.ipynb".
The first task is:
I then
calculated the Euclidean distance between each of the twenty parts of the work based on
the frequency of the remaining words and stored those results in a symmetrical distance
table. In the end, for each work I had a 20x20 table of distances between every part of
a work to every other, in which the distances are considered to be measures of the similarity
of the language between a work’s individual parts. [95]

Piper then calculates the Euclidean distances between each part to every other part. So we'll have to calculate the distance and use our pairwise method. We can write a function for that too! To make it better, let's have it take in a list of texts that our text_splitter will output:
Can you write Python code for it?

def text_distances(text_list):

    from sklearn.feature_extraction.stop_words import ENGLISH_STOP_WORDS
    from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer
    from sklearn.metrics import pairwise

    ye_olde_stop_words = ['thou','thy','thee', 'thine', 'ye', 'hath','hast', 'wilt','aught',\
                          'art', 'dost','doth', 'shall', 'shalt','tis','canst','thyself',\
                         'didst', 'yea', 'wert']
    stop_words = list(ENGLISH_STOP_WORDS)+ye_olde_stop_words
    cv = CountVectorizer(stop_words = stop_words, min_df=0.6)
    dtm = cv.fit_transform(text_list)
    tt = TfidfTransformer(norm='l1',use_idf=False)
    dtm_tf = tt.fit_transform(dtm)
    dist_matrix = pairwise.pairwise_distances(dtm_tf, metric='euclidean')
    return(dist_matrix)


I am working on the file "07-Textual-Similarity/01-Textual-Similarity.ipynb".
The first task is:
If we want to see which novels stick out, we might be interested in the z-score for a particular novel. This is how many standard devations the novel is away from the mean. Let's write a function:
Can you write Python code for it?

def get_zscores(values):

    import numpy as np
    mn = np.mean(values)
    st = np.std(values)
    zs = []

    for x in values:
        z = (x-mn)/st
        zs.append(z)

    return zs


I am working on the file "07-Textual-Similarity/01-Textual-Similarity.ipynb".
The first task is:
Let's add these measures to our Table:
Can you write Python code for it?

metadata_tb['Cross-Half'] = measures[:,0]
metadata_tb['In-Half'] = measures[:,1]
metadata_tb.show(5)


I am working on the file "07-Textual-Similarity/01-Textual-Similarity.ipynb".
The first task is:
Let's go back to two dimensions for the vanilla numpy calculation:
Can you write Python code for it?

a = (2,6)
b = (5,10)

# don't worry about the formula so much as the intuition behind it: angle between vectors
cos_dist = 1 - (a[0]*b[0] + a[1]*b[1]) / ( np.sqrt(a[0]**2 + a[1]**2 ) * np.sqrt(b[0]**2 + b[1]**2 ) )
cos_dist


I am working on the file "07-Textual-Similarity/01-Textual-Similarity.ipynb".
The first task is:
Let's peek at the first:
Can you write Python code for it?


I am working on the file "07-Textual-Similarity/01-Textual-Similarity.ipynb".
The first task is:
Let's read in Augustine's Confessions text:
Can you write Python code for it?

with open('data/Augustine-Confessions.txt') as f:
    confessions = f.read()

confessions


I am working on the file "07-Textual-Similarity/01-Textual-Similarity.ipynb".
The first task is:
Looks good! Now we can read in the corpus Piper used:
Can you write Python code for it?

metadata_tb = Table.read_table('data/2_txtlab_Novel450.csv')
metadata_tb.show(5)


I am working on the file "07-Textual-Similarity/01-Textual-Similarity.ipynb".
The first task is:
Looks to be about 13 counts, one for each book, let's double check!
Can you write Python code for it?


I am working on the file "07-Textual-Similarity/01-Textual-Similarity.ipynb".
The first task is:
Normalization
Piper notes:

The words were thus normalized according to their relative importance within the work. [95]

Let's get the total number of occurences of each word in the whole text. The key to the code below is sum(desparse), which sums the column for all the books in our matrix:
Can you write Python code for it?

toks_tab = Table()
toks_tab.append_column(label="Word List", values=word_list)
toks_tab.append_column(label="Frequency", values=sum(desparse))  # this sum(desparse) will sum the word count column
toks_tab.show()


I am working on the file "07-Textual-Similarity/01-Textual-Similarity.ipynb".
The first task is:
Note: If you are processing a text that uses only contemporary English, it may be unnecessary to import the list of stopwords explicitly. Simply pass the value "english" into the "stop_words" argument in CountVectorizer.
Can you write Python code for it?


I am working on the file "07-Textual-Similarity/01-Textual-Similarity.ipynb".
The first task is:
Now we can add these to the Table too:
Can you write Python code for it?

metadata_tb['Cross-Z-Score'] = get_zscores(measures[:,0])
metadata_tb['In-Z-Score'] = get_zscores(measures[:,1])
metadata_tb.show(5)


I am working on the file "07-Textual-Similarity/01-Textual-Similarity.ipynb".
The first task is:
Of course, scipy has taken care of this for us too:
Can you write Python code for it?


I am working on the file "07-Textual-Similarity/01-Textual-Similarity.ipynb".
The first task is:
Piper the introduces two new ideas.

for the in-half distance I took the average distance of each part in the first half of a work to every other part in that half and subtracted it from the average distance of every part of the second half to itself. [95]

Let's write a function that does that, and have it take in our matrix returned by text_distances:
Can you write Python code for it?

def in_half_dist(matrix):
    n = len(matrix)  # length of work, should be 20
    d1 = []  # will hold distances for first half
    d2 = []  # will hold distances for second half
    for i in range(int(n/2)-1):  # loop through first half of work (10 in our case)
        for j in range(i+1, int(n/2)):  # loop through itself (first half again)
            d1.append(matrix[i,j])  # append distance between one part to another (in first half)
    for i in range(int(n/2), n-1):
        for j in range(i+1, n):
            d2.append(matrix[i,j])
    return(abs(sum(d1)-sum(d2))/len(d1))  # take average of each distance array and subtract 2 from 1


I am working on the file "07-Textual-Similarity/01-Textual-Similarity.ipynb".
The first task is:
Reading the matrix above, we see that the word "abandoned" accounts for .0145406% of words in Book 1, and .0277855% of words in Book 2.
We can still grab out the normalized frequencies of the word 'read' for each book:
Can you write Python code for it?


I am working on the file "07-Textual-Similarity/01-Textual-Similarity.ipynb".
The first task is:
Visualizing hundreds of dimensions is difficult for us. So we can use multi-dimensional scaling (MDS) to put this into a 2-d graph for us:
Can you write Python code for it?

from sklearn.manifold import MDS

mds = MDS(n_components = 2, dissimilarity="precomputed")
embeddings = mds.fit_transform(dist_matrix)

_, ax = plt.subplots(figsize=(10,10))
ax.scatter(embeddings[:,0], embeddings[:,1], alpha=0)
for i in range(13):
    ax.annotate(i+1, ((embeddings[i,0], embeddings[i,1])))


I am working on the file "07-Textual-Similarity/01-Textual-Similarity.ipynb".
The first task is:
We can think of this 2 dimensional distance between 2 points as looking at 2 different texts. In this very simple 2-d model though, we only have 2 words in the entire corpus! (2,6) and (5,10) would be the absolute counts for each text. Imagine:
```
Document 1:
the dog the dog dog dog dog dog
Document 2:
the dog the dog the dog the dog the dog dog dog dog dog dog
```
That would yield the comparison above. If we added a third point (document), we could see which 2 documents were closest to one another!

Ok, not too bad, but how do we do this with hundreds or thousands of dimensions (words) acorss hundreds or thousands of points (documents)? Well it actually scales the same way! Here it is for 3 dimensions:
$\mathrm{d}(\mathbf{b},\mathbf{a})=\sqrt{(a_1-b_1)^2 + (a_2-b_2)^2 + (a_3-b_3)^2}$
Can you write Python code for it?

a = (2,6,15)
b = (5,10,3)

euc_dist = np.sqrt( (a[0]-b[0])**2 +  (a[1]-b[1])**2 + (a[2]-b[2])**2 )
euc_dist

import matplotlib as mpl
from mpl_toolkits.mplot3d import Axes3D

mpl.rcParams['legend.fontsize'] = 10

fig = plt.figure()
ax = fig.gca(projection='3d')
ax.scatter([a[0], b[0]], [a[1], b[1]], [a[2], b[2]])
ax.plot([a[0], b[0]], [a[1], b[1]], [a[2], b[2]])
plt.show()


I am working on the file "07-Textual-Similarity/01-Textual-Similarity.ipynb".
The first task is:
We can visualize this too:
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline

plt.scatter([a[0], b[0]], [a[1], b[1]])
plt.plot([a[0], b[0]], [a[1], b[1]])
plt.show()


I am working on the file "07-Textual-Similarity/01-Textual-Similarity.ipynb".
The first task is:
We don't have to use our cool formula to calculate this, or to scale it up for n dimensions. That's what scipy is for:
Can you write Python code for it?

from scipy.spatial import distance
distance.euclidean(a,b)


I am working on the file "07-Textual-Similarity/01-Textual-Similarity.ipynb".
The first task is:
We'll make a matrix for the points:
Can you write Python code for it?

point_matrix = np.array([a,b,c])
point_matrix


I am working on the file "07-Textual-Similarity/01-Textual-Similarity.ipynb".
The first task is:
We'll slightly change our text_measures function so that it can read in the file of the text we want to read in, instead of taking the confessions string we already had:
Can you write Python code for it?

corpus_path = 'data/2_txtalb_Novel450/'

def text_measures_alt(text_name):
    with open(corpus_path+text_name, 'r') as file_in:
        text = file_in.read()
    text_list = text_splitter(text)
    dist_matrix = text_distances(text_list)
    return(cross_half_dist(dist_matrix), in_half_dist(dist_matrix))


I am working on the file "07-Textual-Similarity/01-Textual-Similarity.ipynb".
The first task is:
text_measures should now return two values. The first values is the cross_half_dist and the second values is the in_half_dist. Let's test this out on Augustine's `Confessions':
Can you write Python code for it?


I am working on the file "07-Word2Vec/Word2Vec.ipynb".
The first task is:
2. Word2Vec
Word Embedding
Word2Vec is the most prominent word embedding algorithm. Word embedding generally attempts to identify semantic relationships between words by observing them in context.
Imagine that each word in a novel has its meaning determined by the ones that surround it in a limited window. For example, in Moby Dick's first sentence, “me” is paired on either side by “Call” and “Ishmael.” After observing the windows around every word in the novel (or many novels), the computer will notice a pattern in which “me” falls between similar pairs of words to “her,” “him,” or “them.” Of course, the computer had gone through a similar process over the words “Call” and “Ishmael,” for which “me” is reciprocally part of their contexts.  This chaining of signifiers to one another mirrors some of humanists' most sophisticated interpretative frameworks of language.
The two main flavors of Word2Vec are CBOW (Continuous Bag of Words) and Skip-Gram, which can be distinguished partly by their input and output during training. Skip-Gram takes a word of interest as its input (e.g. "me") and tries to learn how to predict its context words ("Call","Ishmael"). CBOW does the opposite, taking the context words ("Call","Ishmael") as a single input and tries to predict the word of interest ("me").
In general, CBOW is is faster and does well with frequent words, while Skip-Gram potentially represents rare words better.
Word2Vec Features

Size: Number of dimensions for word embedding model

Window: Number of context words to observe in each direction

min_count: Minimum frequency for words included in model

sg (Skip-Gram): '0' indicates CBOW model; '1' indicates Skip-Gram

Alpha: Learning rate (initial); prevents model from over-correcting, enables finer tuning

Iterations: Number of passes through dataset

Batch Size: Number of words to sample from data during each pass

Note: Script uses default value for each argument
Training
Can you write Python code for it?

# Train word2vec model from txtLab corpus

model = gensim.models.Word2Vec(words_by_sentence, size=100, window=5, \
                               min_count=25, sg=1, alpha=0.025, iter=5, batch_words=10000)


I am working on the file "07-Word2Vec/Word2Vec.ipynb".
The first task is:
3. Saving/Loading Models
Can you write Python code for it?

# Save current model for later use

model.wv.save_word2vec_format('resources/word2vec.txtlab_Novel150_English.txt')
#model.save_word2vec_format('resources/word2vec.txtlab_Novel150_English.txt') # deprecated

# Load up models from disk

# Model trained on Eighteenth Century Collections Online corpus (~2500 texts)
# Made available by Ryan Heuser: http://ryanheuser.org/word-vectors-1/

ecco_model = gensim.models.KeyedVectors.load_word2vec_format('resources/word2vec.ECCO-TCP.txt')
#ecco_model = gensim.models.Word2Vec.load_word2vec_format('resources/word2vec.ECCO-TCP.txt') # deprecated

# What are similar words to BANK?

ecco_model.most_similar('bank')

# What if we remove the sense of "river bank"?

ecco_model.most_similar(positive=['bank'], negative=['river'])


I am working on the file "07-Word2Vec/Word2Vec.ipynb".
The first task is:
Create a DTM with a Few Pseudo-Texts
Can you write Python code for it?

# dataframes!
import pandas

# Construct dataframe
columns = ['eggs','sausage','bacon']
indices = ['Novel A', 'Novel B', 'Novel C']
dtm = [[50,60,60],[90,10,10], [20,70,70]]
dtm_df = pandas.DataFrame(dtm, columns = columns, index = indices)

# Show dataframe
dtm_df


I am working on the file "07-Word2Vec/Word2Vec.ipynb".
The first task is:
Embeddings
Can you write Python code for it?

# Return dense word vector

model['whale']


I am working on the file "07-Word2Vec/Word2Vec.ipynb".
The first task is:
Exercises
Can you write Python code for it?

## EX. Use the most_similar method to find the tokens nearest to 'car' in our model.
##     Do the same for 'motorcar'.

## Q.  What characterizes each word in our corpus? Does this make sense?

## EX. How does our model answer the analogy: MADRID is to SPAIN as PARIS is to __________

## Q.  What has our model learned about nation-states?

## EX. Perform the canonic Word2Vec addition again but leave out a term:
##     Try 'king' - 'man', 'woman' - 'man', 'woman' + 'king'

## Q.  What do these indicate semantically?


I am working on the file "07-Word2Vec/Word2Vec.ipynb".
The first task is:
Multiple Valences
A word embedding may encode both primary and secondary meanings that are both present at the same time. In order to identify secondary meanings in a word, we can subtract the vectors of primary (or simply unwanted) meanings. For example, we may wish to remove the sense of river bank from the word bank. This would be written mathetmatically as RIVER - BANK, which in gensim's interface lists RIVER as a positive meaning and BANK as a negative one.
Can you write Python code for it?

# Get most similar words to BANK, in order
# to get a sense for its primary meaning

model.most_similar('bank')

# Remove the sense of "river bank" from "bank" and see what is left

model.most_similar(positive=['bank'], negative=['river'])


I am working on the file "07-Word2Vec/Word2Vec.ipynb".
The first task is:
Pre-Processing
Word2Vec learns about the relationships among words by observing them in context. This means that we want to split our texts into word-units. However, we  want to maintain sentence boundaries as well, since the last word of the previous sentence might skew the meaning of the next sentence.
Since novels were imported as single strings, we'll first use <i>sent_tokenize</i> to divide them into sentences, and second, we'll split each sentence into its own list of words.
Can you write Python code for it?

# Split each novel into sentences

sentences = [sentence for novel in novel_list for sentence in sent_tokenize(novel)]

# Inspect first sentence

sentences[0]

# Split each sentence into tokens

words_by_sentence = [fast_tokenize(sentence) for sentence in sentences]

# Remove any sentences that contain zero tokens

words_by_sentence = [sentence for sentence in words_by_sentence if sentence != []]

# Inspect first sentence

words_by_sentence[0]


I am working on the file "07-Word2Vec/Word2Vec.ipynb".
The first task is:
Vector-Space Operations
Similarity
Since words are represented as dense vectors, we can ask how similiar words' meanings are based on their cosine similarity (essentially how much they overlap). gensim has a few dout-of-the-box functions that enable different kinds of comparisons.
Can you write Python code for it?

# Find cosine distance between two given word vectors

model.similarity('pride','prejudice')

# Find nearest word vectors by cosine distance

model.most_similar('pride')

# Given a list of words, we can ask which doesn't belong

# Finds mean vector of words in list
# and identifies the word further from that mean

model.doesnt_match(['pride','prejudice', 'whale'])


I am working on the file "07-Word2Vec/Word2Vec.ipynb".
The first task is:
Vectors
At a glance, a couple of points are lying closer to one another. We used the word frequencies of just two words in order to plot our texts in a two-dimensional plane. The term frequency "summaries" of Novel A & Novel C are pretty similar to one another: they both share a major concern with "eggs", whereas Novel B seems to focus primarily on "sausage."
This raises a question: how can we operationalize our intuition that spatial distance expresses topical similarity?

The most common measurement of distance between points is their Cosine Similarity. Imagine that we draw an arrow from the origin of the graph -- point (0,0) -- to the dot representing each text. This arrow is called a vector. The Cosine Similarity between two vectors measures how much they overlap with one another. Values for the cosine similarity between two vectors fall between 0 and 1, so they are easily interpreted and compared.
Cosine Distance
Can you write Python code for it?

# Although we want the Cosine Distance, it is mathematically
# simpler to calculate its opposite: Cosine Similarity

from sklearn.metrics.pairwise import cosine_similarity

# So we will subtract the similarities from 1

cos_sim = cosine_similarity(dtm_df)

# And we'll make it a little easier to read

np.round(cos_sim, 2)


I am working on the file "07-Word2Vec/Word2Vec.ipynb".
The first task is:
Visualization
Can you write Python code for it?

# Dictionary of words in model

model.wv.vocab
#model.vocab # deprecated

# Visualizing the whole vocabulary would make it hard to read

len(model.wv.vocab)
#len(model.vocab) # deprecated

# For interpretability, we'll select words that already have a semantic relation

her_tokens = [token for token,weight in model.most_similar(positive=['she','her','hers','herself'], \
                                                       negative=['he','him','his','himself'], topn=50)]

# Inspect list

her_tokens

# Get the vector for each sampled word

vectors = [model[word] for word in her_tokens]

# Calculate distances among texts in vector space

dist_matrix = pairwise.pairwise_distances(vectors, metric='cosine')

# Multi-Dimensional Scaling (Project vectors into 2-D)

mds = MDS(n_components = 2, dissimilarity='precomputed')
embeddings = mds.fit_transform(dist_matrix)

# Make a pretty graph

_, ax = plt.subplots(figsize=(10,10))
ax.scatter(embeddings[:,0], embeddings[:,1], alpha=0)
for i in range(len(vectors)):
    ax.annotate(her_tokens[i], ((embeddings[i,0], embeddings[i,1])))

# For comparison, here is the same graph using a masculine-pronoun vector

his_tokens = [token for token,weight in model.most_similar(positive=['he','him','his','himself'], \
                                                       negative=['she','her','hers','herself'], topn=50)]
vectors = [model[word] for word in his_tokens]
dist_matrix = pairwise.pairwise_distances(vectors, metric='cosine')
mds = MDS(n_components = 2, dissimilarity='precomputed')
embeddings = mds.fit_transform(dist_matrix)
_, ax = plt.subplots(figsize=(10,10))
ax.scatter(embeddings[:,0], embeddings[:,1], alpha=0)
for i in range(len(vectors)):
    ax.annotate(his_tokens[i], ((embeddings[i,0], embeddings[i,1])))

## Q. What kinds of semantic relationships exist in the diagram above?
##    Are there any words that seem out of place?


I am working on the file "07-Word2Vec/Word2Vec.ipynb".
The first task is:
Visualize
Can you write Python code for it?

# Plot our points
scatter(dtm_df['eggs'], dtm_df['sausage'])

# Make the graph look good
xlim([0,100]), ylim([0,100])
xlabel('eggs'), ylabel('sausage')


I am working on the file "07-case_studies.ipynb".
The first task is:
Dados reais (podemos tentar com dados dos alunos)
Can you write Python code for it?

%%file data.csv
Food,England,Wales,Scotland,N Ireland
Cheese,105,103,103,66
Carcass meat,245,227,242,267
Other meat,685,803,750,586
Fish,147,160,122,93
Fats and oils,193,235,184,209
Sugars,156,175,147,139
Fresh potatoes,720,874,566,1033
Fresh Veg,253,265,171,143
Other Veg,488,570,418,355
Processed potatoes,198,203,220,187
Processed Veg,360,365,337,334
Fresh fruit,1102,1137,957,674
Cereals,1472,1582,1462,1494
Beverages,57,73,53,47
Soft drinks,1374,1256,1572,1506
Alcoholic drinks,375,475,458,135
Confectionery,54,64,62,41

from pandas import read_csv

df = read_csv('data.csv', index_col='Food')
df

def z_score(x):
    """Remove a média e normaliza os pelo desvio padrão"""
    return (x - x.mean()) / x.std()

from sklearn.decomposition import PCA

pca = PCA(n_components=None)
pca.fit(df.apply(z_score).T)

from pandas import DataFrame

loadings = DataFrame(pca.components_.T)
loadings.index = ['PC %s' % pc for pc in loadings.index + 1]
loadings.columns = ['TS %s' % pc for pc in loadings.columns + 1]
loadings

PCs = np.dot(loadings.values.T, df)

marker = dict(linestyle='none', marker='o', markersize=7, color='blue', alpha=0.5)

fig, ax = plt.subplots(figsize=(7, 2.75))
ax.plot(PCs[0], np.zeros_like(PCs[0]),
        label="Scores", **marker)
[ax.text(x, y, t) for x, y, t in zip(PCs[0], loadings.values[1, :], df.columns)]

ax.set_xlabel("PC1")

_ = ax.set_ylim(-1, 1)
marker = dict(linestyle='none', marker='o', markersize=7, color='blue', alpha=0.5)

fig, ax = plt.subplots(figsize=(7, 2.75))
ax.plot(PCs[0], PCs[1], label="Scores", **marker)

ax.set_xlabel("PC1")
ax.set_ylabel("PC2")

text = [ax.text(x, y, t) for x, y, t in
        zip(PCs[0], PCs[1]+0.5, df.columns)]

perc = pca.explained_variance_ratio_ * 100

perc = DataFrame(perc, columns=['Percentage explained ratio'], index=['PC %s' % pc for pc in np.arange(len(perc)) + 1])
ax = perc.plot(kind='bar')

TS1 = loadings['TS 1']
TS1.index = df.index
ax = TS1.plot(kind='bar')

marker = dict(linestyle='none', marker='o', markersize=7, color='blue', alpha=0.5)

fig, ax...


I am working on the file "07-case_studies.ipynb".
The first task is:
Exemplo 3: Ler dados em NetCDF CF-1.6 de CTD
Can you write Python code for it?

import iris

url = './data/wodStandardLevels.nc'
sal = iris.load_cube(url, 'sea_water_salinity')

print(sal)

import cartopy.crs as ccrs
import matplotlib.pyplot as plt

from cartopy.feature import LAND, COASTLINE
from cartopy.mpl.gridliner import LONGITUDE_FORMATTER, LATITUDE_FORMATTER

x = sal.coord('longitude').points
y = sal.coord('latitude').points

fig, ax = plt.subplots(figsize=(11, 13),
                       subplot_kw=dict(projection=ccrs.PlateCarree()))
ax.set_extent([-180, 180, -90, 90])
ax.stock_img()
ax.plot(x, y, 'go')

ax.add_feature(COASTLINE)
kw = dict(linewidth=1.5, color='gray', alpha=0.5, linestyle='--')
gl = ax.gridlines(crs=ccrs.PlateCarree(), draw_labels=True, **kw)
gl.xlabels_top = gl.ylabels_right = False
gl.xformatter = LONGITUDE_FORMATTER
gl.yformatter = LATITUDE_FORMATTER

from shapely.geometry.polygon import LinearRing
lons = [105, 105, 125, 125]
lats = [-40, -20, -20, -40]
ring = LinearRing(list(zip(lons, lats)))
sq = ax.add_geometries([ring], ccrs.PlateCarree(), facecolor='none', edgecolor='red')

lon = iris.Constraint(longitude=lambda l:min(lons) < l < max(lons))
lat = iris.Constraint(latitude=lambda l:min(lats) < l < max(lats))
alt = iris.Constraint(altitude=lambda a: a <= 30)

sal_profile = sal.extract(alt & lon & lat)

print(sal_profile)

import iris.plot as iplt

lon = sal_profile.coord(axis='X').points.squeeze()
lat = sal_profile.coord(axis='Y').points.squeeze()
depth = sal_profile.coord(axis='Z').points.max()

fig, ax = plt.subplots(figsize=(5, 6))
kw = dict(linewidth=2,  color='k', marker='o')
iplt.plot(sal_profile, sal_profile.coord('altitude'), **kw)
ax.grid()
ax.set_ylabel('Depth (m)')
ax.set_xlabel('Salinity (unknown)')
t = ax.set_title('lon: %s\nlat: %s\nMax depth = %s' % (lon, lat, depth))


I am working on the file "07-case_studies.ipynb".
The first task is:
Exemplo 5: SymPy
Can you write Python code for it?

from sympy.interactive import printing
from sympy import Function, symbols, sqrt, S

printing.init_printing()
x, y = symbols("x y")

from IPython.display import Latex

red = S("(x / 7) ** 2 * sqrt(abs(abs(x) - 3) / (abs(x) - 3)) + (y / 3) ** 2 * sqrt(abs(y + 3 / 7 * sqrt(33)) / (y + 3 / 7 * sqrt(33))) - 1")

red

orange = S("abs(x / 2) - ((3 * sqrt(33) - 7) / 112) * x ** 2 -"
           "3 + sqrt(1 - (abs(abs(x) - 2) - 1) ** 2) - y")
orange

green = S("9 * sqrt(abs((abs(x) - 1) * (abs(x) - 0.75)) /"
          "((1 - abs(x)) * (abs(x) - 0.75))) - 8 * abs(x) - y")
green

blue = S("3 * abs(x) + 0.75 * sqrt(abs((abs(x) - 0.75) * (abs(x) - 0.5)) /"
         "((0.75 - abs(x)) * (abs(x) - 0.5))) - y")
blue

pink = S("2.25 * sqrt(abs((x - 0.5) * (x + 0.5)) / ((0.5 - x) * (0.5 + x))) - y")
pink

brown = S("6 * sqrt(10) / 7 + (1.5 - 0.5 * abs(x)) * sqrt(abs(abs(x) - 1) /"
          "(abs(x) - 1)) - (6 * sqrt(10) / 14) * sqrt(4 - (abs(x) - 1) ** 2) - y")
brown

from sympy import lambdify

def lamb(func):
    return lambdify((x, y), func, modules='numpy')

red, orange, green, blue, pink, brown = map(lamb, (red, orange, green, blue, pink, brown))

import numpy as np

spacing = 0.01
x, y = np.meshgrid(np.arange(-7.25, 7.25, spacing),
                   np.arange(-5, 5, spacing))

red = red(x, y)
orange = orange(x, y)
green = green(x, y)
blue = blue(x, y)
pink = pink(x, y)
brown = brown(x, y)

%matplotlib inline
import matplotlib.pyplot as plt

colors = dict(red='#FF0000', orange='#FFA500', green='#008000',
              blue='#003399', pink='#CC0033', brown='#800000')

equations = dict(red=red, orange=orange, green=green,
                 blue=blue, pink=pink, brown=brown)

fig, ax = plt.subplots(figsize=(8, 6))
for key, color in colors.items():
    ax.contour(x, y, equations[key], [0], colors=color, linewidths=3)

_ = ax.set_xticks([])
_ = ax.set_yticks([])


I am working on the file "07-case_studies.ipynb".
The first task is:
Exemplo 6: Subset de dados batimétricos
Can you write Python code for it?

import numpy as np

lon = np.loadtxt('./data/copano_bay/lon.txt')
lat = np.loadtxt('./data/copano_bay/lat.txt')
depth = np.loadtxt('./data/copano_bay/depth.txt')

lon.shape, lat.shape, depth.shape

from oceans import wrap_lon180

lon = wrap_lon180(lon)

%matplotlib inline

import matplotlib.pyplot as plt

import cartopy.crs as ccrs
from cartopy.io import shapereader
from cartopy.mpl.gridliner import LONGITUDE_FORMATTER, LATITUDE_FORMATTER

def make_map(projection=ccrs.PlateCarree()):
    fig, ax = plt.subplots(figsize=(9, 13),
                           subplot_kw=dict(projection=projection))
    gl = ax.gridlines(draw_labels=True)
    gl.xlabels_top = gl.ylabels_right = False
    gl.xformatter = LONGITUDE_FORMATTER
    gl.yformatter = LATITUDE_FORMATTER
    return fig, ax

import cartopy.io.img_tiles as cimgt

extent = [lon.min(), lon.max(), lat.min(), lat.max()]

request = cimgt.GoogleTiles()

fig, ax = make_map(projection=request.crs)
ax.set_extent(extent)
ax.add_image(request, 10)

%matplotlib tk

fig, ax = make_map()
ax.set_extent(extent)
cs = ax.pcolormesh(lon, lat, depth, cmap=plt.cm.ocean,
                   transform=ccrs.PlateCarree(), zorder=1)

poly = plt.ginput()

from matplotlib.path import Path
polygon = Path(poly)

def in_polygon(polygon, xp, yp, transform=None, radius=0.0):
    return polygon.contains_points(np.atleast_2d([xp, yp]).T,
                                   transform=None, radius=0.0)

x_poly = np.array(poly)[:, 0]
y_poly = np.array(poly)[:, 1]

xp = lon.ravel()
yp = lat.ravel()

inside = in_polygon(polygon, xp, yp)

mask = inside.reshape(lon.shape)

%matplotlib inline

fig, ax = make_map()
ax.set_extent(extent)
cs = ax.pcolormesh(lon, lat, depth, cmap=plt.cm.ocean,
                   transform=ccrs.PlateCarree(), zorder=0)

ax.set_boundary(polygon, transform=ccrs.PlateCarree())

import numpy.ma as ma

masked_depth = ma.masked_array(depth, ~mask)

fig, ax = make_map()
ax.set_extent(extent)
cs = ax.pcolormesh(lon, lat, masked_depth, cmap=plt.cm.ocean,
                  ...


I am working on the file "07-case_studies.ipynb".
The first task is:
Exemplo 7: Geopandas e contorno da BTS
Can you write Python code for it?

%matplotlib inline

import matplotlib.pyplot as plt

import cartopy.crs as ccrs
from cartopy.mpl.gridliner import LONGITUDE_FORMATTER, LATITUDE_FORMATTER

def make_map(projection=ccrs.PlateCarree(), figsize=(11, 11)):
    fig, ax = plt.subplots(figsize=figsize,
                           subplot_kw=dict(projection=projection))
    gl = ax.gridlines(draw_labels=True)
    gl.xlabels_top = gl.ylabels_right = False
    gl.xformatter = LONGITUDE_FORMATTER
    gl.yformatter = LATITUDE_FORMATTER
    return fig, ax

import geopandas

basins = geopandas.read_file('./data/hidrografia_dupla_cut/')
basins = basins['geometry'].values.tolist()

from shapely import ops

basin = ops.cascaded_union(basins)

bra = geopandas.read_file('./data/BRA_adm2/')

salvador = bra[bra['NAME_2'] == 'Salvador'].geometry.values[0]
camacari = bra[bra['NAME_2'] == u'Camaçari'].geometry.values[0]

salvador = salvador.geoms[0]  # Throw the islands away.

projection = ccrs.PlateCarree()

fig, ax = make_map(projection=projection)

extent = [-39.15, -38.10, -13.30, -12.4]
ax.set_extent(extent)

ax.add_geometries(basin, projection, facecolor='0.85', edgecolor='black')
ax.add_geometries([salvador], projection, facecolor='0.65')
ax.add_geometries([camacari], projection, facecolor='0.65')

kw = dict(fontweight='bold', color='white', verticalalignment='top')

lon, lat = camacari.centroid.bounds[0], camacari.centroid.bounds[1]
ax.text(lon, lat, u'Camaçari', horizontalalignment='center', **kw)

lon, lat = salvador.centroid.bounds[0], salvador.centroid.bounds[1]
ax.text(lon, lat, 'Salvador', **kw);


I am working on the file "07-case_studies.ipynb".
The first task is:
Exemplo 8: Lendo, processando, e plotando dados de CTD
Can you write Python code for it?

from ctd import DataFrame

cast = DataFrame.from_cnv('./data/CTD_046.cnv.bz2', compression='bz2')

lon, lat = cast.longitude.mean(), cast.latitude.mean()

hours = cast['timeS'].max() / 60. / 60.

print("CTD cast info\nlon: {:.2f}\nlat: "
      "{:.2f}\nduration: {:.2f} hours".format(lon, lat, hours))

cast = cast[cast['pumps']]

downcast, upcast = cast.split()

downcast[['scan', 'c0S/m', 't090C', 'timeS', 'altM', 'longitude', 'latitude']].tail()

downcast[['t090C', 't190C', 'c0S/m', 'c1S/m']].describe()

downcast.columns

keep = set(['t090C', 'c0S/m', 'sbeox0Mm/Kg', 'dz/dtM'])

downcast = downcast.drop(keep.symmetric_difference(cast.columns), axis=1)

downcast.head()


I am working on the file "07-case_studies.ipynb".
The first task is:
Loop Edit:
Can you write Python code for it?

downcast = downcast.press_check()  # Remove pressure reversals.
downcast = downcast[downcast['dz/dtM'] >= 0.25]  # Threshold velocity.


I am working on the file "07-case_studies.ipynb".
The first task is:
Quanto da variância é explicada por cada componente.
Can you write Python code for it?

pca.explained_variance_ratio_

from scipy.stats.stats import pearsonr

r, p = pearsonr(x-x.mean(), y-y.mean())

print('Pearson r: {}\nPC 1: {}:'.format(r, pc[0]))

import statsmodels.api as sm

def OLSreg(y, Xmat):
    return sm.OLS(y, sm.add_constant(Xmat, prepend=True)).fit()

scale = lambda x: (x - x.mean()) / x.std()

ols_fit = OLSreg(Z[:, 1], Z[:, 0])

fig, ax = plt.subplots(figsize=(5, 5))
ax.plot(Z[:, 0], Z[:, 1], **marker)
ax.set_xlim([-2, 2])
ax.set_ylim([-2, 2])
ax.axhline(**line)
ax.axvline(**line)
ax.set_xlabel('x')
ax.set_xlabel('y')

ax.plot(Z[:, 0], ols_fit.fittedvalues, 'r', alpha=0.5)
ax.text(-1, 1, r'R$^2$ = %4.3f' % round(ols_fit.rsquared, 3))

a1 = ax.arrow(0, 0, eigenvalues[1], pc[1] * eigenvalues[1],
              **arrowprops)
a2 = ax.arrow(0, 0, -eigenvalues[1], -pc[1] * eigenvalues[1],
              **arrowprops)


I am working on the file "07-case_studies.ipynb".
The first task is:
Smooth velocity and Oxygen with a 2 seconds window:
Can you write Python code for it?

from ctd import movingaverage

cast['dz/dtM'] = movingaverage(cast['dz/dtM'], window_size=48)
cast['sbeox0Mm/Kg'] = movingaverage(cast['sbeox0Mm/Kg'], window_size=48)


I am working on the file "07-case_studies.ipynb".
The first task is:
Smooth:
Can you write Python code for it?

pmax = max(cast.index)
if pmax >= 500:
    window_len = 21
elif pmax >= 100:
    window_len = 11
else:
    window_len = 5
kw = dict(window_len=window_len, window='hanning')
downcast = downcast.apply(Series.smooth, **kw)

import gsw
from oceans.sw_extras import gamma_GP_from_SP_pt

p = downcast.index.values.astype(float)

downcast.index.name = "Pressure"
downcast['depth'] = -gsw.z_from_p(p, lat)
downcast['SP'] = gsw.SP_from_C(downcast['c0S/m'].values * 10, downcast['t090C'].values, p)
downcast['SA'] = gsw.SA_from_SP(downcast['SP'].values, p, lon, lat)
downcast['CT'] = gsw.CT_from_t(downcast['SA'].values, downcast['t090C'].values, p)
downcast['sigma0_CT'] = gsw.sigma0_CT_exact(downcast['SA'].values, downcast['CT'].values)
downcast['sound'] = gsw.sound_speed_CT_exact(downcast['SA'].values, downcast['CT'].values, p)
downcast['gamma'] = gamma_GP_from_SP_pt(downcast['SA'].values, downcast['CT'].values, p, lon, lat)

# Does't fit the DataFrame because the index in at the middle (p_mid) point of the original data.
N2, p_mid = gsw.Nsquared(downcast['SA'].values, downcast['CT'].values, p, lat=lat)
N2 = DataFrame(N2, index=p_mid, columns=['N2'])

from matplotlib import style
style.use('ggplot')

import matplotlib.pyplot as plt
import mpl_toolkits.axisartist as AA
from mpl_toolkits.axes_grid1 import host_subplot

def adjust_xlim(ax, offset):
    x1, x2 = ax.get_xlim()
    ax.set_xlim(x1 - offset, x2 + offset)

def plot_vars(self, title, **kw):
    fig = plt.figure(figsize=(6, 9))
    fig.subplots_adjust(bottom=0.1, top=0.85)
    if title:
        fig.suptitle(title)
    ax0 = host_subplot(111, axes_class=AA.Axes)
    ax0.invert_yaxis()

    ax1, ax2, ax3 = ax0.twiny(), ax0.twiny(), ax0.twiny()

    new_axis0 = ax0.get_grid_helper().new_fixed_axis
    new_axis1 = ax1.get_grid_helper().new_fixed_axis
    new_axis2 = ax2.get_grid_helper().new_fixed_axis
    new_axis3 = ax3.get_grid_helper().new_fixed_axis

    ax0.axis["bottom"] = new_axis0(loc="bottom", axes=ax0, offset=(0, 0))
    ax1.axis["bottom"] =...


I am working on the file "07-case_studies.ipynb".
The first task is:
Wild Edit:
Can you write Python code for it?

from ctd import Series

kw = dict(n1=2, n2=20, block=150)
downcast = downcast.apply(Series.despike, **kw)


I am working on the file "07-impuestos.ipynb".
The first task is:
Ejemplo.-- Considere un flujo de caja de $ 100, y una duración de 10 períodos. Calcule el impuesto de renta si la tasa es del 30% para los períodos 1 a 5 y del 35% para los períodos restantes.
Can you write Python code for it?

cflo     = cf.cashflow(const_value=[100]*10, start=2000)
tax_rate = cf.interest_rate(const_value=[25]*10, start=2000, chgpts={5:35})
x        = cf.after_tax_cashflow(cflo,              # flujo de efectivo
                                 tax_rate=tax_rate) # impuesto de renta
cf.textplot(x)


I am working on the file "07-impuestos.ipynb".
The first task is:
Ejemplo.-- En el año 0 se compra un terreno por $ 500 para la construcción de una central térmica. Si el avalúo para efectos del cálculo del impuesto predíal es del 80% del valor de compra y el impuesto es del 0.3% del avalúo, construya el flujo de efectivo que representa el pago del impuesto predial para los siguientes 10 años.
Can you write Python code for it?

## costo de la inversión, se hace al final del año 0
avaluo = cf.cashflow([0.8 * 500] + [0] * 10, start = 2000)
avaluo

## valor en libros
bookval = avaluo.cumsum()
bookval

## tasa de impuesto predial
trate = cf.interest_rate([0] + [3] * 10, start=2000)
trate

## impuesto predial
cf.after_tax_cashflow(bookval, tax_rate=trate)


I am working on the file "07-notebook-after-class.ipynb".
The first task is:
Aufgabe 2     CFG: Kasus
Als nächstes sollen Kasusbedingungen in die Grammatik integriert werden:

Es gibt nur eine Nominalphrase im Nominativ (Subjekt).
Je nach Valenzstellen des Verbes sollen nur Nominalphrasen in den korrekten Kasus akzeptiert werden.
Optional Versuchen Sie die freie Satzstellung des Deutschen zu berücksichtigen.
Can you write Python code for it?

grammar = """
S -> NP[KAS=nom] VP

NP[KAS=?y] -> DET[GEN=?x,KAS=?y] NOM[GEN=?x]

NOM[GEN=?x] -> ADJ NOM[GEN=?x] | N[GEN=?x]

ADJ -> "schöne" | "kluge" | "dicke"

DET[GEN=mask,KAS=nom] -> "der"
DET[GEN=fem,KAS=dat] -> "der"

DET[GEN=fem,KAS=nom] -> "die"
DET[GEN=fem,KAS=akk] -> "die"
DET[GEN=neut,KAS=nom] -> "das"
DET[GEN=neut,KAS=akk] -> "das"

N[GEN=mask] -> "Mann"
N[GEN=fem] -> "Frau"
N[GEN=neut] -> "Buch"

VP -> V[SUBCAT=ditr, VAL1=?x, VAL2=?y] NP[KAS=?x] NP[KAS=?y]
VP -> V[VAL=?x,SUBCAT=tr] NP[KAS=?x]
VP -> V[SUBCAT=intr]

V[SUBCAT=ditr, VAL1=dat, VAL2=akk] -> "gibt" | "schenkt"
V[SUBCAT=intr] -> "schläft"
V[SUBCAT=tr,VAL=dat] -> "gefällt"
V[SUBCAT=tr,VAL=akk] -> "kennt"
"""

pos_sentences.extend([
    "das Buch gefällt der Frau",
    "das Buch kennt die Frau"
])

neg_sentences.extend([
    "der Mann schläft das Buch",
    "die Frau gefällt das Buch",
    "das Buch kennt",
    "die Frau gibt das Buch",
    "die Frau gibt die Frau das Buch"
])

test_grammar(grammar, pos_sentences)

test_grammar(grammar, neg_sentences)


I am working on the file "07-notebook-after-class.ipynb".
The first task is:
Hausaufgaben
Aufgabe 7     Plural für das Subjekt
Ergänzen Sie die in den Präsenzaufgaben erstellte Grammatik um die Möglichkeit, das Subjekt in den Plural zu setzen.
Dafür müssen Sie folgendes tun:
  1. Erstellen Sie lexikalische Regeln für Pluralformen der Verben, Adjektive und Substantive (Nominativ ist ausreichend.).
  1. Vervollständigen Sie die lexikalischen Regeln für die Form des Artikels die um die korrekte Merkmalstruktur für den Plural.
  1. Formulieren Sie eine Kongruenzbedingung in Numerus zwischen Verb und Subjekt.
Can you write Python code for it?

grammar = """

BITTE NACH BEARBEITUNG VON (2) VON OBEN KOPIEREN

"""

pos_sentences.extend([
    "die Männer geben der Frau das Buch",
    "die Bücher gefallen der Frau",
    "die Frauen schlafen"
])

neg_sentences.extend([
    "der Mann geben der Frau das Buch",
    "das Buch gefällt der Frauen",
    "die Frauen schläft"
])


I am working on the file "07-notebook-and-data/.ipynb_checkpoints/Homework7-checkpoint.ipynb".
The first task is:
Let's make a graph 'bout it
Can you write Python code for it?


I am working on the file "07-notebook-and-data/07 - Introduction to Pandas.ipynb".
The first task is:
A dataframe is basically a spreadsheet, except it lives in the world of Python or the statistical programming language R. They can't call it a spreadsheet because then people would think those programmers used Excel, which would make them boring and normal and they'd have to wear a tie every day.
Selecting rows
Now let's look at our data, since that's what data is for
Can you write Python code for it?


I am working on the file "07-notebook-and-data/07 - Introduction to Pandas.ipynb".
The first task is:
But maybe we want to make a basketball joke and see the final four?
Can you write Python code for it?


I am working on the file "07-notebook-and-data/07 - Introduction to Pandas.ipynb".
The first task is:
But sometimes instead of just looking at them, I want to do stuff with them. Play some games with them! Dunk on them~ describe them! And we don't want to dunk on everyone, only the players above 7 feet tall.
First, we need to check out boolean things.
Can you write Python code for it?

# Get a big long list of True and False for every single row.

# We could use value counts if we wanted

# But we can also apply this to every single row to say whether YES we want it or NO we don't

# Instead of putting column names inside of the brackets, we instead
# put the True/False statements. It will only return the players above
# seven feet tall

# Or only the guards

# Or only the guards who make more than 15 million

# It might be easier to break down the booleans into separate variables

# We can save this stuff

# Maybe we can compare them to taller players?


I am working on the file "07-notebook-and-data/07 - Introduction to Pandas.ipynb".
The first task is:
But that's ugly. There's a thing called ggplot for R that looks nice. We want to look nice. We want to look like ggplot.
Can you write Python code for it?

# Import matplotlib
# What's available?

# Use ggplot

# Make a histogram

# Try some other styles


I am working on the file "07-notebook-and-data/07 - Introduction to Pandas.ipynb".
The first task is:
If we scroll we can see all of it. But maybe we don't want to see all of it. Maybe we hate scrolling?
Can you write Python code for it?


I am working on the file "07-notebook-and-data/07 - Introduction to Pandas.ipynb".
The first task is:
It's kind of like an array, right? Except where in an array we'd say df[0] this time we need to give it two numbers, the start and the end.
Selecting columns
But jeez, my eyes don't want to go that far over the data. I only want to see, uh, name and age.
Can you write Python code for it?

# Get the names of the columns, just because

# If we want to be "correct" we add .values on the end of it

# Select only name and age

# Combing that with .head() to see not-so-many rows

# We can also do this all in one line, even though it starts looking ugly
# (unlike the cute bears pandas looks ugly pretty often)


I am working on the file "07-notebook-and-data/07 - Introduction to Pandas.ipynb".
The first task is:
Okay that was nice but unfortunately we can't do anything with it. It's just sitting there, separate from our data. If this were normal code we could do blahblah['feet'] = blahblah['Ht (In.)'] / 12, but since this is pandas, we can't. Right? Right?
Can you write Python code for it?


I am working on the file "07-notebook-and-data/07 - Introduction to Pandas.ipynb".
The first task is:
That might look better with a little more customization. So let's customize it.
Can you write Python code for it?

# Pass in all sorts of stuff!
# Most from http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.hist.html
# .range() is a matplotlib thing


I am working on the file "07-notebook-and-data/07 - Introduction to Pandas.ipynb".
The first task is:
That's cool, maybe we could do the same thing with their salary? Take out the $ and the , and convert it to an integer?
Can you write Python code for it?

# Can't just use .replace

# Need to use this weird .str thing

# Can't just immediately replace the , either

# Need to use the .str thing before EVERY string method

# Describe still doesn't work.

# Let's convert it to an integer using .astype(int) before we describe it

# Maybe we can just make them millions?

# Unfortunately one is "n/a" which is going to break our code, so we can make n/a be 0

# Remove the .head() piece and save it back into the dataframe


I am working on the file "07-notebook-and-data/07 - Introduction to Pandas.ipynb".
The first task is:
That's stupid, though, what's an inch even look like? What's 80 inches? I don't have a clue. If only there were some wa to manipulate our data.
Manipulating data
Oh wait there is, HA HA HA.
Can you write Python code for it?

# Take another look at our inches, but only the first few

# Divide those inches by 12

# Let's divide ALL of them by 12

# Can we get statistics on those?

# Let's look at our original data again


I am working on the file "07-notebook-and-data/07 - Introduction to Pandas.ipynb".
The first task is:
The average basketball player makes 3.8 million dollars and is a little over six and a half feet tall.
But who cares about those guys? I don't care about those guys. They're boring. I want the real rich guys!
Sorting and sub-selecting
Can you write Python code for it?

# This is just the first few guys in the dataset. Can we order it?

# Let's try to sort them


I am working on the file "07-notebook-and-data/07 - Introduction to Pandas.ipynb".
The first task is:
Those guys are making nothing! If only there were a way to sort from high to low, a.k.a. descending instead of ascending.
Can you write Python code for it?

# It isn't descending = True, unfortunately

# We can use this to find the oldest guys in the league

# Or the youngest, by taking out 'ascending=False'


I am working on the file "07-notebook-and-data/07 - Introduction to Pandas.ipynb".
The first task is:
Unfortunately because that has dollar signs and commas it's thought of as a string. We'll fix it in a second, but let's try describing one more thing.
Can you write Python code for it?


I am working on the file "07-notebook-and-data/07 - Introduction to Pandas.ipynb".
The first task is:
When you import pandas, you use import pandas as pd. That means instead of typing pandas in your code you'll type pd.
You don't have to, but every other person on the planet will be doing it, so you might as well.
Now we're going to read in a file. Our file is called NBA-Census-10.14.2013.csv because we're sports moguls. pandas can read_ different types of files, so try to figure it out by typing pd.read_ and hitting tab for autocomplete.
Can you write Python code for it?

# We're going to call this df, which means "data frame"
# It isn't in UTF-8 (I saved it from my mac!) so we need to set the encoding


I am working on the file "07-notebook-and-data/Homework7.ipynb".
The first task is:
Some examples from the review
Can you write Python code for it?

df.columns.values
df['countrycode'].value_counts()

df.sort_values(by='networthusbillion', ascending=False).head(10)

#Who is the poorest billionaire? Top ten poorest?
df.sort_values(by='rank', ascending=False).head(2)

df[df['networthusbillion'] == 1]

df['networthusbillion'].describe()

df.groupby("gender")["networthusbillion"].describe()

#Adding the wealt of all the billionaires in a given country.
df.groupby('countrycode')['networthusbillion'].sum().sort_values(ascending=False)

#What are the most common industries for billionaires to come from?
df['industry'].value_counts()

df.groupby('industry')['networthusbillion'].sum()

young_bills = df[df['age'] < 40]
young_bills.plot(kind='barh', x='name', y='networthusbillion').sort_values()

import pandas as pd
import matplotlib.pplot as plt
plt.style.use("ggplot")
%matplot


I am working on the file "07-notebook-solution.ipynb".
The first task is:
Hausaufgaben
Aufgabe 7     Plural für das Subjekt
Ergänzen Sie die in den Präsenzaufgaben erstellte Grammatik um die Möglichkeit, das Subjekt in den Plural zu setzen.
Dafür müssen Sie folgendes tun:
  1. Erstellen Sie lexikalische Regeln für Pluralformen der Verben, Adjektive und Substantive (Nominativ ist ausreichend.).
  1. Vervollständigen Sie die lexikalischen Regeln für die Form des Artikels die um die korrekte Merkmalstruktur für den Plural.
  1. Formulieren Sie eine Kongruenzbedingung in Numerus zwischen Verb und Subjekt.
Can you write Python code for it?

grammar = """
S -> NP[KAS=nom,NUM=?x] VP[NUM=?x]

NP[KAS=?y,NUM=?z] -> DET[GEN=?x,KAS=?y,NUM=?z] NOM[GEN=?x,NUM=?z]

NOM[GEN=?x,NUM=?z] -> ADJ[NUM=?z] NOM[GEN=?x,NUM=?z] | N[GEN=?x,NUM=?z]

ADJ[NUM=sg] -> "schöne" | "kluge" | "dicke"
ADJ[NUM=pl] -> "schönen" | "klugen" | "dicken"

DET[GEN=mask,KAS=nom,NUM=sg] -> "der"
DET[GEN=fem,KAS=dat,NUM=sg] -> "der"

DET[GEN=fem,KAS=nom,NUM=sg] -> "die"
DET[GEN=fem,KAS=akk,NUM=sg] -> "die"
DET[GEN=neut,KAS=nom,NUM=sg] -> "das"
DET[GEN=neut,KAS=akk,NUM=sg] -> "das"

DET[KAS=nom,NUM=pl] -> "die"
DET[KAS=akk,NUM=pl] -> "die"

N[GEN=mask,NUM=sg] -> "Mann"
N[GEN=mask,NUM=pl] -> "Männer"
N[GEN=fem,NUM=sg] -> "Frau"
N[GEN=fem,NUM=pl] -> "Frauen"
N[GEN=neut,NUM=sg] -> "Buch"
N[GEN=neut,NUM=pl] -> "Bücher"

VP[NUM=?z] -> V[SUBCAT=ditr, VAL1=?x, VAL2=?y, NUM=?z] NP[KAS=?x] NP[KAS=?y]
VP[NUM=?z] -> V[VAL=?x,SUBCAT=tr, NUM=?z] NP[KAS=?x]
VP[NUM=?z] -> V[SUBCAT=intr, NUM=?z]

V[SUBCAT=ditr, VAL1=dat, VAL2=akk,NUM=sg] -> "gibt" | "schenkt"
V[SUBCAT=ditr, VAL1=dat, VAL2=akk,NUM=pl] -> "geben" | "schenken"
V[SUBCAT=intr,NUM=sg] -> "schläft"
V[SUBCAT=intr,NUM=pl] -> "schlafen"
V[SUBCAT=tr,VAL=dat,NUM=sg] -> "gefällt"
V[SUBCAT=tr,VAL=dat,NUM=pl] -> "gefallen"
V[SUBCAT=tr,VAL=akk,NUM=sg] -> "kennt"
V[SUBCAT=tr,VAL=akk,NUM=pl] -> "kennen"
"""

pos_sentences.extend([
    "die Männer geben der Frau das Buch",
    "die Bücher gefallen der Frau",
    "die Frauen schlafen"
])

neg_sentences.extend([
    "der Mann geben der Frau das Buch",
    "das Buch gefällt der Frauen",
    "die Frauen schläft"
])

test_grammar(grammar, pos_sentences)

test_grammar(grammar, neg_sentences)


I am working on the file "07-notebook.ipynb".
The first task is:
Aufgabe 2     CFG: Kasus
Als nächstes sollen Kasusbedingungen in die Grammatik integriert werden:

Es gibt nur eine Nominalphrase im Nominativ (Subjekt).
Je nach Valenzstellen des Verbes sollen nur Nominalphrasen in den korrekten Kasus akzeptiert werden.
Optional Versuchen Sie die freie Satzstellung des Deutschen zu berücksichtigen.
Can you write Python code for it?

grammar = """

BITTE NACH BEARBEITUNG VON (1) VON OBEN KOPIEREN

"""

pos_sentences.extend([
    "das Buch gefällt der Frau",
    "das Buch kennt die Frau",
    "der Mann gibt der Frau das Buch"
])

neg_sentences.extend([
    "der Mann schläft das Buch",
    "die Frau gefällt das Buch",
    "das Buch kennt",
    "die Frau gibt das Buch"
])


I am working on the file "07-notebook.ipynb".
The first task is:
Aufgabe 8     Adverben und Verbzweitstellung
Fügen Sie der Grammatik jetzt die zwei Adverben heute und morgen hinzu. Adverben können prinzipiell sehr frei im Satz platziert werden. Eine Besonderheit des Deutschen ist aber die sog. Verbzweitstellung, wie sie z. B. in Sätzen wie Heute schläft der Mann. deutlich wird.
Versuchen Sie alle Möglichkeiten zu implementieren:
Can you write Python code for it?

pos_sentences.extend([
  "heute gibt der Mann der Frau das Buch",
  "der Mann gibt heute der Frau das Buch",
  "der Mann gibt der Frau heute das Buch",
  "der Mann gibt der Frau das Buch heute"
])

neg_sentences.extend([
    "heute der Mann gibt der Frau das Buch"
])


I am working on the file "07-notebook.ipynb".
The first task is:
Hausaufgaben
Aufgabe 7     Plural für das Subjekt
Ergänzen Sie die in den Präsenzaufgaben erstellte Grammatik um die Möglichkeit, das Subjekt in den Plural zu setzen.
Dafür müssen Sie folgendes tun:
  1. Erstellen Sie lexikalische Regeln für Pluralformen der Verben, Adjektive und Substantive (Nominativ ist ausreichend.).
  1. Vervollständigen Sie die lexikalischen Regeln für die Form des Artikels die um die korrekte Merkmalstruktur für den Plural.
  1. Formulieren Sie eine Kongruenzbedingung in Numerus zwischen Verb und Subjekt.
Can you write Python code for it?


I am working on the file "07-pandas DataFrame-03.ipynb".
The first task is:
DataFrame のグラフ描画機能を使用する例です。
3回分のランダムウォークのデータを並べた DataFrame を作成します。
Can you write Python code for it?

result = DataFrame()
for c in range(3):
    y = 0
    t = []
    for delta in np.random.normal(loc=0.0, scale=1.0, size=100):
        y += delta
        t.append(y)
    result['Trial %d' % c] = t

result.head()


I am working on the file "07-pandas DataFrame-03.ipynb".
The first task is:
DataFrame を array に変換する例です。
Can you write Python code for it?

data = {'City': ['Tokyo','Osaka','Nagoya','Okinawa'],
                 'Temperature': [25.0,28.2,27.3,30.9],
                 'Humidity': [44,42,np.nan,62]}
cities = DataFrame(data)
cities

cities.as_matrix()


I am working on the file "07-pandas DataFrame-03.ipynb".
The first task is:
Series を array に変換する例です。
Can you write Python code for it?


I am working on the file "07-pandas DataFrame-03.ipynb".
The first task is:
permutation 関数で、index の順番をランダムにシャッフルします。
Can you write Python code for it?


I am working on the file "07-pandas DataFrame-03.ipynb".
The first task is:
ランダムにシャッフルした index を用いて行を並べ替えます。
Can you write Python code for it?

deck = deck.reindex(np.random.permutation(deck.index))
deck.head()


I am working on the file "07-pandas DataFrame-03.ipynb".
The first task is:
次は、1000円の所持金において、100円を賭けた場合の結果を示します。
Can you write Python code for it?

money = 1000
money = coin_game(money, 100)
money


I am working on the file "07-pandas DataFrame-03.ipynb".
The first task is:
練習問題
次の関数 coin_game は、所持金と掛け金を引数に渡すと、1/2の確率で掛け金の分だけ所持金が増減した値が返ります。
Can you write Python code for it?

from numpy.random import randint
def coin_game(money, bet):
    coin = randint(2)
    if coin == 0:
        money += bet
    else:
        money -= bet
    return money


I am working on the file "07-pca-feature-selection.ipynb".
The first task is:
Question: Consider there is a feature with a score far better than other features. Is this feature a good choice?
Unsupervised Learning: Principal Component Analysis (PCA)
Principal component analysis is a dimensionality reduction algorithm that we can use to find structure in our data.The main aim is to find surface onto which projection errros are minimized.This surface is a lower the dimensional subspace spanned by principal components of the data. These principal components are the direction along which the projection of the data onto that axis hae the maximum variance.The main component along which the data varies is called the principal axis of variation.
Intuitive interpretation of PCA
At the first step we plot a data an with 2 dimenstion and try to reduce it to one

 Don't get confused
PCA is different from linear regression

How about If we have three-dimensional

Using PCA in Sklearn
Can you write Python code for it?

import sklearn
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split

import gzip, pickle, sys
f = gzip.open('Datasets/mnist.pkl.gz', 'rb')
(input_train, output_train), (input_test, output_test), _ = pickle.load(f, encoding='bytes')

for i in range(4):
    plt.subplot(2,2,i+1)
    plt.imshow(input_train[i].reshape((28,28)), cmap=plt.cm.gray_r, interpolation='nearest')
plt.show()

from sklearn.ensemble import RandomForestClassifier

randomforest = RandomForestClassifier(n_estimators=30)

randomforest.fit(input_train,output_train)

from sklearn.metrics import classification_report
print(classification_report(output_test, randomforest.predict(input_test)))

from sklearn.decomposition import PCA

pca = PCA(n_components=500)
pca.fit(input_train)

plt.figure(figsize=(12,6))
plt.plot(np.cumsum(pca.explained_variance_ratio_[0:500]),marker = 'o')
plt.show()

np.cumsum(pca.explained_variance_ratio_[0:500])[200]

pca = PCA(n_components= 200)
pca.fit(input_train)

x_train = pca.transform(input_train)
x_test = pca.transform(input_test)

for i in range(4):
    plt.subplot(2,2,i+1)
    plt.imshow(pca.inverse_transform(x_train)[i].reshape((28, 28)), cmap=plt.cm.gray_r, interpolation='nearest' )
plt.show()

randomforest = RandomForestClassifier(n_estimators=30)

randomforest.fit(x_train,output_train)

from sklearn.metrics import classification_report
print(classification_report(output_test, randomforest.predict(x_test)))


I am working on the file "07-prediction-serving.ipynb".
The first task is:
(Optional) Build the ML container image
This is the TFX runtime environment for the training pipeline steps.
Can you write Python code for it?

!echo $TFX_IMAGE_URI

!gcloud builds submit --tag $TFX_IMAGE_URI . --timeout=15m --machine-type=e2-highcpu-8


I am working on the file "07-prediction-serving.ipynb".
The first task is:
1. Making Online Predicitons
Can you write Python code for it?

vertex_ai.init(
    project=PROJECT,
    location=REGION,
    staging_bucket=BUCKET
)

endpoint_name = vertex_ai.Endpoint.list(
    filter=f'display_name={ENDPOINT_DISPLAY_NAME}',
    order_by="update_time")[-1].gca_resource.name

endpoint = vertex_ai.Endpoint(endpoint_name)

test_instances = [
    {
        "dropoff_grid": ["POINT(-87.6 41.9)"],
        "euclidean": [2064.2696],
        "loc_cross": [""],
        "payment_type": ["Credit Card"],
        "pickup_grid": ["POINT(-87.6 41.9)"],
        "trip_miles": [1.37],
        "trip_day": [12],
        "trip_hour": [16],
        "trip_month": [2],
        "trip_day_of_week": [4],
        "trip_seconds": [555]
    }
]

predictions = endpoint.predict(test_instances).predictions

for prediction in predictions:
    print(prediction)

explanations = endpoint.explain(test_instances).explanations

for explanation in explanations:
    print(explanation)


I am working on the file "07-prediction-serving.ipynb".
The first task is:
2. Batch Prediction
Can you write Python code for it?

WORKSPACE = f"gs://{BUCKET}/{DATASET_DISPLAY_NAME}/"
SERVING_DATA_DIR = os.path.join(WORKSPACE, 'serving_data')
SERVING_INPUT_DATA_DIR = os.path.join(SERVING_DATA_DIR, 'input_data')
SERVING_OUTPUT_DATA_DIR = os.path.join(SERVING_DATA_DIR, 'output_predictions')

if tf.io.gfile.exists(SERVING_DATA_DIR):
    print("Removing previous serving data...")
    tf.io.gfile.rmtree(SERVING_DATA_DIR)

print("Creating serving data directory...")
tf.io.gfile.mkdir(SERVING_DATA_DIR)
print("Serving data directory is ready.")


I am working on the file "07-prediction-serving.ipynb".
The first task is:
3. Run the batch prediction pipeline using Vertex Pipelines
Can you write Python code for it?

WORKSPACE = f"gs://{BUCKET}/{DATASET_DISPLAY_NAME}/"
ARTIFACT_STORE = os.path.join(WORKSPACE, 'tfx_artifacts')
PIPELINE_NAME = f'{MODEL_DISPLAY_NAME}-predict-pipeline'


I am working on the file "07-prediction-serving.ipynb".
The first task is:
Extract serving data to Cloud Storage as JSONL
Can you write Python code for it?

from src.common import datasource_utils
from src.preprocessing import etl

LIMIT = 10000

sql_query = datasource_utils.get_serving_source_query(
    bq_dataset_name=SERVE_BQ_DATASET_NAME,
    bq_table_name=SERVE_BQ_TABLE_NAME,
    limit=LIMIT
)

print(sql_query)

job_name = f"extract-{DATASET_DISPLAY_NAME}-serving-{datetime.now().strftime('%Y%m%d%H%M%S')}"

args = {
    'job_name': job_name,
    #'runner': 'DataflowRunner',
    'sql_query': sql_query,
    'exported_data_prefix': os.path.join(SERVING_INPUT_DATA_DIR, "data-"),
    'temporary_dir': os.path.join(WORKSPACE, 'tmp'),
    'gcs_location': os.path.join(WORKSPACE, 'bq_tmp'),
    'project': PROJECT,
    'region': REGION,
    'setup_file': './setup.py'
}

tf.get_logger().setLevel('ERROR')

print("Data extraction started...")
etl.run_extract_pipeline(args)
print("Data extraction completed.")

!gsutil ls {SERVING_INPUT_DATA_DIR}


I am working on the file "07-prediction-serving.ipynb".
The first task is:
Set configurations
Can you write Python code for it?

VERSION = 'v01'
DATASET_DISPLAY_NAME = 'chicago-taxi-tips'
MODEL_DISPLAY_NAME = f'{DATASET_DISPLAY_NAME}-classifier-{VERSION}'
ENDPOINT_DISPLAY_NAME = f'{DATASET_DISPLAY_NAME}-classifier'

SERVE_BQ_DATASET_NAME = 'playground_us' # Change to your serving BigQuery dataset name.
SERVE_BQ_TABLE_NAME = 'chicago_taxitrips_prep' # Change to your serving BigQuery table name.


I am working on the file "07-prediction-serving.ipynb".
The first task is:
Set the pipeline configurations for the Vertex AI run
Can you write Python code for it?

os.environ["PROJECT"] = PROJECT
os.environ["REGION"] = REGION
os.environ["GCS_LOCATION"] = f"gs://{BUCKET}/{DATASET_DISPLAY_NAME}"
os.environ["MODEL_DISPLAY_NAME"] = MODEL_DISPLAY_NAME
os.environ["PIPELINE_NAME"] = PIPELINE_NAME
os.environ["ARTIFACT_STORE_URI"] = ARTIFACT_STORE
os.environ["BATCH_PREDICTION_BQ_DATASET_NAME"] = SERVE_BQ_DATASET_NAME
os.environ["BATCH_PREDICTION_BQ_TABLE_NAME"] = SERVE_BQ_TABLE_NAME
os.environ["SERVE_LIMIT"] = "1000"
os.environ["BEAM_RUNNER"] = "DirectRunner"
os.environ["TFX_IMAGE_URI"] = f"gcr.io/{PROJECT}/{DATASET_DISPLAY_NAME}:{VERSION}"

import importlib
from src.tfx_pipelines import config
importlib.reload(config)

for key, value in config.__dict__.items():
    if key.isupper(): print(f'{key}: {value}')


I am working on the file "07-prediction-serving.ipynb".
The first task is:
Setup Google Cloud project
Can you write Python code for it?

PROJECT = '[your-project-id]' # Change to your project id.
REGION = 'us-central1' # Change to your region.
BUCKET = '[your-bucket-name]' # Change to your bucket name.

if PROJECT == "" or PROJECT is None or PROJECT == "[your-project-id]":
    # Get your GCP project id from gcloud
    shell_output = !gcloud config list --format 'value(core.project)' 2>/dev/null
    PROJECT = shell_output[0]

if BUCKET == "" or BUCKET is None or BUCKET == "[your-bucket-name]":
    # Get your bucket name to GCP project id
    BUCKET = PROJECT
    # Try to create the bucket if it doesn't exists
    ! gsutil mb -l $REGION gs://$BUCKET
    print("")

print("Project ID:", PROJECT)
print("Region:", REGION)
print("Bucket name:", BUCKET)


I am working on the file "07-prediction-serving.ipynb".
The first task is:
Submit run to Vertex Pipelines
Can you write Python code for it?

from kfp.v2.google.client import AIPlatformClient

pipeline_client = AIPlatformClient(
    project_id=PROJECT, region=REGION)

pipeline_client.create_run_from_job_spec(
    job_spec_path=pipeline_definition_file
)


I am working on the file "07-prediction-serving.ipynb".
The first task is:
Submit the batch prediction job
Can you write Python code for it?

model_name =  vertex_ai.Model.list(
    filter=f'display_name={MODEL_DISPLAY_NAME}',
    order_by="update_time")[-1].gca_resource.name

job_resources =  {
    "machine_type": 'n1-standard-2',
    #'accelerator_count': 1,
    #'accelerator_type': 'NVIDIA_TESLA_T4'
    "starting_replica_count": 1,
    "max_replica_count": 10,
}

job_display_name = f"{MODEL_DISPLAY_NAME}-prediction-job-{datetime.now().strftime('%Y%m%d%H%M%S')}"

vertex_ai.BatchPredictionJob.create(
    job_display_name=job_display_name,
    model_name=model_name,
    gcs_source=SERVING_INPUT_DATA_DIR + '/*.jsonl',
    gcs_destination_prefix=SERVING_OUTPUT_DATA_DIR,
    instances_format='jsonl',
    predictions_format='jsonl',
    sync=True,
    **job_resources,
)


I am working on the file "07-sympy.ipynb".
The first task is:
A parameter can control how many times to differentiate:
Can you write Python code for it?


I am working on the file "07-sympy.ipynb".
The first task is:
Again, the doit method will explicitly evaluate the result where possible.
Differential equations
Defining and solving differential equations uses the pattern from the previous sections. We'll use the same example problem as in the scipy case,
$$  \frac{\text{d} y}{\text{d} t} = e^{-t} - y, \qquad y(0) = 1. $$
First we define that $y$ is a function, currently unknown, and $t$ is a variable.
Can you write Python code for it?

y = sympy.Function('y')
t = sympy.Symbol('t')


I am working on the file "07-sympy.ipynb".
The first task is:
Again, the doit method will explicitly evaluate the result where possible.
Differential equations
Defining and solving differential equations uses the pattern from the previous sections. We'll use the same example problem as in the scipy case,
\begin{equation}
  \frac{\text{d} y}{\text{d} t} = e^{-t} - y, \qquad y(0) = 1.
\end{equation}
First we define that $y$ is a function, currently unknown, and $t$ is a variable.
Can you write Python code for it?


I am working on the file "07-sympy.ipynb".
The first task is:
Again, there is an unevaluated integral:
Can you write Python code for it?

sympy.Integral(integrand, x)

sympy.Integral(integrand, (x, 1, 10))


I am working on the file "07-sympy.ipynb".
The first task is:
And we can directly apply this function to the times at which the scipy solution is constructed, for comparison:
Can you write Python code for it?


I am working on the file "07-sympy.ipynb".
The first task is:
Differentiation and integration
Differentiation
There is a standard function for differentiation, diff:
Can you write Python code for it?

expression = x**2*sympy.sin(sympy.log(x))
sympy.diff(expression, x)


I am working on the file "07-sympy.ipynb".
The first task is:
Here we have used diff as a method applied to the function. As sympy can't differentiate $y(t)$ (as it doesn't have an explicit value), it leaves it unevaluated.
We can now use the dsolve function to get the solution to the ODE. The syntax is very similar to the solve function used above:
Can you write Python code for it?


I am working on the file "07-sympy.ipynb".
The first task is:
In addition to variables, we can also define general functions. There is only one option for this:
Can you write Python code for it?


I am working on the file "07-sympy.ipynb".
The first task is:
In-built functions
We have seen already that mathematical functions can be found in different packages. For example, the $\sin$ function appears in math as math.sin, acting on a single number. It also appears in numpy as numpy.sin, where it can act on vectors and arrays in one go. sympy re-implements many mathematical functions, for example as sympy.sin, which can act on abstract (sympy) variables.
Whenever using sympy we should use sympy functions, as these can be manipulated and simplified. For example:
Can you write Python code for it?

c = sympy.sin(x)**2 + sympy.cos(x)**2

c

c.simplify()


I am working on the file "07-sympy.ipynb".
The first task is:
Integration
Integration uses the integrate function. This can calculate either definite or indefinite integrals, but will not include the integration constant.
Can you write Python code for it?

integrand=sympy.log(x)**2
sympy.integrate(integrand, x)

sympy.integrate(integrand, (x, 1, 10))


I am working on the file "07-sympy.ipynb".
The first task is:
Multiple integration for higher dimensional integrals can be performed:
Can you write Python code for it?

sympy.integrate(sympy.exp(-(x+y))*sympy.cos(x)*sympy.sin(y), x, y)

sympy.integrate(sympy.exp(-(x+y))*sympy.cos(x)*sympy.sin(y),
                (x, 0, sympy.pi), (y, 0, sympy.pi))


I am working on the file "07-sympy.ipynb".
The first task is:
Note that the "name" of the variable does not need to match the symbol with which it is displayed. We have used this with z0 above:
Can you write Python code for it?


I am working on the file "07-sympy.ipynb".
The first task is:
Note the steps taken here. c is an object, something that sympy has created. Once created it can be manipulated and simplified, using the methods on the object. It is useful to use tab completion to look at the available commands. For example,
Can you write Python code for it?


I am working on the file "07-sympy.ipynb".
The first task is:
Now type d. and then tab, to inspect all the available methods. As before, we could do
Can you write Python code for it?


I am working on the file "07-sympy.ipynb".
The first task is:
Now we can we matplotlib to plot both on the same figure:
Can you write Python code for it?

from matplotlib import pyplot
pyplot.plot(t_scipy, y_scipy[:,0], 'b-', label='scipy')
pyplot.plot(t_scipy, y_sympy, 'k--', label='sympy')
pyplot.xlabel(r'$t$')
pyplot.ylabel(r'$y$')
pyplot.legend(loc='upper right')
pyplot.show()


I am working on the file "07-sympy.ipynb".
The first task is:
Once we have variables, we can define new variables by operating on old ones:
Can you write Python code for it?

a = x + y
b = y * z0
print("a={}. b={}.".format(a, b))

a


I am working on the file "07-sympy.ipynb".
The first task is:
Partial differentiation with respect to multiple variables can also be performed by increasing the number of arguments:
Can you write Python code for it?

expression2 = x*sympy.cos(y**2 + x)
sympy.diff(expression2, x, 2, y, 3)


I am working on the file "07-sympy.ipynb".
The first task is:
Plotting
sympy provides an interface to matplotlib so that expressions can be directly plotted. For example,
Can you write Python code for it?


%matplotlib inline
from matplotlib import rcParams
rcParams['figure.figsize']=(12,9)

sympy.plot(sympy.sin(x));