I am working on the file "Datasets/Pigoo/Pigoo_Desagregacion.ipynb".
The first task is:
Gracias a que este dataset ya contiene etiquetas con claves geoestadísticas, es posible clasificarlas a su vez dentro de acuerdo con el Sistema Urbano Nacional. Para hacer la clasificación, se utiliza un algoritmo elaborado previamente que identifica la clave geoestadística municipal de 5 dígitos ("CVE_MUN") en cada renglón y la clasifica el municipio identificado asignándole la clave del Sistema Urbano Nacional ("CVE_SUN") a la que pertenece.
Es importante señalar las limitaciones de este algoritmo:
1. El algoritmo, además de clasificar los municipios, les asigna el nombre estándar como aparece en el marco geoestadístico del INEGI y el nombre de las ciudades como aparece en el Sistema Urbano Nacional.
2. Unicamente clasifica municipios que forma parte del Subsistema Principal del SUN. Desecha aquellos que no forman parte del Subsistema Principal.
3. Unicamente clasifica renglones cuya clave geoestadística municipal es de 5 dígitos ("CVE_MUN"). Esto es relevante pues en el dataset de entrada, la "Ciudad de México"  únicamente cuentan con 2 dígitos de clave geoestadística
Can you write Python code for it?

# Estandarizacion de clave geoestadistica municipal a 5 dígitos
pigoo_inegi_df['CVE_MUN'] = pigoo_inegi_df['Clave-Estado-Inegi'].map(str) + pigoo_inegi_df['Clave-Municipio-Inegi']
# Clasificación de acuerdo al Sistema Urbano Nacional
variables_SUN = ['CVE_MUN', 'NOM_MUN', 'CVE_SUN', 'NOM_SUN', 'TIPO_SUN', 'NOM_ENT']
pigoo_sun = asignar_sun(pigoo_inegi_df, vars=variables_SUN)
Columnas = ['Nombre- PIGOO', 'CVE_MUN', 'NOM_MUN', 'CVE_SUN', 'NOM_SUN', 'TIPO_SUN', 'NOM_ENT', 'Organismo Operador', 'Siglas', 'Loc-Inegi']
pigoo_sun[Columnas].head()


I am working on the file "Datasets/Pigoo/Pigoo_Desagregacion.ipynb".
The first task is:
Ll hecho de que esta ciudad aparezca etiquetada como "Francisco y Madero" crea confusión respecto a cuál es la informacion con la que se construyó este indicador. La pregunta es ¿Valdría la pena tener consideraciones especiales para esta ciudad?
4. Estandarizacion del dataset PIGOO
Can you write Python code for it?

# Descarga del dataset
dirdestino = r'D:\PCCS\00_RawData\01_CSV'
archivodestino = r'pigoo.csv'
destino = r'{}\{}'.format(dirdestino, archivodestino)
fuente = r'http://www.pigoo.gob.mx/dashboard/exportar/excel/exportar_consulta_excel.php?tipo=max'
print('Descargando dataset ... ... ... ...')
urllib.request.urlretrieve(fuente, destino)
print('dataset descargado en {}'.format(destino))
dataset = pd.read_csv(destino,
                        header = 3)
dataset = dataset.set_index('ciudad')
dataset['ciudad'] = dataset.index


I am working on the file "Datasets/Pigoo/Pigoo_Desagregacion.ipynb".
The first task is:
Mientras que para el SUN, el municipio 28038 - Tampico, forma parte de la Zona Metropolitana 042 - "Tampico", que está compuesta por 5 municipios incluyento 28009 - Ciudad Madero
Can you write Python code for it?

tamp = sun[sun.index == '042']
tamp[datos_sun]


I am working on the file "Datasets/Pigoo/Pigoo_Desagregacion.ipynb".
The first task is:
Si la ciudad que en el dataset PIGOO aparece como Ciudad de México se considera como un resumen de las 16 delegaciones anteriormente mencionadas, sería necesario verificar que existen datos para todos los municipios restantes existen en el dataset PIGOO. No es el caso, de 60 municipios que integran la Zona Metropolitana del Valle de México fuera de la Ciudad de México (Antes Distrito Federal), el dataset PIGOO únicamente contiene 15:
Can you write Python code for it?

sun_vdmx = vdmx['CVE_MUN'].tolist()    # Lista de municipios en el SUN que componen el Valle de México (vdmx)
pigoo_muns = pigoo_inegi_df['CVE_MUN'].tolist()   # Lista de municipios en el dataset Pigoo
vdmx_en_PIGOO = [i for i in pigoo_muns if i in sun_vdmx]   # Lista de municipios del vdmx en el dataset PIGOO

sun_vdmx = vdmx['CVE_MUN'].tolist()
pigoo_muns = pigoo_inegi_df['CVE_MUN'].tolist()
pigoo_muns

len()

pigoo_inegi_df[pigoo_inegi_df['CVE_MUN'] == '15109']


I am working on the file "Datasets/Pigoo/Pigoo_Desagregacion.ipynb".
The first task is:
Utilizando este dataset, ya es posible asignar claves geoestadísticas municipales a las ciudades del dataset PIGOO
Can you write Python code for it?

dataset = pd.merge(dataset, nom_estandar[['CVE_MUN', 'Nombre en Dataset PIGOO']], how='left', left_on='ciudad', right_on='Nombre en Dataset PIGOO')
del(dataset['Nombre en Dataset PIGOO'])
dataset.head()


I am working on the file "Datasets/SEPOMEX/SEPOMEX.ipynb".
The first task is:
Debido a que el estado y municipio pueden ser derivados desde la CVE_MUN de 5 dígitos, voy a eliminar las columnas:
* idEstado
* estado
* idMunicipio
* municipio
Tambien eliminaré otras columnas por diferentes motivos:

Ciudad, pues esta será asignada acorde a la Clave SUN
Zona, porque 'rural/urbano' no se utiliza para la construccion de parametros
Asentamiento, porque es un nivel de desagregación que no se utiliza
tipo, porque no se utiliza para la construccion de parametros
Can you write Python code for it?

columnasnointeresantes = ['idEstado', 'estado', 'idMunicipio', 'municipio', 'ciudad', 'zona', 'asentamiento', 'tipo']
for columna in columnasnointeresantes:
    del(SEPOMEX1[columna])
orden = ['CP', 'CVE_MUN']
SEPOMEX1 = SEPOMEX1[orden]
SEPOMEX1.head(3)


I am working on the file "Datasets/SNIIV/.ipynb_checkpoints/SNIIV-checkpoint.ipynb".
The first task is:
2. Descarga de datos
Los datos se descargan por medio de una conexión a un servicio SOAP proporcionado por el SNIIV. Para acceder a los datos proporcionados por cada uno de los servicios del SNIIV, tiene que hacerse un POST request, especificando en el encabezado el servicio al que se busca acceder y en el cuerpo de la petición, un XML con parametros para que el servidor de SNIIV pueda regresar una respuesta
Can you write Python code for it?

# Esta celda contiene textos estándar para los encabezados y el cuerpo de la operación que se solicita al servidor.
# los textos entre corchetes {} sirven para especificar la operacion a la que se busca tener acceso
scheme = r'http://www.conavi.gob.mx:8080/WS_App_SNIIV.asmx?WSDL'       #El scheme siempre es el mismo
SOAPAction = ('http://www.conavi.gob.mx:8080/WS_App_SNIIV/{}')
xmlbody = (
'<?xml version="1.0" encoding="utf-8"?>'
'<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">'
  '<soap:Body>'
    '<{} xmlns="http://www.conavi.gob.mx:8080/WS_App_SNIIV">'
      '<dat></dat>'
    '</{}>'
  '</soap:Body>'
'</soap:Envelope>'
)

# Conexion y descarga de datos
operacion = 'Subsidios'
heads = {'Content-Type': 'text/xml; charset=utf-8',
           'SOAPAction': SOAPAction.format(operacion)}
body = xmlbody.format(operacion, operacion)
r = requests.post(scheme, data=body, headers=heads)
print(r.content[0:300])

list(r.headers.keys())

r = xmltodict.parse(r.content)
r.keys()

# Después de parseada la respuesta a un diccionario de Python, los datos se encuentran varios niveles por debajo, por lo que es
# necesario explorar estos niveles hasta llegar a los datos útiles
rodict = r['soap:Envelope']['soap:Body']['{}Response'.format(operacion)]['{}Result'.format(operacion)]['app_sniiv_rep_subs']
rodict[0]

# Con los datos parseados en forma de OrderedDict ya es posible hacer un DataFrame de la siguiente manera.
pd.DataFrame(rodict, columns=rodict[0].keys()).head()


I am working on the file "Datasets/SNIIV/.ipynb_checkpoints/SNIIV-checkpoint.ipynb".
The first task is:
Lo anterior es un ejemplo de cómo pueden obtenerse datos desde el servicio montado por el SNIIV. Con base en este ejemplo es posible hacer una función que realice las peticiones de manera más compacta en cada caso.
Can you write Python code for it?

def getsoap(operacion):
    scheme = r'http://www.conavi.gob.mx:8080/WS_App_SNIIV.asmx?WSDL'       #El scheme siempre es el mismo
    SOAPAction = ('http://www.conavi.gob.mx:8080/WS_App_SNIIV/{}')
    xmlbody = (
    '<?xml version="1.0" encoding="utf-8"?>'
    '<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">'
      '<soap:Body>'
        '<{} xmlns="http://www.conavi.gob.mx:8080/WS_App_SNIIV">'
          '<dat></dat>'
        '</{}>'
      '</soap:Body>'
    '</soap:Envelope>'
    )
    heads = {'Content-Type': 'text/xml; charset=utf-8',
           'SOAPAction': SOAPAction.format(operacion)}
    body = xmlbody.format(operacion, operacion)
    r = requests.post(scheme, data=body, headers=heads)
    if r.status_code != 200:          #
        print('status: {}\n**Operacion terminada**'.format(r.status_code))
        return
    else:
        print('status: {}\nContent Type: {}'.format(r.status_code, r.headers['Content-Type']))
        print('Date: {}\nContent Lenght: {}'.format(r.headers['Date'], r.headers['Content-Length']))
        r = xmltodict.parse(r.content)
#        return r
        rodict = r['soap:Envelope']['soap:Body']['{}Response'.format(operacion)]['{}Result'.format(operacion)][list(test.keys())[0]]
        return pd.DataFrame(rodict, columns=rodict[0].keys()).head()

test = getsoap('viv_vig_x_avnc')

test


I am working on the file "Datasets/SNIIV/SNIIV.ipynb".
The first task is:
2. Descarga de datos
Los datos se descargan por medio de una conexión a un servicio SOAP proporcionado por el SNIIV. Para acceder a los datos proporcionados por cada uno de los servicios del SNIIV, tiene que hacerse un POST request, especificando en el encabezado el servicio al que se busca acceder y en el cuerpo de la petición, un XML con parametros para que el servidor de SNIIV pueda regresar una respuesta
Can you write Python code for it?


I am working on the file "Datasets/SNIIV/SNIIV.ipynb".
The first task is:
Lo anterior es un ejemplo de cómo pueden obtenerse datos desde el servicio montado por el SNIIV. Con base en este ejemplo es posible hacer una función que realice las peticiones de manera más compacta en cada caso.
Can you write Python code for it?

def getsoap(operacion):
    scheme = r'http://www.conavi.gob.mx:8080/WS_App_SNIIV.asmx?WSDL'       # El scheme siempre es el mismo
    SOAPAction = ('http://www.conavi.gob.mx:8080/WS_App_SNIIV/{}')
    xmlbody = (
    '<?xml version="1.0" encoding="utf-8"?>'
    '<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">'
      '<soap:Body>'
        '<{} xmlns="http://www.conavi.gob.mx:8080/WS_App_SNIIV">'
          '<dat></dat>'
        '</{}>'
      '</soap:Body>'
    '</soap:Envelope>'
    )
    heads = {'Content-Type': 'text/xml; charset=utf-8',
           'SOAPAction': SOAPAction.format(operacion)}
    body = xmlbody.format(operacion, operacion)
    r = requests.post(scheme, data=body, headers=heads)
    if r.status_code != 200:          #
        print('status: {}\n**Operacion terminada**'.format(r.status_code))
        return None
    else:
        print(operacion)
        print('status: {}\nContent Type: {}'.format(r.status_code, r.headers['Content-Type']))
        print('Date: {}\nContent Lenght: {}'.format(r.headers['Date'], r.headers['Content-Length']))
        r = xmltodict.parse(r.content)
        try:
            rodict = r[list(test.keys())[0]]['soap:Body']['{}Response'.format(operacion)]['{}Result'.format(operacion)][list(test.keys())[0]]
            df = pd.DataFrame(rodict, columns=rodict[0].keys())
            print('dataframe creado')
            return df
        except:
            print('---------\nNo fue posible crear dataframe. Regresando OrderedDict')
            return r

# Prueba del Script
test = getsoap('get_tot_fech')
test


I am working on the file "Datasets/SNIIV/SNIIV.ipynb".
The first task is:
Se van a revisar los siguientes:
get_subs_x_valviv_mun - Subsidios CONAVI
get_subs_x_rgoing_mun - Subsidios CONAVI
get_regviv_evol_acum - Vivienda en PCU
get_cont_vv_mun - Vivienda en PCU
get_tipohv_vv_mun - Vivienda Vigente por tipo
ID |Descripción
---|:----------
P0405|Viviendas Verticales
P0406|Viviendas urbanas en PCU U1 y U2
P0411|Subsidios CONAVI
Subsidios CONAVI
Los datos para este indicador pueden estar en:
get_subs_x_valviv_mun
get_subs_x_rgoing_mun
Can you write Python code for it?

# Agrupadores estándar para todos los xml
senv = 'soap:Envelope'
sbo = 'soap:Body'

revdata = datos['get_subs_x_valviv_mun'][senv][sbo]['get_subs_x_valviv_munResponse']['get_subs_x_valviv_munResult']
revdata

# Los datos que regresó el servidor para esta operación parecen ser un JSON, pero vienen en forma de str
type(revdata)

revdic = ast.literal_eval(revdata)
revdic = collections.OrderedDict(revdic)
revdic

revdic.keys()

pd.DataFrame(revdic2['01'], columns=revdic2['01'][list(revdic2['01'].keys())])

revdic2['01']['AGUASCALIENTES']

revdic2 = {}
for estado in revdic.keys():
    tempdict = {}
    revdic2[estado] = collections.OrderedDict(revdic[estado][0])
revdic2['01']


I am working on the file "Datasets/ipython_notebook/ex1_the_digits_dataset.ipynb".
The first task is:
(二)資料集介紹
digits = datasets.load_digits() 將一個dict型別資料存入digits，我們可以用下面程式碼來觀察裏面資料
Can you write Python code for it?

for key,value in digits.items() :
    try:
        print (key,value.shape)
    except:
        print (key)


I am working on the file "Datasets/ipython_notebook/ex1_the_digits_dataset.ipynb".
The first task is:
| 顯示 | 說明 |
| -- | -- |
| ('images', (1797L, 8L, 8L))| 共有 1797 張影像，影像大小為 8x8 |
| ('data', (1797L, 64L)) | data 則是將8x8的矩陣攤平成64個元素之一維向量 |
| ('target_names', (10L,)) | 說明10種分類之對應 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] |
| DESCR | 資料之描述 |
| ('target', (1797L,))| 記錄1797張影像各自代表那一個數字 |
接下來我們試著以下面指令來觀察資料檔，每張影像所對照的實際數字存在digits.target變數中
Can you write Python code for it?

images_and_labels = list(zip(digits.images, digits.target))
for index, (image, label) in enumerate(images_and_labels[:4]):
    plt.subplot(2, 4, index + 1)
    plt.axis('off')
    plt.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')
    plt.title('Training: %i' % label)


I am working on the file "Datasets/ipython_notebook/ex2_the_iris_dataset.ipynb".
The first task is:
(二)資料集介紹
digits = datasets.load_digits() 將一個dict型別資料存入digits，我們可以用下面程式碼來觀察裏面資料
Can you write Python code for it?

for key,value in iris.items() :
    try:
        print (key,value.shape)
    except:
        print (key)
print(iris['feature_names'])


I am working on the file "Datasets/ipython_notebook/ex2_the_iris_dataset.ipynb".
The first task is:
| 顯示 | 說明 |
| -- | -- |
| ('target_names', (3L,))| 共有三種鳶尾花 setosa, versicolor, virginica |
| ('data', (150L, 4L)) | 有150筆資料，共四種特徵 |
| ('target', (150L,))| 這150筆資料各是那一種鳶尾花|
| DESCR | 資料之描述 |
| feature_names| 四個特徵代表的意義，分別為 萼片(sepal)之長與寬以及花瓣(petal)之長與寬
為了用視覺化方式呈現這個資料集，下面程式碼首先使用PCA演算法將資料維度降低至3
Can you write Python code for it?


I am working on the file "DateAndTimes/time.ipynb".
The first task is:
Monotonic Clocks
Can you write Python code for it?

import time
start = time.monotonic()
time.sleep(0.1)
end = time.monotonic()
print('Start: {:>9.2f}'.format(start))
print('End: {:>9.2f}'.format(end))
print('Span: {:>9.2f}'.format(end-start))


I am working on the file "DateAndTimes/time.ipynb".
The first task is:
Performance Counter
Can you write Python code for it?

import hashlib
import time

# Data to use to calculate md5 checksums
data = open('time.ipynb', 'rb').read()

loop_start = time.perf_counter()

for i in range(5):
    iter_start = time.perf_counter()
    h = hashlib.sha1()
    for i in range(300000):
        h.update(data)
    cksum = h.digest()
    now = time.perf_counter()
    loop_elapsed = now - loop_start
    iter_elapsed = now - iter_start
    print(time.ctime(), ': {:0.3f} {:0.3f}'.format(
        iter_elapsed, loop_elapsed))


I am working on the file "DateAndTimes/time.ipynb".
The first task is:
Processor Clock Time
Can you write Python code for it?

import hashlib
import time
data = open('time.ipynb', 'rb').read()
for i in range(5):
    h = hashlib.sha1()
    print(time.ctime(), ': {:0.3f} {:0.3f}'.format(
        time.time(), time.clock()))
    for i in range(300000):
        h.update(data)
    cksum=h.digest()

import time

template = '{} - {:0.2f} - {:0.2f}'

print(template.format(
    time.ctime(), time.time(), time.clock())
)

for i in range(3, 0, -1):
    print('Sleeping', i)
    time.sleep(i)
    print(template.format(
        time.ctime(), time.time(), time.clock())
    )


I am working on the file "DateAndTimes/time.ipynb".
The first task is:
Time Components
Can you write Python code for it?

import time

def show_struct(s):
    print('  tm_year :', s.tm_year)
    print('  tm_mon  :', s.tm_mon)
    print('  tm_mday :', s.tm_mday)
    print('  tm_hour :', s.tm_hour)
    print('  tm_min  :', s.tm_min)
    print('  tm_sec  :', s.tm_sec)
    print('  tm_wday :', s.tm_wday)
    print('  tm_yday :', s.tm_yday)
    print('  tm_isdst:', s.tm_isdst)

print('gmtime:')
show_struct(time.gmtime())
print('\nlocaltime:')
show_struct(time.localtime())
print('\nmktime:', time.mktime(time.localtime()))


I am working on the file "DateAndTimes/time.ipynb".
The first task is:
Wall Clock Time
Can you write Python code for it?

import time
print('The Time is:', time.time())

import time
print('The time is   :', time.ctime())
later = time.time() + 15
print('15 secs from now:', time.ctime(later))


I am working on the file "Dates3.ipynb".
The first task is:
LAB CHALLENGE
What time is it right now in:

Moscow
Berlin
Tokyo
Can you write Python code for it?

pd.Timestamp.now(tz='US/Pacific')  #  getting you started


I am working on the file "Dates3.ipynb".
The first task is:
Timestamp type (for individual datetimes)
Can you write Python code for it?

stamp = pd.Timestamp('2011-03-12 04:00')

stamp2 = pd.Timestamp('Wed May 23 11:35:54 2018') # will this work too?

type(stamp2)

stamp2_pac = stamp2.tz_localize('US/Pacific')
stamp2_pac

stamp2_pac.tz_convert('Europe/Moscow')

stamp2_pac.value # nanoseconds since the UNIX Epoch, Jan 1 1970

stamp2_pac.tz_convert('Europe/Moscow').value

stamp3 = pd.Timestamp('Wed May 23 11:35:54 1950')

stamp3.value  # negative number because before the UNIX Epoch

ts

ts_sum = ts_eastern + ts_utc.tz_convert("Europe/Moscow")

ts_sum.index


I am working on the file "Dates4.ipynb".
The first task is:
Quarterly Period Frequencies
Can you write Python code for it?

p = Period('2018Q4', freq='Q-JAN')

p

p.asfreq('D', 'start')

p.asfreq('D', 'end')

p.asfreq('Y', 'start')

p.asfreq('M', 'start')

p.asfreq('M', 'start') + 3

p.asfreq('Q-JAN', 'start')

p.asfreq('Q-JAN', 'end')

p.asfreq('Q', 'end')


I am working on the file "Dates4.ipynb".
The first task is:
Timestamps to Periods (and Back)
Can you write Python code for it?

rng = pd.date_range('1/1/2018', periods=6, freq='M')

rng

ts = Series(np.random.randn(6), index=rng)

ts

pts = ts.to_period()

pts

rng = pd.date_range('1/28/2018', periods=10, freq='D')
ts2 = Series(np.random.randn(len(rng)), index=rng)

ts2

ts2.to_period('M')

pts

pts.to_timestamp(how='end')

pts.to_timestamp(how='start')

pts2 = ts2.to_period('M')

pts2

pts2.to_timestamp(how='end')

ts

%matplotlib inline
ts.plot(kind="bar")

ts.plot(kind="line")


I am working on the file "Dates_Times/datetime-Date and Time Value Manipulation.ipynb".
The first task is:
A timedelta object also supports arithmetic with integers, floats, and other timedelta instances.
Can you write Python code for it?

import datetime

one_day = datetime.timedelta(days=1)
print('1 day    :', one_day)
print('5 days   :', one_day * 5)
print('1.5 days :', one_day * 1.5)
print('1/4 day  :', one_day / 4)

# assume an hour for lunch
work_day = datetime.timedelta(hours=7)
meeting_length = datetime.timedelta(hours=1)
print('meetings per day :', work_day / meeting_length)


I am working on the file "Dates_Times/datetime-Date and Time Value Manipulation.ipynb".
The first task is:
Another way to create new date instances uses the replace() method of an existing date.
Can you write Python code for it?

import datetime

d1 = datetime.date(2008, 3, 29)
print('d1:', d1.ctime())

d2 = d1.replace(year=2009)
print('d2:', d2.ctime())


I am working on the file "Dates_Times/datetime-Date and Time Value Manipulation.ipynb".
The first task is:
Combining Dates and Times
Use the datetime class to hold values consisting of both date and time components. As with date, there are several convenient class methods to make creating datetime instances from other common values.
Can you write Python code for it?

import datetime

print('Now    :', datetime.datetime.now())
print('Today  :', datetime.datetime.today())
print('UTC Now:', datetime.datetime.utcnow())
print()

FIELDS = [
    'year', 'month', 'day',
    'hour', 'minute', 'second',
    'microsecond',
]

d = datetime.datetime.now()
for attr in FIELDS:
    print('{:15}: {}'.format(attr, getattr(d, attr)))


I am working on the file "Dates_Times/datetime-Date and Time Value Manipulation.ipynb".
The first task is:
Comparing Values
Both date and time values can be compared using the standard comparison operators to determine which is earlier or later.
Can you write Python code for it?

import datetime
import time

print('Times:')
t1 = datetime.time(12, 55, 0)
print('  t1:', t1)
t2 = datetime.time(13, 5, 0)
print('  t2:', t2)
print('  t1 < t2:', t1 < t2)

print()
print('Dates:')
d1 = datetime.date.today()
print('  d1:', d1)
d2 = datetime.date.today() + datetime.timedelta(days=1)
print('  d2:', d2)
print('  d1 > d2:', d1 > d2)


I am working on the file "Dates_Times/datetime-Date and Time Value Manipulation.ipynb".
The first task is:
Dates
Calendar date values are represented with the date class. Instances have attributes for year, month, and day. It is easy to create a date representing the current date using the today() class method.
Can you write Python code for it?

import datetime

today = datetime.date.today()

print(today)
print("ctime  :", today.ctime())
tt = today.timetuple()

print('tuple  : tm_year  =', tt.tm_year)
print('         tm_mon   =', tt.tm_mon)
print('         tm_mday  =', tt.tm_mday)
print('         tm_hour  =', tt.tm_hour)
print('         tm_min   =', tt.tm_min)
print('         tm_sec   =', tt.tm_sec)
print('         tm_wday  =', tt.tm_wday)
print('         tm_yday  =', tt.tm_yday)
print('         tm_isdst =', tt.tm_isdst)
print('ordinal:', today.toordinal())
print('Year   :', today.year)
print('Mon    :', today.month)
print('Day    :', today.day)


I am working on the file "Dates_Times/datetime-Date and Time Value Manipulation.ipynb".
The first task is:
Note:
the resolution for time is limited to whole microseconds
Can you write Python code for it?

import datetime

for m in [1, 0, 0.1, 0.6]:
    try:
        print('{:02.1f} :'.format(m),
              datetime.time(0, 0, 0, microsecond=m))
    except TypeError as err:
        print('ERROR:', err)


I am working on the file "Dates_Times/datetime-Date and Time Value Manipulation.ipynb".
The first task is:
The full duration of a timedelta can be retrieved as a number of seconds using total_seconds().
Can you write Python code for it?

import datetime

for delta in [datetime.timedelta(microseconds=1),
              datetime.timedelta(milliseconds=1),
              datetime.timedelta(seconds=1),
              datetime.timedelta(minutes=1),
              datetime.timedelta(hours=1),
              datetime.timedelta(days=1),
              datetime.timedelta(weeks=1),
              ]:
    print('{:15} = {:8} seconds'.format(
        str(delta), delta.total_seconds())
    )


I am working on the file "Dates_Times/datetime-Date and Time Value Manipulation.ipynb".
The first task is:
There are also class methods for creating instances from POSIX timestamps or integers representing date values from the Gregorian calendar, where January 1 of the year 1 is 1 and each subsequent day increments the value by 1.
Can you write Python code for it?

import datetime
import time

o = 733114
print('o               :', o)
print('fromordinal(o)  :', datetime.date.fromordinal(o))

t = time.time()
print('t               :', t)
print('fromtimestamp(t):', datetime.date.fromtimestamp(t))


I am working on the file "Dates_Times/datetime-Date and Time Value Manipulation.ipynb".
The first task is:
datetime min max
Can you write Python code for it?

import datetime

print('Earliest    :', datetime.time.min)
print("Latest      :", datetime.time.max)
print("Resolution  :", datetime.time.resolution)


I am working on the file "Dates_Times/datetime-Date and Time Value Manipulation.ipynb".
The first task is:
timedeltas
Future and past dates can be calculated using basic arithmetic on two datetime objects, or by combining a datetime with a timedelta. Subtracting dates produces a timedelta, and a timedelta can be added or subtracted from a date to produce another date. The internal values for a timedelta are stored in days, seconds, and microseconds.
Can you write Python code for it?

import datetime

print('microseconds:', datetime.timedelta(microseconds=1))
print('milliseconds:', datetime.timedelta(milliseconds=1))
print('seconds     :', datetime.timedelta(seconds=1))
print('minutes     :', datetime.timedelta(minutes=1))
print('hours       :', datetime.timedelta(hours=1))
print('days        :', datetime.timedelta(days=1))
print('weeks       :', datetime.timedelta(weeks=1))


I am working on the file "Dates_Times/time_clock_time.ipynb".
The first task is:
As with monotonic(), the epoch for perf_counter() is undefined, and the values are meant to be used for comparing and computing values, not as absolute times.
Time Components
Storing times as elapsed seconds is useful in some situations, but there are times when a program needs to have access to the individual fields of a date (year, month, etc.). The time module defines struct_time for holding date and time values with components broken out so they are easy to access. There are several functions that work with struct_time values instead of floats.
Can you write Python code for it?


I am working on the file "Dates_Times/time_clock_time.ipynb".
The first task is:
In this example, the formatted ctime() is printed along with the floating point values from time(), and clock() for each iteration through the loop.
Typically, the processor clock does not tick if a program is not doing anything.
Can you write Python code for it?

import time

template = '{} - {:0.2f} - {:0.2f}'

print(template.format(
    time.ctime(), time.time(), time.clock())
)

for i in range(3, 0, -1):
    print('Sleeping', i)
    time.sleep(i)
    print(template.format(
        time.ctime(), time.time(), time.clock())
    )


I am working on the file "Dates_Times/time_clock_time.ipynb".
The first task is:
The epoch is the start of measurement for time, which for Unix systems is 0:00 on January 1, 1970. Although the value is always a float, actual precision is platform-dependent.
The float representation is useful when storing or comparing dates, but not as useful for producing human readable representations. For logging or printing time ctime() can be more useful.
Can you write Python code for it?

import time

print('The time is      :', time.ctime())
later = time.time() + 15
print('15 secs from now :', time.ctime(later))


I am working on the file "Dates_Times/time_clock_time.ipynb".
The first task is:
The gmtime() function returns the current time in UTC.

localtime() returns the current time with the current time zone applied.

mktime() takes a struct_time and converts it to the floating point representation.

Working with Time Zones
The functions for determining the current time depend on having the time zone set, either by the program or by using a default time zone set for the system. Changing the time zone does not change the actual time, just the way it is represented.
To change the time zone, set the environment variable TZ, then call tzset(). The time zone can be specified with a lot of detail, right down to the start and stop times for daylight savings time. It is usually easier to use the time zone name and let the underlying libraries derive the other information, though.
This example program changes the time zone to a few different values and shows how the changes affect other settings in the time module.
Can you write Python code for it?

import time
import os

def show_zone_info():
    print('  TZ    :', os.environ.get('TZ', '(not set)'))
    print('  tzname:', time.tzname)
    print('  Zone  : {} ({})'.format(
        time.timezone, (time.timezone / 3600)))
    print('  DST   :', time.daylight)
    print('  Time  :', time.ctime())
    print()

print('Default :')
show_zone_info()

ZONES = [
    'GMT',
    'Europe/Amsterdam',
]

for zone in ZONES:
    os.environ['TZ'] = zone
    time.tzset()
    print(zone, ':')
    show_zone_info()


I am working on the file "Dato-tutorials/anomaly-detection/Anomaly Detection - Demo 1 [Local Outlier Factor - LOF Model].ipynb".
The first task is:
Firstly, note that the model worked successfully, scoring each of the 124,681 input rows. Secondly, the anomaly score for many observations in our AirBnB dataset is nan which indicates the point has many neighbors at exactly the same location, making the ratio of densities undefined. These points cannot be outliers.
However, for the problem at hand we are interested to find if any outliers exist and under what circumstances this happens. This is where the real business value exists!
Using the LOF Model to detect anomalies
There are two common ways to detect which observations of your data set are anomalous or not:
A. Ask from the trained model to return the k more anomalous observations:
By applying the .topk() method of the model scores SFrame
Can you write Python code for it?

top10_anomalies = model_lof['scores'].topk('anomaly_score', k=10)
top10_anomalies.print_rows(num_rows=10)


I am working on the file "Dato-tutorials/anomaly-detection/Anomaly Detection - Demo 1 [Local Outlier Factor - LOF Model].ipynb".
The first task is:
From the quick exploratory data analysis below:
Can you write Python code for it?

%matplotlib inline
univariate_summary_plot(customer_data, features, nsubplots_inrow=3, subplots_wspace=0.7)

gl.canvas.set_target('browser')
customer_data[['age']].show()

print 'Number of customer records with ages larger than 2013: %d' %\
len(customer_data[customer_data['age'] >= 2013])


I am working on the file "Dato-tutorials/anomaly-detection/Anomaly Detection - Demo 1 [Local Outlier Factor - LOF Model].ipynb".
The first task is:
If we would like to explore in more detail the countplot for the variable language, we can temporarily exclude the english-speaking customers and redraw the graph:
Can you write Python code for it?

# exclude the english-speaking customers
customer_data_df_nen = customer_data_df[customer_data_df['language']!='en']

# define seaborn style, palette, color codes
sns.set(style="whitegrid", palette="deep",color_codes=False)
# initialize the matplotlib figure
plt.figure(figsize=(7,11))
plt.ylabel('language', {'fontweight': 'bold'})
plt.title('Countplot of Customer Languages\n[English-speaking people excluded]',
          {'fontweight': 'bold'})

# draw countplot
ax2 = sns.countplot(y='language', data=customer_data_df_nen, palette='deep', color='b')


I am working on the file "Dato-tutorials/anomaly-detection/Anomaly Detection - Demo 1 [Local Outlier Factor - LOF Model].ipynb".
The first task is:
Note that the model can automatically choose a suitable metric for the data type of the features we have available. Here, a composite distance of a 'jaccard' and 'euclidean' metric has been chosen for the 'encoded_features' and the 'age' columns respectively. Both these two metrics have been weighted with 1.0.
If we want what has been built by the model internally we can simply write:
Can you write Python code for it?


I am working on the file "Dato-tutorials/anomaly-detection/Anomaly Detection - Demo 1 [Local Outlier Factor - LOF Model].ipynb".
The first task is:
The univariate summary statistics plot for this new customer_data_df_nen set is as follows.
Can you write Python code for it?


I am working on the file "Dato-tutorials/anomaly-detection/Anomaly Detection - Demo 1 [Local Outlier Factor - LOF Model].ipynb".
The first task is:
Training a Local Outlier Factor (LOF) Model
Next, we train the LOF model by using this transformed customer_data2 set.
Can you write Python code for it?

model_lof = gl.anomaly_detection.local_outlier_factor.create(customer_data1,
                                                             features = ['age', 'encoded_features'],
                                                             threshold_distances=True,
                                                             verbose=False)

model_lof.save('./model_lof')


I am working on the file "Dato-tutorials/anomaly-detection/Anomaly Detection - Demo 1 [Local Outlier Factor - LOF Model].ipynb".
The first task is:
and more specifically the remaining customer ages follow the distribution below:
Can you write Python code for it?

# transform the SFrame into a Pandas DataFrame
customer_data_df = customer_data.to_dataframe()
customer_data_df['gender'] = customer_data_df['gender'].astype(str)
customer_data_df['age'] = customer_data_df['age'].astype(float)
customer_data_df['language'] = customer_data_df['language'].astype(str)

# define seaborn style, palette, color codes
sns.set(style="whitegrid", palette="deep",color_codes=True)
# initialize the matplotlib figure
plt.figure(figsize=(12,7))

# draw distplot
ax1 = sns.distplot(customer_data_df.age, bins=None, hist=True, kde=False, rug=False, color='b')


I am working on the file "Dato-tutorials/anomaly-detection/Anomaly Detection - Demo 1 [Local Outlier Factor - LOF Model].ipynb".
The first task is:
model_lof = gl.load_model('./model_lof/')
Can you write Python code for it?

print 'The LOF model has been trained with the following options:'
print '-------------------------------------------------------------'
print model_lof.get_current_options()


I am working on the file "Dato-tutorials/anomaly-detection/Anomaly Detection - Demo 1 [Local Outlier Factor - LOF Model].ipynb".
The first task is:
we notice that there about 750 records having an 'age' value of '2013' or '2014', which is of course wrong. Most probably the year was recorded accidentally in this field.  The remaining 'age' values seams absolutely reasonable with only some rare customer entries that have ages greater than '100'. In fact more than 128 thousand customer entries are found to have ages in the [1, 142] interval. More specifically, we have choosen to assume any value falling in the [1,150] interval as an elligible recording of a customer age, re-assigning all the remaining ones as missing:
Can you write Python code for it?

customer_data['age'] = customer_data['age'].apply(lambda age: age if age < 150 else None)
customer_data = customer_data.dropna(columns = features, how='any')
print 'Number of Rows in dataset: %d' % len(customer_data)


I am working on the file "Dato-tutorials/anomaly-detection/Anomaly Detection - Demo 2 [Moving Z-Score and Bayesian Changepoint Models].ipynb".
The first task is:
In the figure below, we plot the original FRED/DCOILBRENTEU time series of "Dollars per Barrel of Crude Oil Brent - Europe", its Moving Average across the years, and the data points that we found to be anomalous with both the Moving Z-Score and the Bayesian Changepoint model.
Can you write Python code for it?

%matplotlib inline

plt.rcParams['figure.figsize'] = 14, 24
plt.figure(1)

plt.subplot(3,1,1)
plt.plot_date(fred_dcoilbrenteu['DATE'], fred_dcoilbrenteu['VALUE'],\
              fmt='g-', tz='utc', label='FRED/DCOILBRENTEU')
plt.plot_date(scores['DATE'], scores['moving_average'],\
              fmt='b-', tz='utc', lw=2, label='Moving Average')
plt.xlabel('Year')
plt.ylabel('Dollars per Barrel')
plt.title('Crude Oil Prices: Brent - Europe [FRED/DCOILBRENTEU]')
plt.rcParams.update({'font.size': 16})
plt.plot(anomalies['DATE'], anomalies['VALUE'],\
         'bx', markersize=12, markeredgewidth=1.3, label='Anomalies [Moving Z-Score]')
plt.legend(loc='upper left', prop={'size': 16})

plt.subplot(3,1,2)
plt.plot_date(fred_dcoilbrenteu['DATE'], fred_dcoilbrenteu['VALUE'],\
              fmt='g-', tz='utc', label='FRED/DCOILBRENTEU')
plt.plot_date(scores['DATE'], scores['moving_average'],\
              fmt='b-', tz='utc', lw=2, label='Moving Average')
plt.xlabel('Year')
plt.ylabel('Dollars per Barrel')
plt.title('Crude Oil Prices: Brent - Europe [FRED/DCOILBRENTEU]')
plt.rcParams.update({'font.size': 16})
plt.plot(changepoints['DATE'], changepoints['VALUE'],\
         'rx', markersize=12, markeredgewidth=1.3, label='Anomalies [Bayesian Changepoints]')
plt.legend(loc='upper left', prop={'size': 16})

plt.subplot(3,1,3)
plt.plot_date(scores2['DATE'], scores2['changepoint_score'],\
              fmt='r-', tz='utc', lw=2, label='Bayesian Changepoint Probability')
plt.rcParams.update({'font.size': 16})
plt.xlabel('Year')
plt.ylabel('Changepoint Probability')
plt.title('Crude Oil Prices: Brent - Europe [FRED/DCOILBRENTEU]')
plt.legend(loc='upper left', prop={'size': 16})

plt.show()


I am working on the file "Dato-tutorials/anomaly-detection/Anomaly Detection - Demo 2 [Moving Z-Score and Bayesian Changepoint Models].ipynb".
The first task is:
In the figure below, we plot the original FRED/DCOILBRENTEU time series of "Dollars per Barrel of Crude Oil Brent - Europe", its Moving Average across the years, and the data points that we found to be anomalous.
Can you write Python code for it?

%matplotlib inline

plot_time_series(fred_dcoilbrenteu['DATE'], fred_dcoilbrenteu['VALUE'],\
                 'Crude Oil Prices: Brent - Europe [FRED/DCOILBRENTEU]', label='FRED/DCOILBRENTEU')
plt.plot_date(scores['DATE'], scores['moving_average'], fmt='b-', tz='utc', lw=2, label='Moving Average')
plt.plot(anomalies['DATE'], anomalies['VALUE'], 'rx', markersize=12, markeredgewidth=1.3, label='Anomalies')
plt.legend(loc='upper left', prop={'size': 16})
plt.show()


I am working on the file "Dato-tutorials/anomaly-detection/Anomaly Detection - Demo 2 [Moving Z-Score and Bayesian Changepoint Models].ipynb".
The first task is:
Next we transform the DATE column in an appropriate timestamp format, and the fred_dcoilbrenteu SFrame in a TimeSeries object.
Can you write Python code for it?

import time
import dateutil

def _unix_timestamp_to_datetime(x):
    import datetime
    import pytz
    return dateutil.parser.parse(x)

fred_dcoilbrenteu['DATE'] = fred_dcoilbrenteu['DATE'].apply(_unix_timestamp_to_datetime)
fred_dcoilbrenteu = gl.TimeSeries(fred_dcoilbrenteu, index='DATE')
fred_dcoilbrenteu


I am working on the file "Dato-tutorials/anomaly-detection/Anomaly Detection - Demo 2 [Moving Z-Score and Bayesian Changepoint Models].ipynb".
The first task is:
Of cource, a lot more anomalous observations may exist in the fred_dcoilbrenteu time series. A good way to make a final decision on that, is to look at the approximate distribution of the anomaly scores with the SArray.sketch_summary() tool, then get a threshold for the anomaly score with the sketch summary's quantile method. Here we declare the top 1% of the data to be anomalies, characterizing that way 71 data points as "anomalous".
Can you write Python code for it?

sketch = scores['anomaly_score'].sketch_summary()
threshold = sketch.quantile(0.99)
anomalies = scores[scores['anomaly_score'] > threshold]
anomalies.print_rows(num_rows=30, max_row_width=100)


I am working on the file "Dato-tutorials/anomaly-detection/Anomaly Detection - Demo 2 [Moving Z-Score and Bayesian Changepoint Models].ipynb".
The first task is:
Of course, the first 252 rows of the scores output don't have a moving average or Z-score. This is because the moving window does not have sufficient data for those observations.
To reveal the 30, lets say, more anomalous data points we can sort the scores SFrame as follows.
Can you write Python code for it?


I am working on the file "Dato-tutorials/anomaly-detection/Anomaly Detection - Demo 2 [Moving Z-Score and Bayesian Changepoint Models].ipynb".
The first task is:
One interesting thing is that if you look at the tail of scores, you will see a handful of missing values. These data points have insufficient data after them to compute lagged changepoint scores. The number of missing values in the tail of the dataset can be reduced by equally reducing the lag parameter in our learning model. However, the returned results will be less accurate. Alternatively, one can choose to update the model with new data.
Can you write Python code for it?


I am working on the file "Dato-tutorials/anomaly-detection/Anomaly Detection - Demo 2 [Moving Z-Score and Bayesian Changepoint Models].ipynb".
The first task is:
The primary output of the Moving Z-score model is the scores field. This TimeSeries object contains:

row id/time: ID of the corresponding row in the input dataset. Here the dataset is a TimeSeries object and the model returns the DATE timestamp. If it was an SFrame, this column would be filled with the row numbers of the input data.
anomaly score: absolute value of the moving Z-score. A score of 0 indicates that the value is identical to the moving average. The higher the score, the more likely a point is to be an anomaly.
VALUE: the recorded value of Dollars per Barrel of "Crude Oil Brent - Europe".
model update time: time that the model was updated. This is particularly useful for model updating.
Can you write Python code for it?

scores = model_moving_zscore.scores.to_sframe()
scores.print_rows(num_rows=10, max_row_width=100)

scores[252-10:252+10].print_rows(num_rows=60, max_row_width=100)


I am working on the file "Dato-tutorials/anomaly-detection/Anomaly Detection - Demo 2 [Moving Z-Score and Bayesian Changepoint Models].ipynb".
The first task is:
To reveal the 30, lets say, more anomalous data points we can sort the scores SFrame as follows.
Can you write Python code for it?


I am working on the file "Dato-tutorials/anomaly-detection/Anomaly Detection - Demo 2 [Moving Z-Score and Bayesian Changepoint Models].ipynb".
The first task is:
Training a Moving Z-Score Model
In this section we train a Moving Z-Score model to reveal where any anomalies exist in the fred_dcoilbrenteu time series.
Can you write Python code for it?

window_size = 252 # average trading days per year
model_moving_zscore =gl.anomaly_detection.moving_zscore.create(fred_dcoilbrenteu,
                                                               window_size, feature='VALUE')


I am working on the file "Dato-tutorials/marketing-analytics/Bank Marketing [Lead Scoring].ipynb".
The first task is:
A large proportion of customers who opened deposit accounts were employed (not students), under  38
Target them as leads and measure our ROI: Major improvement, 28.75%
Can you write Python code for it?

target_leadscore = test.apply(lambda x: 1 if x['age']<median_age
                              and ((x['job']!='unemployed') & (x['job']!='student') & (x['job']!='unknown'))
                              and x['y']==1 else 0)
age_targeting_roi = calc_call_roi(test, target_leadscore, 0.2)
print 'ROI for targeted calls [employed (not students) and age < %d (median)] to 20%% of contacts: %.2f%%' % (median_age, age_targeting_roi)


I am working on the file "Dato-tutorials/marketing-analytics/Bank Marketing [Lead Scoring].ipynb".
The first task is:
By comparing these two cases, we note that:

These two quadratic interactions:
cons.price.idx ∗ euribor3m
cons.price.idx ∗ emp.var.rate
enters the new_boostedtrees_model as significant attributes.

euribor3m and campaign features are significant in both cases whereas age is far less important in this new tweaked model.
Various characteristics concerning the campaign, such as the number of days that passed by after the client was last contacted from a previous campaign (pdays), the number of contacts performed before this campaign and for this client (previous), the channel through which the contact has been made (contact: 'telephone'), as well as the specific day_of_week ('mon') that this contact occured seems to be important for both models.
In this tweaked new_boostedtrees_model, attributes describing if the contact has a personal loan (loan) or credit in default (default), his/her educational (education) and employment status (job), as well as the outcome of the previous marketing campaign (poutcome) gets greater importance than before.

Lead score the contact list and measure our ROI: New Improvement achieved, 35.13%
As a last step of evaluating the new_boostedtrees_model, lets compute our ROI if we again contact 20% of the contact leads as scored by this new model.
Can you write Python code for it?

## show ROI for experimentation model
test1 = age_binning.transform(quadratic.transform(test))
boostedtrees_leadscore = new_boostedtrees_model.predict(test1, output_type='probability')
boostedtrees_roi = calc_call_roi(test1, boostedtrees_leadscore, 0.2)
print 'ROI for calling predicted contacts: %.2f%%' % boostedtrees_roi


I am working on the file "Dato-tutorials/marketing-analytics/Bank Marketing [Lead Scoring].ipynb".
The first task is:
Data Dictionary
The original dataset has the following attribute information:
| Field Num | Field Name | Description |
|---|---|---|
| 1 | age | (numeric) |
| 2 | job | type of job (categorical: 'admin.', 'blue-collar', 'entrepreneur', 'housemaid', 'management', 'retired', 'self-employed', 'services', 'student', 'technician', 'unemployed', 'unknown')|
| 3 | marital | marital status (categorical: 'divorced', 'married', 'single', 'unknown'; note: 'divorced' means divorced or widowed) |
| 4 | education | (categorical: 'basic.4y', 'basic.6y',  'basic.9y', 'high.school', 'illiterate', 'professional.course', 'university.degree', 'unknown') |
| 5 | default | has credit in default? (categorical: 'no', 'yes', 'unknown') |
| 6 | housing | has housing loan? (categorical: 'no', 'yes', 'unknown') |
| 7 | loan | has personal loan? (categorical: 'no', 'yes', 'unknown') |
|---|---|---|
| 8 | contact | contact communication type (categorical: 'cellular', 'telephone') |
| 9 | month | last contact month of year (categorical: 'jan', 'feb', 'mar', ..., 'nov', 'dec') |
| 10 | day_of_week | last contact day of the week (categorical: 'mon', 'tue', 'wed', 'thu', 'fri') |
| 11 | duration | last contact duration, in seconds (numeric). Important note: this attribute highly affects the output target (e.g., if duration=0 then y='no'). Yet, the duration is not known before a call is performed. Also, after the end of the call y is obviously known. Thus, this input should only be included for benchmark purposes and should be discarded if the intention is to have a realistic predictive model. |
|---|---|---|
| 12 | campaign | number of contacts performed during this campaign and for this client (numeric, includes last contact) |
| 13 | pdays | number of days that passed by after the client was last contacted from a previous campaign (numeric; 999 means client was not previously contacted) |
| 14 | previous | number of contacts performed before this campaign and for this client (numeric) |
| 15 | poutcome | outcome of the previous marketing...
Can you write Python code for it?


I am working on the file "Dato-tutorials/marketing-analytics/Bank Marketing [Lead Scoring].ipynb".
The first task is:
It is also important to note that the original data set has many more prospects (36548) than existent customers (4640). However, may be a bad idea to make a stratified split over this data set since we will loose that way the time dimension of the problem. In order to better check if the time dimension is important for this problem and the record provided, we need to re-create the missing calendar dates and transform the original data set in a timeseries object.
Data Transformation and train/test split
In the few lines of code below:

We add calendar dates (date) for the year, month and day of week of each provided record line, and produce the corresponding datetimes.
Transform the data set in a Timeseries object, and
Take a second look in the data we have available to check if the time-dimension is necessary for this problem.
Can you write Python code for it?

from helper_functions import *

def _month_to_number(x):
    from dateutil import parser
    return parser.parse(x).strftime('%m')

def _wkday_to_number(x):
    from dateutil import parser
    return parser.parse(x).strftime('%w')

def _str_to_datetime(x):
    import datetime
    import pytz
    from dateutil import parser
    return parser.parse(x).strftime('%Y-%m-%d')

def _unix_timestamp_to_datetime(x):
    import time
    import datetime
    import pytz
    from dateutil import parser
    return parser.parse(x)

bank_marketing['y'] = bank_marketing['y'].apply(lambda x: 1 if x=='yes' else 0)
bank_marketing['month_nr'] = bank_marketing['month'].apply(_month_to_number)
bank_marketing['wkday_nr'] = bank_marketing['day_of_week'].apply(_wkday_to_number)
bank_marketing['year'] = add_running_year(bank_marketing['month_nr'], 2008)
bank_marketing['date'] = add_running_date(bank_marketing, 'year', 'month_nr', 'wkday_nr')
bank_marketing['date'] = bank_marketing.apply(lambda row: '-'.join(map(str,(row['year'], row['month_nr'], row['date']))))
bank_marketing['date'] = bank_marketing['date'].apply(_str_to_datetime)
bank_marketing['date'] = bank_marketing['date'].apply(_unix_timestamp_to_datetime)

bank_marketing

bank_marketing = gl.TimeSeries(bank_marketing, index='date')


I am working on the file "Dato-tutorials/marketing-analytics/Bank Marketing [Lead Scoring].ipynb".
The first task is:
Lead Scoring Modeling
Part 1: Informed Decision
Targeting employed contacts with age less than 38 (median)
Usually middle-aged, employed people who have good annual earnings are much better prospects to contact and keep them informed for a new product. Indeed, as shown below 11.27% of the contacts in the training set opened a long-term deposit account, whereas 43.06% of them were employed (but not students) and had age less than 38.
Can you write Python code for it?

num_customers = float(len(train))
numY = gl.Sketch(train['y']).frequency_count(1)
print "%.2f%% of contacts in training set opened long-term deposit accounts." % (numY/num_customers * 100.0)

median_age = gl.Sketch(train['age']).quantile(0.5)

num_purchasing_emp_under_median_age = sum(train.apply(lambda x: 1 if x['age']<median_age
                                                      and ((x['job']!='unemployed') &
                                                           (x['job']!='student') &
                                                           (x['job']!='unknown'))
                                                      and x['y']==1 else 0))
probY_emp_under_median_age = (num_purchasing_emp_under_median_age / float(numY)) * 100.0

print "%.2f%% of the clients who opened long-term deposit accounts, were employed (but not students) and had age < %d (median)." % (probY_emp_under_median_age, median_age)


I am working on the file "Dato-tutorials/marketing-analytics/Bank Marketing [Lead Scoring].ipynb".
The first task is:
Next we add quadratic interactions between the four features below:
python
qfeatures0 = ['emp.var.rate','cons.price.idx','cons.conf.idx','euribor3m']
Can you write Python code for it?

## define a quadratic transformer object
quadratic_transformer = gl.feature_engineering.QuadraticFeatures(features=qfeatures0)

## fit the quadratic transformer object over the train set
quadratic = gl.feature_engineering.create(train, quadratic_transformer)

## transform the train data set
qtrain = quadratic.transform(train)

## remove the features that may worse our predictions
qfeatures = qtrain.column_names()
qfeatures.remove('duration')
qfeatures.remove('y')
qfeatures.remove('month_nr')
qfeatures.remove('wkday_nr')
qfeatures.remove('year')

qtrain.head(5)


I am working on the file "Dato-tutorials/marketing-analytics/Bank Marketing [Lead Scoring].ipynb".
The first task is:
Next we evaluate the new_boostedtrees_model on the test data set.
Can you write Python code for it?

results = new_boostedtrees_model.evaluate(age_binning.transform(quadratic.transform(test)))
print "accuracy: %.5f, precision: %.5f, recall: %.5f" % (results['accuracy'], results['precision'], results['recall'])


I am working on the file "Dato-tutorials/marketing-analytics/Bank Marketing [Lead Scoring].ipynb".
The first task is:
Note that this model is almost as accurate as the previous one, with similar precision (~66% of the predicted sales were actually converted to sales) and recall (~24% of actual sales were actually predicted by the model). However, to have a better feeling of the model just trained (newtoolkit_model) and how this differs from the previous one (toolkit_model), we can review the importance of the input features in these two cases.
Can you write Python code for it?

print '\'newtoolkit_model\'\n[GLC AutoML Classifier wt quadratic interactions]:\n'
print new_toolkit_model.get_feature_importance()

print '\'toolkit_model\'\n[GLC AutoML Classifier wo quadratic interactions]:\n'
print toolkit_model.get_feature_importance()


I am working on the file "Dato-tutorials/marketing-analytics/Bank Marketing [Lead Scoring].ipynb".
The first task is:
Result:

ROI for 20% of targeted contacts (employed and not students, had age < 38): 28.75%

Major improvement over the ROI achieved by calling EVERYONE in list

Calling everyone in list does not have greater ROI!

Part 2: Train a Machine Learning model instead
Learn from ALL features, not just age or status of employment
GraphLab Create AutoML to choose the most effective classifer model automatically!
Dato's classifier toolkit can choose the most effective classifier model automatically.
Can you write Python code for it?

## remove features that introduce noise in ML prediction
features = train.column_names()
features.remove('duration')
features.remove('y')
features.remove('month_nr')
features.remove('wkday_nr')
features.remove('year')

## GLC AutoML Classifier
toolkit_model = gl.classifier.create(train, features=features, target='y')


I am working on the file "Dato-tutorials/marketing-analytics/Bank Marketing [Lead Scoring].ipynb".
The first task is:
Result:

ROI for 20% of contacts as sorted by the descending lead score which was returned by the AutoML Classifier: 34.87%.
Huge improvement (3x greater) over the ROI achieved by calling EVERYONE in list.
Improved ROI (6.12% greater) than the ROI achieved by calling 20% of the targeted contacts of Part 1 above (employed, middle-aged people).

Part 3: Tweak the ML Classifier
Adding quadratic interactions, age grouping, tweak parameters
The scatter plots diagrams among the various int and float variables of the train data set, and more specifically the four plots below, suggest to investigate if interactions between these features exist:
python
qfeatures0 = ['emp.var.rate','cons.price.idx','cons.conf.idx','euribor3m']
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline

qfeatures0 = ['emp.var.rate','cons.price.idx','cons.conf.idx','euribor3m']

plt.figure(figsize=(10,10))
subplot_idx = 1
for attrib1 in qfeatures0:
    for attrib2 in qfeatures0:
        if(attrib2 != attrib1):
            if subplot_idx < 5:
                plt.subplot(2,2,subplot_idx)
                plt.scatter(train[attrib1], train[attrib2])
                plt.xlabel(attrib1)
                plt.ylabel(attrib2)
                plt.title('\'%s\' vs \'%s\'' % (attrib1, attrib2))
                subplot_idx +=1
plt.show()


I am working on the file "Dato-tutorials/marketing-analytics/Bank Marketing [Lead Scoring].ipynb".
The first task is:
Result:

ROI for 20% of the leads returned by the AutoML Classifier [wt quadratic interactions]: 34.87%.
Improved ROI (0.14% greater) than the ROI achieved by calling 20% of the leads returned by the AutoML Classifier [wo quadratic interactions].

GraphLab Create Boosted Trees Classifier:
Contacts' age grouping, hyperparameters fine-tuning
As it is obvious from the histogram below most of our contacts were between 30 to 36 years old, 4518 contacts were between 36 to 43, 2931 people between 43 to 50 and about 2500 people between 23 to 30 and 50 to 56 years old. The remaining contacts were either younger or older than these ages, but certainly not more than 1000 cases or so in a specific age group. Therefore, grouping the age values of our contacts into a pre-defined number of bins may be beneficial for the learning algorithm of choice, and may improve the ROI of our telemarketing campaign even further.
Can you write Python code for it?


I am working on the file "Dato-tutorials/marketing-analytics/Bank Marketing [Lead Scoring].ipynb".
The first task is:
The provided data set, bank_marketing, has 41188 record lines describing various customers and prospects attributes, as well as their response in the telemarketing campaign of interest. The percentage of unique calendar dates across this record is low, whereas much more people seems to response positevely as the time goes by. However, some months are missing from the data set and adding time dimension in this problem cannot help to provide better predictions.
Can you write Python code for it?

print 'Number of record lines [bank_marketing]: %d' % len(bank_marketing)
print 'Unique calendar dates across data set [bank_marketing]: %d' % len(bank_marketing['date'].unique())
unique_dates_pct = (len(bank_marketing['date'].unique())*100/float(len(bank_marketing)))
print 'Percentage of unique calendar dates across data set [bank_marketing]: %.2f%%'% unique_dates_pct

bank_marketing.filter_by(2008,'year')['month_nr'].unique().sort()

print 'Full Data Set [year: 2008]:'
print '------------------------------'
bank_marketing_2008 = bank_marketing.filter_by(2008,'year')
customers = len(bank_marketing_2008[bank_marketing_2008['y']==1])
prospects = len(bank_marketing_2008[bank_marketing_2008['y']==0])
print 'Number of examples in year segment [bank_marketing]: %d' % len(bank_marketing_2008)
print 'Number of existent customers: %d (%.2f%%)' % (customers, 100*customers/float(len(bank_marketing_2008)))
print 'Number of prospects: %d (%.2f%%)\n' % (prospects, 100*prospects/float(len(bank_marketing_2008)))

bank_marketing.filter_by(2009,'year')['month_nr'].unique().sort()

print 'Full Data Set [year: 2009]:'
print '------------------------------'
bank_marketing_2009 = bank_marketing.filter_by(2009,'year')
customers = len(bank_marketing_2009[bank_marketing_2009['y']==1])
prospects = len(bank_marketing_2009[bank_marketing_2009['y']==0])
print 'Number of examples in year segment [bank_marketing]: %d' % len(bank_marketing_2009)
print 'Number of existent customers: %d (%.2f%%)' % (customers, 100*customers/float(len(bank_marketing_2009)))
print 'Number of prospects: %d (%.2f%%)\n' % (prospects, 100*prospects/float(len(bank_marketing_2009)))

bank_marketing.filter_by(2010,'year')['month_nr'].unique().sort()

print 'Full Data Set [year: 2010]:'
print '------------------------------'
bank_marketing_2010 = bank_marketing.filter_by(2010,'year')
customers = len(bank_marketing_2010[bank_marketing_2010['y']==1])
prospects = len(bank_marketing_2010[bank_marketing_2010['y']==0])
print 'Number of examples in year segment...


I am working on the file "Dato-tutorials/marketing-analytics/Bank Marketing [Lead Scoring].ipynb".
The first task is:
The toolkit automatically evaluates several types of algorithms, including: Boosted Trees, Random Forests, Decision Trees, Support Vector Machines, Logistic regression - with intelligent default paramters. Based on a validation set, it chooses the most accurate model which in our case is a Boosted Trees Classifier. We can then evaluate this model on the test data set.
Can you write Python code for it?

results = toolkit_model.evaluate(test)
print "accuracy: %.5f, precision: %.5f, recall: %.5f" % (results['accuracy'], results['precision'], results['recall'])


I am working on the file "Dato-tutorials/marketing-analytics/Bank Marketing [Lead Scoring].ipynb".
The first task is:
This initial model can be considered accurate given that it correctly predicts the purchasing decisions of ~90% of the contacts. However, the toolkit_model leaves room for improvement. Specifically only ~66% of predicted sales actually convert to sales. Furthermore, only ~24% of actual sales were actually predicted by the model. In order to better understand the model we can review the importance of the input features.
Can you write Python code for it?


I am working on the file "Dato-tutorials/marketing-analytics/Bank Marketing [Lead Scoring].ipynb".
The first task is:
This new model (new_boostedtrees_model) is almost as accurate as the previous one, has higher precision (~66% of the predicted sales were actually converted to sales) and similar recall (~23% of actual sales were actually predicted by the model). To have a better feeling of the model just trained (newtoolkit_model) and how this differs from the previous one (new_toolkit_model), we can review the importance of the input features in these two cases.
Can you write Python code for it?

print '\'new_boostedtrees_model\'\n[GLC Boosted Trees Classifier wt quadratic interactions,\
age grouping & hyperparams tuned]:\n'
new_boostedtrees_model.get_feature_importance().print_rows(num_rows=20)

print '\'newtoolkit_model\'\n[GLC AutoML Classifier wt quadratic interactions]:\n'
print new_toolkit_model.get_feature_importance()


I am working on the file "Dato-tutorials/marketing-analytics/Bank Marketing [Lead Scoring].ipynb".
The first task is:
To group the age values of our contacts we leverage the FeatureBinner method of the feature_engineering  toolkit of GraphLab Create as shown below.
Can you write Python code for it?

## define a binning transformer for the age attribute of contacts
age_binning_transformer = gl.feature_engineering.FeatureBinner(features='age', strategy='quantile', num_bins=12)

## fit the age binning transformer over the train set
age_binning = gl.feature_engineering.create(train, age_binning_transformer)

## transform the train data set
qtrain1 = age_binning.transform(qtrain)

## remove the features that may worse our predictions
qfeatures1 = qtrain1.column_names()
qfeatures1.remove('duration')
qfeatures1.remove('y')
qfeatures1.remove('month_nr')
qfeatures1.remove('wkday_nr')
qfeatures1.remove('year')

qtrain1['age'].show()


I am working on the file "Dato-tutorials/marketing-analytics/Bank Marketing [Lead Scoring].ipynb".
The first task is:
and re-train the GraphLab Create AutoML Classifier for this new data set qtrain.
Can you write Python code for it?

new_toolkit_model = gl.classifier.create(qtrain, target='y', features=qfeatures)


I am working on the file "Dato-tutorials/pattern-mining/Pattern Mining - Association Rule Mining.ipynb".
The first task is:
A simple retailer example: Loading Data, Exploratory Data Analysis
Here we discuss a simple example of receipt data from a bakery. The dataset consists of items like 'ApplePie' and 'GanacheCookie'. The task is to identify sets of items that are frequently bought together. The dataset consists of 266209 rows and 6 columns which look like the following. The dataset was constructed by modifying the Extended BAKERY dataset.
Can you write Python code for it?

bakery_sf = gl.SFrame('./bakery_sf')
bakery_sf


I am working on the file "Dato-tutorials/pattern-mining/Pattern Mining - Association Rule Mining.ipynb".
The first task is:
Alternatively, by decreasing the min_support one can obtain more patterns of sold coffee products which are again assumed frequent but with respect to this new threshold.
Can you write Python code for it?

min_support = int(train.num_rows()*(1e-04))

model = gl.frequent_pattern_mining.create(train, 'Item',
                                          features=['Receipt', 'StoreNum'],
                                          min_support=min_support,
                                          max_patterns=500,
                                          min_length=4)

print 'The most frequent feature patters are:'
print '-----------------------------------------'
model.frequent_patterns.print_rows(num_rows=60, max_row_width=90, max_column_width=80)


I am working on the file "Dato-tutorials/pattern-mining/Pattern Mining - Association Rule Mining.ipynb".
The first task is:
As we can see below, all the coffee products have similar sale frequencies and there is no some particular subset of products that is more preferred than the remaining ones.
Can you write Python code for it?

%matplotlib inline
item_freq_plot(bakery_sf, 'Item', ndigits=3, topk=30,
               seaborn_style='whitegrid', seaborn_palette='deep', color='b')


I am working on the file "Dato-tutorials/pattern-mining/Pattern Mining - Association Rule Mining.ipynb".
The first task is:
Example: Employee Space Clustering by using occurrences of frequency patterns
First, we provide an aggregated form of our data by selecting one Selling Employee (EmpId) at random.
Can you write Python code for it?

emps = train.groupby(['Receipt', 'StoreNum'],
                     {'EmpId': agg.SELECT_ONE('EmpId')})
emps


I am working on the file "Dato-tutorials/pattern-mining/Pattern Mining - Association Rule Mining.ipynb".
The first task is:
Feature Engineering to help other ML Tasks in Pipeline
The .extract_feature() method
Using the set of closed patterns, we can convert pattern data to binary features vectors. These feature vectors can be used for other machine learning tasks, such as clustering or classification. For each input pattern x, the j-th extracted feature f_{x}[j] is a binary indicator of whether the j-th closed pattern is contained in x.
First, we train the top100_freq_patterns model as shown below:
Can you write Python code for it?

top100_freq_patterns = gl.frequent_pattern_mining.\
create(train, 'Item',
       features=['Receipt', 'StoreNum'],
       # occurs at least once in our data record
       min_support=1,
       # do not search for more than 100 patterns
       max_patterns = 100,
       # test data have only one coffee product sold per tid .
       # We search for patterns of at least 2 coffee products
       min_length=2)


I am working on the file "Dato-tutorials/pattern-mining/Pattern Mining - Association Rule Mining.ipynb".
The first task is:
Finally, we train a kmeans algorithm to produce a 3 centered cluster.
Can you write Python code for it?

cl_model = gl.kmeans.create(emp_space,
                            features = ['all_features'],
                            num_clusters=3)

emp_space['cluster_id'] = cl_model['cluster_id']['cluster_id']
emp_space


I am working on the file "Dato-tutorials/pattern-mining/Pattern Mining - Association Rule Mining.ipynb".
The first task is:
Here are the 100 unique closed patterns which are found frequent:
Can you write Python code for it?

top100_freq_patterns.frequent_patterns.\
print_rows(num_rows=100, max_row_width=90, max_column_width=80)


I am working on the file "Dato-tutorials/pattern-mining/Pattern Mining - Association Rule Mining.ipynb".
The first task is:
Next, we apply the extract_features() method of this newly trained top100_freq_patterns model on the test data set.
Can you write Python code for it?


I am working on the file "Dato-tutorials/pattern-mining/Pattern Mining - Association Rule Mining.ipynb".
The first task is:
Next, we count the instances that each of the top100_freq_patterns occurs per EmpId.
Can you write Python code for it?

emp_space = emps.join(features).\
groupby('EmpId', {'all_features': agg.SUM('extracted_features')})

emp_space


I am working on the file "Dato-tutorials/pattern-mining/Pattern Mining - Association Rule Mining.ipynb".
The first task is:
To see some details of the trained model:
Can you write Python code for it?


I am working on the file "Dato-tutorials/pattern-mining/Pattern Mining - Association Rule Mining.ipynb".
The first task is:
Top-k frequent patterns
In practice, we rarely know the appropriate min_support threshold to use. As an alternative to specifying a minimum support, we can specify a maximum number of patterns to mine using the max_patterns parameter. Instead of mining all patterns above a minimum support threshold, we mine the most frequent patterns until the maximum number of closed patterns are found. For large data sets, this mining process can be time-consuming. We recommend specifying a somehow large initial minimum support bound to speed up the mining.
Can you write Python code for it?

min_support = int(train.num_rows()*1e-03)

top5_freq_patterns = gl.frequent_pattern_mining.create(train, 'Item',
                                                       features=['Receipt', 'StoreNum'],
                                                       min_support=min_support,
                                                       max_patterns=5,
                                                       min_length=4)


I am working on the file "Dato-tutorials/sentiment-analysis/Product Reviews [Applying NLP Pipeline].ipynb".
The first task is:
Comparing the sentence-level sentiment for each aspect of each product:
Can you write Python code for it?


I am working on the file "Dato-tutorials/sentiment-analysis/Product Reviews [Applying NLP Pipeline].ipynb".
The first task is:
Comparing the use of adjectives for each aspect:
Can you write Python code for it?


I am working on the file "Dato-tutorials/sentiment-analysis/Product Reviews [Applying NLP Pipeline].ipynb".
The first task is:
Feature engineering: Applying NLP Pipeline
Can you write Python code for it?

def nlp_pipeline(reviews, title, aspects):

    from graphlab.toolkits.text_analytics import trim_rare_words, split_by_sentence, extract_parts_of_speech, stopwords, PartOfSpeech

    print(title)

    print('1. Get reviews for this product')
    reviews = reviews.filter_by(title, 'name')

    print('2. Splitting reviews into sentences')
    reviews['sentences'] = split_by_sentence(reviews['review'])
    sentences = reviews.stack('sentences', 'sentence').dropna()

    print('3. Tagging relevant reviews')
    tags = gl.SFrame({'tag': aspects})
    tagger_model = gl.data_matching.autotagger.create(tags, verbose=False)
    tagged = tagger_model.tag(sentences, query_name='sentence', similarity_threshold=.3, verbose=False)\
                         .join(sentences, on='sentence')

    print('4. Extracting adjectives')
    tagged['cleaned']    = trim_rare_words(tagged['sentence'], stopwords=list(stopwords()))
    tagged['adjectives'] = extract_parts_of_speech(tagged['cleaned'], [PartOfSpeech.ADJ])

    print('5. Predicting sentence-level sentiment')
    model = gl.sentiment_analysis.create(tagged, target=None, features=['review'])
    tagged['sentiment']  = model.predict(tagged)
    return tagged

reviews = gl.SFrame('./amazon_baby.gl/')

reviews


I am working on the file "Dato-tutorials/sentiment-analysis/Product Reviews [Applying NLP Pipeline].ipynb".
The first task is:
Focus on chosen aspects about baby monitors
First, we import helper function from helper_util.py file.
Can you write Python code for it?


I am working on the file "Dato-tutorials/sentiment-analysis/Product Reviews [Applying NLP Pipeline].ipynb".
The first task is:
Investigating good and bad sentences
Can you write Python code for it?

good, bad = get_extreme_sentences(reviews_a)


I am working on the file "Dato-tutorials/sentiment-analysis/Product Reviews [Applying NLP Pipeline].ipynb".
The first task is:
Next, we collect the baby monitor reviews:
Can you write Python code for it?

reviews = search(reviews, 'monitor')

reviews.print_rows(num_rows=100,max_row_width=300)

for review in reviews['review'][0:10]:
    print review, '\n'


I am working on the file "Dato-tutorials/sentiment-analysis/Product Reviews [Applying NLP Pipeline].ipynb".
The first task is:
Note, that we have grouped the result by the product 'name' but we have limit summaries to only those having at least three (3) of product reviews.
Under the hood
While creating the model, several operations are completed under the hood:

A data structure is created that helps facilitate searching for text snippets, including doing a TF-IDF transform of the text and creating an inverted index.
Each piece of text is tokenized into sentences using NLTK's punkt sentence parser.
A pre-trained sentiment classifier scores all reviews (or sentences) and stores these scores within the model.

Providing a splitby='sentence' argument when training the model implies that all analysis should be performed at the sentence-level rather than using the entire text. Thus any calls to sentiment_summary() will concern predictions for each sentence rather than the entire review.
Can you write Python code for it?

reviews_sentiment1 = gl.product_sentiment.create(reviews,
                                                 target=None,
                                                 features=['review'],
                                                 method='auto',
                                                 splitby='sentence')

reviews_sentiment1.sentiment_summary(keywords=aspects, groupby='name', k=10, threshold=3)


I am working on the file "Dato-tutorials/sentiment-analysis/Product Reviews [Applying NLP Pipeline].ipynb".
The first task is:
Print good sentences for the first item, where adjectives and aspects are highlighted.
Can you write Python code for it?


I am working on the file "Dato-tutorials/sentiment-analysis/Product Reviews [Applying NLP Pipeline].ipynb".
The first task is:
Process reviews for the most common product
First, we define the aspects (product properties) of our current interest:
Can you write Python code for it?

aspects = ['audio', 'price', 'signal', 'range', 'battery life']

item_a = 'Snuza Baby Monitor, Hero'
reviews_a = nlp_pipeline(reviews, item_a, aspects)

reviews_a

reviews_a.save('./reviews_a')


I am working on the file "Dato-tutorials/sentiment-analysis/Product Reviews [Applying NLP Pipeline].ipynb".
The first task is:
To have an overview of the model that searches for text snippets:
Can you write Python code for it?


I am working on the file "Dato-tutorials/sentiment-analysis/Product Reviews [Applying NLP Pipeline].ipynb".
The first task is:
we make the call:
Can you write Python code for it?

reviews_sentiment.sentiment_summary(keywords=aspects, groupby='name', k=10, threshold=3)


I am working on the file "Day 1 - Unit 1.1.ipynb".
The first task is:
1. Your first steps with Python

1.1 Introduction
Python is a general purpose programming language. It is used extensively for scientific computing, data analytics and visualization, web development and software development. It has a wide user base and excellent library support.
There are many ways to use and interact with the Python language. The first way is to access it directly from the command prompt and calling python <script>.py. This runs a script written in Python and does whatever you have programmed the computer to do. But scripts have to be written and how do we actually write Python scripts?
Actually Python scripts are just .txt files. So you could just open a .txt file and write a script, saving the file with a .py extension. The downsides of this approach is obvious to anyone working with Windows. Usually, Python source code is written-not with Microsoft Word- but with and Integrated Development Environment. An IDE combines a text editor with a running Python console to test code and actually do work with Python without switching from one program to another. If you learnt the C, or C++ language, you will be familiar with Vim. Other popular IDE's for Python are Pycharm, Spyder and the Jupyter Notebook.
In this course, we will use the Jupyter Notebook as our IDE because of its ease of use ability to execute code cell by cell. It integrates with markdown so that one can annotate and document your code on the fly! All in all, it is an excellent tool for teaching and learning Python before one migrates to more advanced tools like Spyder for serious scripting and development work.
1.2 Your best friends.
In order to get the most from Python, your best source of reference is the
Python documentation. Getting good at Python is a matter using it regularly and familiarizing yourself with the keywords, constructs and commonly used idioms.
Learn to use the Shift-Tab when coding. This activates a hovering tooltip that provides documentation for keywords, functions and even variables that you have...
Can you write Python code for it?

# change this cell into a Markdown cell. Then type something here and execute it (Shift-Enter)


I am working on the file "Day 1 - Unit 1.1.ipynb".
The first task is:
2.2 Your first script
It is a time honoured tradition that your very first program should be to print "Hello world!" How is this achieved in Python?
Can you write Python code for it?

'''Make sure you are in "edit" mode and that this cell is for Coding ( You should see the In [ ]:)
on the left of the cell.  '''

print("Hello world!")


I am working on the file "Day 1 - Unit 1.1.ipynb".
The first task is:
2.3 Commenting
Commenting is a way to annotate and document code. There are two ways to do this: Inline using the # character or by using ''' <documentation block> ''', the latter being multi-line and hence used mainly for documenting functions or classes. Comments enclosed using '''  '''' style commenting are actually registed in Jupyter notebook and can be accessed from the Shift-Tab tooltip!
One should use # style commenting very sparingly. By right, code should be clear enough that # inline comments are not needed.
However, # has a very important function. It is used for debugging and trouble-shooting. This is because commented code sections are never executed when you execute a cell (Shift-Enter)
3. Python's building blocks

Python is an Object Oriented Programming language. That means to all of python is made out of objects which are instances of classes. The main point here is that I am going to introduce 4 basic objects of Python which form the backbone of any program or script.

Integers or int.
Strings or str. You've met one of these: "Hello world!". For those who know about character encoding, it is highly encouraged to code Python with UTF-8 encoding.
Float or float. Basically the computer version of real numbers.
Booleans or bool. In Python, true and false are indicated by the reserved keywords True and False. Take note of the capitalized first letter.

3.1 Numbers
You can't call yourself a scientific computing language without the ability to deal with numbers. The basic arithmetic operations for numbers are exactly as you expect it to be
Can you write Python code for it?

# Addition
5+3

# Subtraction
8-9

# Multiplication
3*12

# Division
48/12


I am working on the file "Day 1 - Unit 1.1.ipynb".
The first task is:
3.1.3 Floating point precision
All of the above applies equally to floating point numbers (or real numbers). However, we must be mindful of floating point precision.
Can you write Python code for it?


I am working on the file "Day 1 - Unit 1.1.ipynb".
The first task is:
4. list, here's where the magic begins

list are the fundamental data structure in Python. These are analogous to arrays in C or Java. If you use R, lists are analogous to vectors (and not R list)
Declaring a list is as simple as using square brackets [ ] to enclose a list of objects (or variables) seperated by commas.
Can you write Python code for it?

# Here's a list called staff containing his name, his age and current renumeration

staff = ['Andy', 28, 980.15]


I am working on the file "Day 1 - Unit 1.1.ipynb".
The first task is:
4.3 List methods
Right now, let us look at four very useful list methods. Methods are basically operations which modify lists. These are:

pop which allows us to remove an item in a list.

So for example if $x_0, x_1, \ldots, x_n$ are items in a list, calling my_list.pop(r) will modify the list so that it contains only $$x_0, \ldots, x_{r-1}, x_{r+1},\ldots, x_n$$ while returning the element $x_r$.

append which adds items to the end of the list.

Let's say $x_{n+1}$ is the new object you wish to append to the end of the list. Calling the method my_list.append(x_n+1) will modify the list inplace so that the list will now contain $$x_0, \ldots, x_n, x_{n+1}$$ Note that append does not return any output!

insert which as the name suggests, allows us to add items to a list in a particular index location

When using this, type my_list.insert(r, x_{n+1}) with  the second argument to the method the object you wish to insert and r the position (still 0 indexed) where this object ought to go in that list. This method modifies the list inplace and does not return any output. After calling the insert method, the list now contains $$x_0,\ldots, x_{r-1}, x_{n+1}, x_{r}, \ldots, x_n$$ This means that my_list[r] = $x_{n+1}$ while my_list[r+1] = $x_{r}$

+ is used to concatenate two lists. If you have two lists and want to join them together producing a union of two (or more lists), use this binary operator.

This works by returning a union of two lists. So $$[ x_1,\ldots, x_n] + [y_1,\ldots, y_m]$$ is the list containing $$ x_1,\ldots, x_n,y_1, \ldots, y_m$$ This change is not permanent unless you assign the result of the operation to another variable.
Can you write Python code for it?

# append

staff.append('Finance')
print(staff)

# pop away the information about his salary

andys_salary = staff.pop(2)
print(andys_salary)
print(staff)

# oops, made a mistake, I want to reinsert information about his salary

staff.insert(3, andys_salary)
print(staff)

contacts = [99993535, "andy@company.com"]

staff = staff+contacts # reassignment of the concatenated list back to staff
print(staff)


I am working on the file "Day 1 - Unit 1.1.ipynb".
The first task is:
Answer: eciuj elppA
3.3 The type function
All objects in python are instances of classes. It is useful sometimes to find out what type of object we are looking at, especially if it has been assigned to a variable. For this we use the type function.
Can you write Python code for it?

x = 5.0
type(x)

type(favourite_drink)

type(True)

type(500)


I am working on the file "Day 1 - Unit 1.1.ipynb".
The first task is:
In the above 5%2 means return me the remainder after 5 is divided by 2 (which is indeed 1).
3.1.1 Precedence
A note on arithmetic precedence. As one expects, () have the highest precedence, following by * and /. Addition and subtraction have the lowest precedence.
Can you write Python code for it?

# Guess the output before executing this cell. Come on, don't cheat!

6%(1+3)


I am working on the file "Day 1 - Unit 1.1.ipynb".
The first task is:
Note the floating point answer. In previous versions of Python, / meant floor division. This is no longer the case in Python 3
Can you write Python code for it?

# Exponentiation. Limited precision though!

16**0.5

# Residue class modulo n

5%2


I am working on the file "Day 1 - Unit 1.1.ipynb".
The first task is:
Notice that Hello world! is printed at the bottom of the cell as an output. In general, this is how output of a python code is displayed to you.
print is a special function in Python. It's purpose is to display output to the console. Notice that we pass an argument-in this case a string "Hello world!"- to the function. All arguments passed to the function must be enclosed in round brackets and this signals to the Python interpreter to execute a function named print with the argument "Hello world!".
2.2.1 Self introductions
Your next exercise is to print your own name to the console. Remember to enclose your name in " " or ' '
Can you write Python code for it?


I am working on the file "Day 1 - Unit 1.1.ipynb".
The first task is:
Notice that Python still outputs to console even though we did not use the print function. Actually the print function prints a particularly "nice" string representation of the object, which is why Andy is printed without the quotation marks if print was used.
Can you find me Andy's age now?
Can you write Python code for it?

# type your answer here and run the cell


I am working on the file "Day 1 - Unit 1.1.ipynb".
The first task is:
Notice that after assignment, I can access the variables in a different cell. However, if you reassign a variable to a different number, the old values for that variable are overwritten.
Can you write Python code for it?


I am working on the file "Day 1 - Unit 1.1.ipynb".
The first task is:
Notice that if I type nested_list[2], Python will return me the list [1.50, .40]. This can be accessed again using indexing (or slicing notation) [ ].
Can you write Python code for it?

# Accesing items from within a nested list structure.

print(nested_list[2])

# Assigning nested_list[2] to a variable. The variable price represents a list

price = nested_list[2]
print(type(price))

# Getting the smaller of the two floats
print(nested_list[2][1])


I am working on the file "Day 1 - Unit 1.1.ipynb".
The first task is:
Slicing is a way of get specific subsets of the string. If you let $x_n$ denote the $n+1$-th letter (note zero indexing) in a string (and by letter this includes whitespace characters as well!) then writing my_string[i:j] returns a subset $$x_i, x_{i+1}, \ldots, x_{j-1}$$ of letters in a string. That means the slice [i:j] takes all subsets of letters starting from index i and stops one index before the index indicated by j.
0 indexing and stopping point convention frequently trips up first time users. So take special note of this convention. 0 indexing is used throughout Python especially in matplotlib and pandas.
Can you write Python code for it?

favourite_drink = fruit+' '+drink
print("Printing the first to 3rd letter.")
print(favourite_drink[0:3])
print("\nNow I want to print the second to seventh letter:")
print(favourite_drink[1:7])


I am working on the file "Day 1 - Unit 1.1.ipynb".
The first task is:
So what happened above? Well [3:0:-1] means that starting from the 4-th letter $x_3$ which is 'l' return a subtring including $x_{2}, x_{1}$ as well. Note that the progression does not include $x_0 =$ 'A' because the stopping point is non-inclusive of j.
The slice [:j] or [i:] means take substrings starting from the beginning up to the $j$-th letter (i.e. the $x_{j-1}$ letter) and substring starting from the $i+1$-th (i.e. the $x_{i}$) letter to the end of the string.
3.2.3 A mini challenge
Print the string favourite_drink in reverse order. How would you do it?
Can you write Python code for it?

# Write your answer here and check it with the output below


I am working on the file "Day 1 - Unit 1.1.ipynb".
The first task is:
When using % to indicate string substitution, take note of the common formatting "placeholders"

%s to substitue strings.
%d for printing integer substitutions
%.1f means to print a floating point number up to 1 decimal place. Note that there is no rounding

The utility of the .format method arises when the same string needs to printed in various places in a larger body of text. This avoids duplicating code. Also did you notice I used double quotation. Why?
More about string formats can be found in this excellent blog post
3.2.2 Weaving strings into one beautiful tapestry of text
Besides the .format and % operation on text, we can concatenate strings using + operator. However, strings cannot be changed once declared and assigned to variables. This property is called immutability
Can you write Python code for it?

fruit = 'Apple'
drink = 'juice'

print(fruit+drink) # concatenation

#Don't like the lack of spacing between words?

print(fruit+' '+drink)


I am working on the file "Day 1 - Unit 1.2.ipynb".
The first task is:
6.2 Updating dictionaries
Very often, we need to change dictionary values and/or add more entries to our dictionary.
Can you write Python code for it?

# Hey, Andy mistakenly keyed in his age. He is actually 29 years old!

staff['age'] = 29
print(staff)

# HR wants us to record down his staff ID.

staff['id'] = 12345
print(staff)

# Let's check the list of keys
staff.keys()


I am working on the file "Day 1 - Unit 1.2.ipynb".
The first task is:
6.2.1 Dictionary exercise
Here's what I want you to do. Go around the room and find out the favourite food of three colleagues here. Then update this empty dictionary with 3 key-value pairings where keys are your colleague's name and value his/her favourite food. Have fun!
Can you write Python code for it?

# update your dictionary here

# and print the dictionary
print(favourite_food)


I am working on the file "Day 1 - Unit 1.2.ipynb".
The first task is:
6.2.2 Using the .update method
To combine two dictionaries, we use the .update method.
Can you write Python code for it?

staff.update({'salary': 980.15, 'department':'finance', 'colleagues': ['George', 'Liz']})

# Who are Andy's colleagues? Enter answer below

# Which department does he work in? Enter answer below


I am working on the file "Day 1 - Unit 1.2.ipynb".
The first task is:
6.2.3 Creating dictionaries using kwargs
Dictionaries can also be created by calling dict() with keyword arguments (or kwargs). When we create dictionaries like this, the key value pairs are written as
$$ key_1 = value_1, key_2 = value_2, \ldots $$ again each seperated by a comma.
Can you write Python code for it?

my_favourite_things = dict(food="Assam Laksa", music="classical", number = 2)

# my favourite number

my_favourite_things['number']


I am working on the file "Day 1 - Unit 1.2.ipynb".
The first task is:
7. If and boolen conditionals

Any algorithm worth its salt will have conditionals in it. Very rarely does an algorithm just proceed in a linear fashion, one step following another step. More often than not, one step will follow from another if some condition is satisfied.
This situation can be programmed in Python by using the if keyword. The general construct of an if statement looks like this:
if condition :
do something

Python will only execute the code block do something only if condition evaluate to True. Otherwise it ignores it. Notice the lack of { } surrounding the code block to be executed. The Python interpreter relies on indentation and newline to "know" which code block is conditioned on.
This enforces good coding style and makes it so nice to program with Python. No more worrying about unmatched braces!
7.1 Conditionals
Since conditionals are logical statements, we need some binary comparison operators

== logical equals to
> strictly greater than
< strictly less than
!= logical not equals to

We can create compound statements by using the keywords or and and. Their bitwise versions are | and &.
Can you write Python code for it?

# examples of binary comparison

1 < 2

# compound statements

1 < 2 or 1 == 2

# using bitwise operators

1<2 & 1==2


I am working on the file "Day 1 - Unit 1.2.ipynb".
The first task is:
8.2.1 Your mission, should you choose to accept it
...is to list down all prime numbers less than 100.
A prime number $p$ is a number which is divisible only by 1 and itself.
Can you write Python code for it?


I am working on the file "Day 1 - Unit 1.2.ipynb".
The first task is:
An implication of this is that we can start with an empty dictionary and add keys as we go along.
And empty dictionary is created by assigning a variable to an instance of class dict by calling dict().
Can you write Python code for it?

favourite_food = dict() # You could also type favourite_food = {}
print(favourite_food)


I am working on the file "Day 1 - Unit 1.2.ipynb".
The first task is:
As before, play around with the various values of marks to make sure that elif structure is working as intended. Notice that I phrased the conditional only to check agains a lower bound. This is because Python will only execute the code block corresponding to first True conditional in the elif sequence. Even if subsequent conditionals evaluate to true, their code is not run.
8. Performing repetitive tasks. Using the for loop

Most algorithms consist of repeating a certain calculations or computer tasks in an almost similiar manner. To give a scenario, I could instruct Python to print the names of three staff members to the output. I could go about fulfilling this task by writing the following code
print("Staff member Lisa")
print("Staff member Mark")
print("Staff member Andy")

Clearly this is repetitive and tedious. The for keyword allows us to simplify code by writing a single intruction - here print and looping this over a list consisting of staff names [Lisa, Mark, Andy].
The general format of a for loop is given by the following:
for item_n in list:

    do code substituted with item_n

8.1 How a for loop works
Let's start by doing the above by looping over the index of the list. The way it is usually done in Javascript or C is to  declare a "counter variable" i and increment it by one to perform the next loop of the algorithm. We could do something like this in Python.
Can you write Python code for it?

staff = ['Lisa', 'Mark', 'Andy']

for i in range(0,3): # range(0,3) is a function that produces a sequence of numbers in the form of a list: [0,1,2]
    print("Staff member "+staff[i])


I am working on the file "Day 1 - Unit 1.2.ipynb".
The first task is:
But Python allows us to write a more readable form of the for loop. So the following is equivalent to the above and is preferred.
Can you write Python code for it?

for staff_name in staff:
    print("Staff member "+ staff_name)


I am working on the file "Day 1 - Unit 1.2.ipynb".
The first task is:
Here is our first example using the if keyword.
Can you write Python code for it?

x = 300

if x == 300:
    print('This is Sparta!')


I am working on the file "Day 1 - Unit 1.2.ipynb".
The first task is:
Note from the example above that when creating dictionaries, DO NOT enclose the keys within quotation marks. However, when accessing the value of a dictionary by its key, you MUST use quotation marks.
Can you write Python code for it?

# An error

my_favourite_things[food]

# ...but this is correct

food = 'food'

my_favourite_things[food]


I am working on the file "Day 1 - Unit 1.2.ipynb".
The first task is:
The conditional checks and see whether the remainder of 13 when divided by 2 is 0. (Recall that's what % does.) Since 13 returns 1 remainder when divided by 2, the condition is false. Hence the string 'This is an even number' is not printed.
7.2 if else statements
If you look at the previous example,  one can see an obvious problem with this block of code. What if y=14? Then the output would consist of two print statements 'This is an even number' and followed by "I guess it's odd then". This is not desireable. In fact we want to print 'This is an even number' only when y is even and "I guess it's odd then" only when y is odd. To code this into Python, we use the else keyword.
Code blocks under the else keyword are executed when the conditionals evaluate to False.
The format of of if else statements are as below:
if condition:
do something

else:
do something else
Can you write Python code for it?

y = 22

if y%2 ==0:
    print("{} is an even number".format(y))
else:
    print("{} is an odd number".format(y))

y = 13

if y%2 ==0:
    print("{} is an even number".format(y))
else:
    print("{} is an odd number".format(y))

# Nested if else statements

y = 25

remainder = y%3

if remainder == 0:
    print("{} is divisible by 3".format(y))
else:
    print("{} is not divisible by 3".format(y))
    if remainder ==1:
        print("But has remainder {}".format(remainder))
    else:
        print("But has remainder {}".format(remainder))


I am working on the file "Day 1 - Unit 1.2.ipynb".
The first task is:
Try re-executing the cell above with various values of y and see the effect on the output.
Nested if else statements make for horrible coding. It makes code hard to read and understand. Surely there must be a better way!
7.3 elif to rescue
We use elif when we need to test conditionals in a sequential manner. A sequence of conditionals is tested, one after another until the first True condition is encountered. Then, the code block corresponding to that conditional is executed and program continues on without checking the other conditionals.
The format of elif statements looks like this:
if $condition_1$:
do code 1

elif $condition_2$:
do code 2

$\vdots$
elif $condition_n$:
do code n

elif chains can be "terminated" either by an elif statement itself or by an else if there is no final conditional to be evaluated.
Let's try to refactor (recode) the nested if else statements above using the elif keyword.
Can you write Python code for it?

y=25

remainder = y%3

if remainder == 0:
    div = 'is'
    s = 'Hence'
elif remainder == 1:
    div = 'is not'
    s = 'But'
elif remainder == 2:
    div = 'is not'
    s = 'But'

print('{} {} divisible by 3\n{} has remainder {}'.format(y, div, s, remainder))


I am working on the file "Day 1 - Unit 1.3.ipynb".
The first task is:
10. Lambda expressions

Lambda expressions are used to define short functions that may be written in one line of code. This is more than just a convenience. Most arguments to pandas and seaborn functions are intended to take in callables (functions) and lambda expressions provide a good syntactical way of passing functions as arguments to other functions.
Recall our definition of my_first_function:
Can you write Python code for it?

def my_first_function(name):
    print("Hello %s" % (name))


I am working on the file "Day 1 - Unit 1.3.ipynb".
The first task is:
11.3 List comprehension
One one the great things about for loops in Python is there are easy to write and understand. However, this comes at a cost: time. for loops in Python are slow. Thus, if the code is iterated over a large number of loops, it will take time.
Let's see this. The scenario here is that we want to assign a running serial number to staff id. This will involve coercing int into str type. The thing is, we have 5000 staff. So let's see how much computer time it takes by using %timeit.
Can you write Python code for it?

%%timeit

serial_numbers = list()
for i in range(0,5000): # 5000 staff, so we need 5000 int's
    serial_numbers.append('s'+str(i)) # our serial numbers are prefixed with 's'