I am working on the file "PageviewAnalysis.ipynb".
The first task is:
Prepare the four dataframes for merging on timestamp.
Can you write Python code for it?

# Merge mobile and desktop of pagecounts dataframes.
df_pagecounts = pd.merge(df_pagecounts_desktop, df_pagecounts_mobile, on='timestamp', how='left')
# Merge mobile and desktop of pageviews dataframes
df_pageviews = pd.merge(df_pageviews_desktop, df_pageviews_mobile, on='timestamp')
# Subset the merged dataframes and rename column names
df_pagecounts = df_pagecounts[['timestamp', 'count_x', 'count_y']]
df_pagecounts.columns = ['timestamp', 'pagecount_desktop_views', 'pagecount_mobile_views']
df_pageviews = df_pageviews[['timestamp', 'views_x', 'views_y']]
df_pageviews.columns = ['timestamp', 'pageview_desktop_views', 'pageview_mobile_views']
# Finally, merge pagecount and pageview dataframes
df_total = pd.merge(df_pagecounts, df_pageviews, on='timestamp', how='left')
# Fill the missing value with 0
df_total = df_total.fillna(value = 0)
# Create two new columns, 'pagecount_all_views' and 'pageview_all_views', that are both the combinition of
# mobile and desktop counts respectively.
df_total['pagecount_all_views'] = df_total.pagecount_desktop_views + df_total.pagecount_mobile_views
df_total['pageview_all_views'] = df_total.pageview_desktop_views + df_total.pageview_mobile_views
# Create two new columns, 'year' and 'month', that are both based on the column 'timestamp'
df_total.timestamp = pd.to_datetime(df_total.timestamp, format='%Y%m%d%H')
df_total['year'] = df_total['timestamp'].dt.year
df_total['month'] = df_total['timestamp'].dt.month
# Do subsetting to get the final dataframe
df_total = df_total[['year', 'month',
                     'pagecount_all_views', 'pagecount_desktop_views', 'pagecount_mobile_views',
                     'pageview_all_views', 'pageview_desktop_views', 'pageview_mobile_views']]

df_total.head()


I am working on the file "PageviewAnalysis.ipynb".
The first task is:
Save the df_total dataframe to csv file.
Can you write Python code for it?


I am working on the file "PageviewAnalysis.ipynb".
The first task is:
Use Plotly to create an interactive visualization to present the page view information on English Wikipedia.
Can you write Python code for it?

trace1 = go.Scatter(
        x = df[df.pagecount_all_views != 0].date,
        y = df[df.pagecount_all_views != 0].pagecount_all_views,
        name='Total(Pagecount definition)',
        line = dict(color = 'black', dash = 'dash')
        )
trace2 = go.Scatter(
        x = df[df.pagecount_desktop_views != 0].date,
        y = df[df.pagecount_desktop_views != 0].pagecount_desktop_views,
        name='Main site(Pagecount definition)',
        line = dict(color = 'green', dash = 'dash')
        )
trace3 = go.Scatter(
        x = df[df.pagecount_mobile_views != 0].date,
        y = df[df.pagecount_mobile_views != 0].pagecount_mobile_views,
        name='Mobile(Pagecount definition)',
        line = dict(color = 'blue', dash = 'dash')
        )
trace4 = go.Scatter(
        x = df[df.pageview_all_views != 0].date,
        y = df[df.pageview_all_views != 0].pageview_all_views,
        name='Total(Pageview definition)',
        line = dict(color = 'black')
        )
trace5 = go.Scatter(
        x = df[df.pageview_desktop_views != 0].date,
        y = df[df.pageview_desktop_views != 0].pageview_desktop_views,
        name='Main site(Pageview definition)',
        line = dict(color = 'green')
        )
trace6 = go.Scatter(
        x = df[df.pageview_mobile_views != 0].date,
        y = df[df.pageview_mobile_views != 0].pageview_mobile_views,
        name='Mobile(Pagecount definition)',
        line = dict(color = 'blue')
        )

data = [trace1, trace2, trace3, trace4, trace5, trace6]
layout = dict(title = 'Page Views on English Wikipedia',
              height= 600, width = 1000,
              xaxis = dict(title = ''),
              yaxis = dict(title = 'Page views', range=[0, 12000000000]),
              legend=dict(orientation="h")
              )

fig = dict(data=data, layout=layout)
plotly.offline.iplot(fig, filename='PageViewsonEnglishWikipedia')


I am working on the file "PanSTARRS_WISE_ML.ipynb".
The first task is:
Compute the ML ratio
Can you write Python code for it?

radius = 15

idx_wise, idx_panstarrs, d2d, d3d = search_around_sky(
    coords_wise, coords_panstarrs, radius*u.arcsec)

idx_wise_unique = np.unique(idx_wise)

wise["lr"] = np.nan                   # Likelihood ratio
wise["lr_dist"] = np.nan              # Distance to the selected source
wise["lr_panstarrs_index"] = np.nan   # Index of the PanSTARRS source

total_sources = len(idx_wise_unique)

total_sources

panstarrs_aux_index = np.arange(len(panstarrs))

def ml(i):
    idx_0 = idx_panstarrs[idx_wise == i]
    d2d_0 = d2d[idx_wise == i]
    i_mag = panstarrs["i"][idx_0]
#     sigma = get_sigma(wise["ra"][i],
#                       wise["dec"][i],
#                       wise["sigra"][i],
#                       wise["sigdec"][i],
#                       panstarrs[idx_0]["raMean"][idx_0],
#                       panstarrs[idx_0]["decMean"][idx_0],
#                       panstarrs[idx_0]["raMeanErr"][idx_0],
#                       panstarrs[idx_0]["decMeanErr"][idx_0],
#              )
    sigma = 1.
    lr_0 = likelihood_ratio(i_mag, d2d_0.arcsec, sigma)
    chosen_index = np.argmax(lr_0)

    result = [panstarrs_aux_index[idx_0[chosen_index]], # Index
              (d2d_0.arcsec)[chosen_index],             # distance
              lr_0[chosen_index]]                       # LR

    return result

step_size = 100000
nsteps = total_sources//step_size + 1
res = []
for k in tnrange(nsteps, desc="Blocks"):
    low_limit = k*step_size
    high_limit = (k+1)*step_size
    res += parallel_process(idx_wise_unique[low_limit:high_limit],
                            ml,
                            n_jobs=8,
                            notebook=True)

len(res)

(wise["lr_panstarrs_index"][idx_wise_unique],
 wise["lr_dist"][idx_wise_unique],
 wise["lr"][idx_wise_unique]) = list(map(list, zip(*res)))


I am working on the file "PanSTARRS_WISE_ML.ipynb".
The first task is:
Define the main functions
We define the main functions used in the ML computing
Can you write Python code for it?

likelihood_ratio = SingleMLEstimator(q0, n_m, q_m, center)


I am working on the file "PanSTARRS_WISE_ML.ipynb".
The first task is:
Import the ML parameeters
Can you write Python code for it?

bin_list, center, q0, n_m, q_m = pickle.load(open("pw_params.pckl", "rb"))


I am working on the file "PanSTARRS_WISE_ML.ipynb".
The first task is:
Load the data
Load the catalogues
Can you write Python code for it?

panstarrs = Table.read("panstarrs_u2.fits")

wise = Table.read("wise_u2.fits")

len(panstarrs)

len(wise)


I am working on the file "PanSTARRS_WISE_ML.ipynb".
The first task is:
Save catalogue
Can you write Python code for it?

combined['ra'] = combined['raMean']
combined['dec'] = combined['decMean']
combined['raErr'] = combined['raMeanErr']
combined['decErr'] = combined['decMeanErr']

combined['ra'][np.isnan(combined['raMean'])] = combined['raWise'][np.isnan(combined['raMean'])]
combined['dec'][np.isnan(combined['decMean'])] = combined['decWise'][np.isnan(combined['decMean'])]
combined['raErr'][np.isnan(combined['raMean'])] = combined['raWiseErr'][np.isnan(combined['raMean'])]
combined['decErr'][np.isnan(combined['decMean'])] = combined['decWiseErr'][np.isnan(combined['decMean'])]


I am working on the file "PanSTARRS_WISE_ML.ipynb".
The first task is:
The following function could be used to get the sigma using the errors in the two optical catalogues but it may underestimate the error (not used)
Can you write Python code for it?

def get_sigma(ra1, dec1, ra1err, dec1err, ra2, dec2, ra2err, dec2err):
    """Input positions in degrees.
    Errors in arcsecs
    Output in arcsecs
    """
    cosadj = np.cos(np.deg2rad(0.5*(dec1 + dec2)))
    phi = np.arctan2((dec2 - dec1), ((ra2 - ra1)*cosadj))
    sigma = np.pi - phi
    err1squared  = (ra1err * np.cos(sigma))**2 + (dec1err * np.sin(sigma))**2
    err2squared  = (ra2err * np.cos(phi))**2 + (dec2err * np.sin(phi))**2
    return np.sqrt(err1squared + err2squared)


I am working on the file "PanSTARRS_WISE_ML.ipynb".
The first task is:
We can also save a version with all the data
Can you write Python code for it?

np.array(combined.colnames)

combined.write('pw_lrdata.fits', format="fits")


I am working on the file "PanSTARRS_WISE_Q0.ipynb".
The first task is:
$Q_0$ dependence on the radius
We will iterate 10 times for each radius. However, the error is so small that a direct computation can be accurate to 4 significant figures.
Can you write Python code for it?


I am working on the file "PanSTARRS_WISE_Q0.ipynb".
The first task is:
Cross-match and random cross-match
We create an instance of Q_0 using as the input the two catalogues and the area. It will compute the $Q_0$ for different radius.
The following function is not used but shows the internal code use for the computing of the $Q_0$.
Can you write Python code for it?

# Example function (not used, we use a class that contains this code)
def q_0_r(coords_wise, coords_panstarrs, field, radius=5):
    """Compute the Q_0 for a given radius"""
    random_wise = field.random_catalogue(len(coords_wise))

    idx_random_wise, idx_panstarrs, d2d, d3d = search_around_sky(
        random_wise, coords_panstarrs, radius*u.arcsec)

    nomatch_random = len(coords_wise) - len(np.unique(idx_random_wise))

    idx_wise, idx_panstarrs, d2d, d3d = search_around_sky(
        coords_wise, coords_panstarrs, radius*u.arcsec)

    nomatch_wise = len(coords_wise) - len(np.unique(idx_wise))

    return (1. - float(nomatch_wise)/float(nomatch_random))

q_0_comp = Q_0(coords_wise, coords_panstarrs, field)

q_0_comp(radius=5)


I am working on the file "PanSTARRS_WISE_Q0.ipynb".
The first task is:
The radius tested ranges from 1 to 25
Can you write Python code for it?

rads = list(range(1,26))

q_0_rad = []
for radius in rads:
    q_0_rad_aux = []
    for i in range(n_iter):
        out = q_0_comp(radius=radius)
        q_0_rad_aux.append(out)
    q_0_rad.append(np.mean(q_0_rad_aux))
    print("{:2d} {:7.5f} +/- {:7.5f} [{:7.5f} {:7.5f}]".format(radius,
            np.mean(q_0_rad_aux), np.std(q_0_rad_aux),
            np.min(q_0_rad_aux), np.max(q_0_rad_aux)))

plt.rcParams["figure.figsize"] = (5,5)
plot(rads, q_0_rad)
xlabel("Radius (arcsecs)")
ylabel("$Q_0$")
ylim([0, 1]);


I am working on the file "PanSTARRS_WISE_catalogues.ipynb".
The first task is:
Areas without forced photometry
There are some areas without forced photometry data. We emailed the responsibles of the archive and they are investigating the origing of the issue.
The next figure shows the distribution of the sources lacking forced photometry measurements.
Can you write Python code for it?

plot(base_panstarrs["raMean"][np.isnan(base_panstarrs['iFApFlux'])],
     base_panstarrs["decMean"][np.isnan(base_panstarrs['iFApFlux'])],
     ls="", marker=",");


I am working on the file "PanSTARRS_WISE_catalogues.ipynb".
The first task is:
Describe data
Can you write Python code for it?

describe(base_wise['ra'])
describe(base_wise['dec'])

hexbin(base_wise["ra"], base_wise["dec"]);


I am working on the file "PanSTARRS_WISE_catalogues.ipynb".
The first task is:
Filter the catalogue
Filter the WISE catalogue by selection the area in the field
Can you write Python code for it?


I am working on the file "PanSTARRS_WISE_catalogues.ipynb".
The first task is:
Free memory
Can you write Python code for it?


I am working on the file "PanSTARRS_WISE_catalogues.ipynb".
The first task is:
In the auxiliary catalogue we will use NaN to define the missing values for the magnitudes.
Can you write Python code for it?

all_bands = ["g", "r", "i", "z", "y"]

for band in all_bands:
    panstarrs.loc[(panstarrs["%sPSFMag"%band] == -999.), "%sPSFMag"%band] = np.nan
    panstarrs.loc[(panstarrs["%sPSFMagErr"%band] == -999.), "%sPSFMagErr"%band] = np.nan


I am working on the file "PanSTARRS_WISE_catalogues.ipynb".
The first task is:
Now we compute the interpolated value of "i" for the missing data. The computed values, including the measured ones, are entered in a column named "i".
Can you write Python code for it?

aux_weights = np.array([1,2,2,1])
mags_i_diff = np.array([band_properties[a]["mean"] for a in bands])

def interpolate_i(x):
    if np.isnan(x["iPSFMag"]):
        mags_raw = x[["gPSFMag",
                      "rPSFMag",
                      "zPSFMag",
                      "yPSFMag"]].values
        mags_i = mags_raw - mags_i_diff
        try:
            return wmedian(mags_i[~np.isnan(mags_raw)],
                          aux_weights[~np.isnan(mags_raw)])
        except ValueError:
            return np.nan
    else:
        return x["iPSFMag"]

panstarrs["i"] = panstarrs.apply(interpolate_i, axis=1)


I am working on the file "PanSTARRS_WISE_catalogues.ipynb".
The first task is:
PanSTARRS
Load PanSTARRS catalogue
Can you write Python code for it?

base_panstarrs = Table.read(os.path.join("data", "hetdex_full_jsabater.fits"))

len(base_panstarrs)


I am working on the file "PanSTARRS_WISE_catalogues.ipynb".
The first task is:
Positional offsets
Between the Mean and Stacked computed positions
Can you write Python code for it?

offset_dec = base_panstarrs['decMean']-base_panstarrs['decStack']
offset_ra = base_panstarrs['raMean']-base_panstarrs['raStack']

describe(offset_ra, decimals=7)
describe(offset_dec, decimals=7)

hexbin(offset_ra, offset_dec, bins='log')


I am working on the file "PanSTARRS_WISE_catalogues.ipynb".
The first task is:
Rename columns
Rename some columns to more common names
Can you write Python code for it?

new_wise.rename_column("designation", 'AllWISE')
new_wise.rename_column("ra", 'raWise')
new_wise.rename_column("dec", 'decWise')
new_wise.rename_column("sigra", 'raWiseErr')
new_wise.rename_column("sigdec", 'decWiseErr')
new_wise.rename_column('w1sigmpro', "W1magErr")


I am working on the file "PanSTARRS_WISE_catalogues.ipynb".
The first task is:
The error in the measurement is also computed and entered in a new column.
Can you write Python code for it?

aux_err = np.array([band_properties[a]["std"]/np.sqrt(band_properties[a]["n"]) for a in bands])
def interpolate_error_i(x):
    if np.isnan(x["iPSFMagErr"]):
        err_mags_raw = x[["gPSFMagErr",
                          "rPSFMagErr",
                          "zPSFMagErr",
                          "yPSFMagErr"]].values
        errs1 = err_mags_raw[~np.isnan(err_mags_raw)]
        errs2 = aux_err[~np.isnan(err_mags_raw)]
        try:
            return np.sqrt(np.sum(errs1**2) + np.sum(errs2**2))/np.sqrt(0.5*(len(errs1)+len(errs2)))
        except ValueError:
            return np.nan # There are some sources here with no errors in any band FIXME
    else:
        return x["iPSFMagErr"]

panstarrs["iErr"] = panstarrs.apply(interpolate_error_i, axis=1)

panstarrs["i"].describe()

panstarrs["iErr"].describe()


I am working on the file "PanSTARRS_WISE_catalogues.ipynb".
The first task is:
The original PanSTARRS catalogue had ~4.4M sources but the catalogue that covers all the area has ~27M sources. The columns are the following:
Can you write Python code for it?


I am working on the file "PanSTARRS_WISE_catalogues.ipynb".
The first task is:
This is the object representing the region considered. It can be used to filter the catalogues and to compute the area.
Can you write Python code for it?


I am working on the file "PanSTARRS_WISE_catalogues.ipynb".
The first task is:
Transform magnitudes
We will transform the magnitude of the W1 band from Vega to AB magnitudes
Can you write Python code for it?

new_wise["W1mag"] = new_wise['w1mpro'] + 2.683


I am working on the file "PanSTARRS_WISE_catalogues.ipynb".
The first task is:
We get the statistical properties for each of the other bands with data
Can you write Python code for it?

bands = ["g", "r", "z", "y"]

def get_diff(band):
    assert band in bands
    column = "{}PSFMag".format(band)
    mask = ((panstarrs[column] != -999.) &
            (panstarrs["iPSFMag"] != -999.) &
            (~np.isnan(panstarrs[column])) &
            (~np.isnan(panstarrs["iPSFMag"])))
    return panstarrs[mask][column] - panstarrs[mask]["iPSFMag"]

band_properties = {}
for band in bands:
    diff = get_diff(band)
    band_properties.update({band: {"mean": diff.mean(),
                                   "median": np.median(diff),
                                   "std": diff.std(),
                                   "n": len(diff)}
                     })
    if True:
        print("{} {n} {mean:7.4f} {std:7.4f} {median:7.4f}".format(band, **band_properties[band]))


I am working on the file "PanSTARRS_WISE_catalogues.ipynb".
The first task is:
Wrong magnitudes
Can you write Python code for it?

len(base_panstarrs[
        ((base_panstarrs["gPSFMag"] == - 999.) &
         (base_panstarrs["rPSFMag"] == - 999.) &
         (base_panstarrs["iPSFMag"] == - 999.) &
         (base_panstarrs["zPSFMag"] == - 999.) &
         (base_panstarrs["yPSFMag"] == - 999.))])


I am working on the file "PanSTARRS_WISE_pre_ml.ipynb".
The first task is:
Compute real(m) and q(m)
The first step is to crossmatch the catalogues to make an estimation
Can you write Python code for it?

radius = 5 # arcseconds

q_m = estimate_q_m(panstarrs["i"], bin_list, n_m, coords_wise, coords_panstarrs, radius=5)

plot(center, q_m);


I am working on the file "PanSTARRS_WISE_pre_ml.ipynb".
The first task is:
Compute the ML parameters
Number of sources per magnitude in i-band
Can you write Python code for it?

bin_list = np.linspace(12., 30., 1801)

center = get_center(bin_list)

n_m = get_n_m(panstarrs["i"], bin_list, field.area)


I am working on the file "PanSTARRS_WISE_pre_ml.ipynb".
The first task is:
Load the data
Load the catalogues
Can you write Python code for it?

panstarrs_full = Table.read("panstarrs_u2.fits")

wise_full = Table.read("wise_u2.fits")

panstarrs = field.filter_catalogue(
    panstarrs_full,
    colnames=("raMean", "decMean"))

# Free memory
del panstarrs_full

wise = field.filter_catalogue(
    wise_full,
    colnames=("raWise", "decWise"))

# Free memory
del wise_full


I am working on the file "PanSTARRS_WISE_reddening.ipynb".
The first task is:
Apply the correction to each position
Can you write Python code for it?

i_correction = ext_panstarrs * FILTER_EXT["i"]

w1_correction = ext_wise * FILTER_EXT["W1"]

hist(i_correction, bins=100);

hist(w1_correction, bins=100);

panstarrs.rename_column("i", 'iUncor')

wise.rename_column("W1mag", 'W1magUncor')

panstarrs["i"] = panstarrs["iUncor"] - i_correction

wise["W1mag"] = wise["W1magUncor"] - w1_correction


I am working on the file "PanSTARRS_WISE_reddening.ipynb".
The first task is:
Coordinates
As we will use the coordinates to retrieve the extinction in their positions
Can you write Python code for it?

coords_panstarrs = SkyCoord(panstarrs['raMean'], panstarrs['decMean'], unit=(u.deg, u.deg), frame='icrs')

coords_wise = SkyCoord(wise['raWise'], wise['decWise'], unit=(u.deg, u.deg), frame='icrs')


I am working on the file "PanSTARRS_WISE_reddening.ipynb".
The first task is:
Load the data
Load the catalogues
Can you write Python code for it?

panstarrs = Table.read("panstarrs_u1.fits")

wise = Table.read("wise_u1.fits")


I am working on the file "PanSTARRS_WISE_reddening.ipynb".
The first task is:
Reddening
Get the extinction for the positions of the sources in the catalogues.
Can you write Python code for it?

ext_panstarrs = get_eb_v(coords_panstarrs.ra.deg, coords_panstarrs.dec.deg)

ext_wise = get_eb_v(coords_wise.ra.deg, coords_wise.dec.deg)


I am working on the file "Pandas I - Series and DataFrames.ipynb".
The first task is:
1.4 Mathematical Operations
Mathematical operations can be done using scalars and functions.
Can you write Python code for it?

# divide city values by 3
cities / 3

# square city values
np.square(cities)


I am working on the file "Pandas I - Series and DataFrames.ipynb".
The first task is:
2. DataFrame
A DataFrame is a tabular data structure comprised of rows and columns, akin to a spreadsheet, database table, or R's data.frame object. You can also think of a DataFrame as a group of Series objects (rows) that share an index (the column names).
2.1 From Dictionnary of Lists
To create a DataFrame out of common Python data structures, we can pass a dictionary of lists to the DataFrame constructor.
Using the columns parameter allows us to tell the constructor how we'd like the columns ordered. By default, the DataFrame constructor will order the columns alphabetically (though this isn't the case when reading from a file - more on that next).
Can you write Python code for it?


I am working on the file "Pandas I - Series and DataFrames.ipynb".
The first task is:
2.5 From Clipboard
While the results of a query can be read directly into a DataFrame, I prefer to read the results directly from the clipboard. I'm often tweaking queries in my SQL client (Sequel Pro), so I would rather see the results before I read it into pandas. Once I'm confident I have the data I want, then I'll read it into a DataFrame.
This works just as well with any type of delimited data you've copied to your clipboard. The function does a good job of inferring the delimiter, but you can also use the sep parameter to be explicit.
Hank Aaron
Can you write Python code for it?


I am working on the file "Pandas I - Series and DataFrames.ipynb".
The first task is:
2.7 From Google Analytics API
pandas also has some integration with the Google Analytics API, though there is some setup required. I won't be covering it, but you can read more about it here and here.
3. Merge
Use the pandas.merge() static method to merge/join datasets in a relational manner. (See DOC)<br>
Like SQL's JOIN clause, pandas.merge() allows two DataFrames to be joined on one or more keys.

parameter how : specify which keys are to be included in the resulting table
parameters on, left_on, right_on, left_index, right_index : to specify the columns or indexes on which to join.

how : {"inner", "left", "right", "outer"}

"left" : use keys from left frame only
"right" : use keys from right frame only
"inner" (default) : use intersection of keys from both frames
"outer" : use union of keys from both frames

There are several cases to consider which are very important to understand:
- one-to-one joins: to define these relationships, only one table is necessary (no join)
    - one user has one phone number
    - one phone number belongs to one user
- one-to-many joins: to define these relationships, two tables are necessary
    - one post has many comments
    - one comment belongs to one post
        - merge(left, right, on=['key'], how='?')
- many-to-many joins: to define these relationships, three tables are necessary
    - one playlist has many songs
    - one song belongs to many playlists
        - merge(left.reset_index(), right.reset_index(), on=['key'], how='?').set_index(['key_left','key_right'])
Below are the different joins in SQL.
Can you write Python code for it?

left = pd.DataFrame({'key': range(5),
                           'left_value': ['L0', 'L1', 'L2', 'L3', 'L4']})
right = pd.DataFrame({'key': range(2, 7),
                           'right_value': ['R0', 'R1', 'R2', 'R3', 'R4']})
print left, '\n'
print right


I am working on the file "Pandas I - Series and DataFrames.ipynb".
The first task is:
3.1 Inner Join (default)
Selects the rows from both tables with matching keys.
Can you write Python code for it?

print pd.merge(left, right, on='key', how='inner')


I am working on the file "Pandas I - Series and DataFrames.ipynb".
The first task is:
3.3 Right Outer Join
Returns all rows from the right frame, with the matching rows in the left frame. The result is NULL in the left side when there is no match (NaN).
Can you write Python code for it?

print pd.merge(left, right, on='key', how='right')


I am working on the file "Pandas I - Series and DataFrames.ipynb".
The first task is:
4. Concatenate
Use pandas .concat() static method to combine Series/DataFrames into one unified object. (See DOC)
pandas.concat() takes a list of Series or DataFrames and returns a Series or DataFrame of the concatenated objects. Note that because the function takes list, you can combine many objects at once.
Use axis parameter to define along which axis to concatenate:

axis = 0 : concatenate vertically (default)<br>
   axis = 1 : concatenante side-by-side
Can you write Python code for it?


I am working on the file "Pandas I - Series and DataFrames.ipynb".
The first task is:
If our key columns had different names, we could have used the left_on and right_on parameters to specify which fields to join from each frame.
python
    pd.merge(left, right, left_on='left_key', right_on='right_key')
If our key columns were indexes, we could use the left_index or right_index parameters to specify to use the index column, with a True/False value. You can mix and match columns and indexes like so:
python
    pd.merge(left, right, left_on='key', right_index=True)

3.2 Left Outer Join
Returns all rows from the left frame, with the matching rows in the right frame. The result is NULL in the right side when there is no match (NaN).
Can you write Python code for it?

print pd.merge(left, right, on='key', how='left')


I am working on the file "Pandas I - Series and DataFrames.ipynb".
The first task is:
Much more often, you'll have a dataset you want to read into a DataFrame. Let's go through several common ways of doing so.
2.2 From/To CSV
Reading a CSV is as simple as calling the read_csv() function. By default, the read_csv() function expects the column separator to be a comma, but you can change that using the sep parameter.
Can you write Python code for it?

%cd ~/Dropbox/tutorials/pandas/

# Source: baseball-reference.com/players/r/riverma01.shtml
!head -n 5 data/mariano-rivera.csv

from_csv = pd.read_csv('data/mariano-rivera.csv')
from_csv.head()


I am working on the file "Pandas I - Series and DataFrames.ipynb".
The first task is:
Notice that because Austin, Chicago, and Portland were not found in both Series, they were returned with NULL/NaN values.
5. Missing Values
What if you aren't sure whether an item is in the Series?  You can check using idiomatic Python.
Can you write Python code for it?

print 'Seattle' in cities
print 'San Francisco' in cities


I am working on the file "Pandas I - Series and DataFrames.ipynb".
The first task is:
Or you can use boolean indexing for selection.
Can you write Python code for it?


I am working on the file "Pandas I - Series and DataFrames.ipynb".
The first task is:
You can add two Series together, which returns a union of the two Series with the addition occurring on the shared index values.  Values on either Series that did not have a shared index will produce a NULL/NaN (not a number).
Can you write Python code for it?

print cities[['Chicago', 'New York', 'Portland']]
print'\n'
print cities[['Austin', 'New York']]
print'\n'
print cities[['Chicago', 'New York', 'Portland']] + cities[['Austin', 'New York']]


I am working on the file "Pandas I - Series and DataFrames.ipynb".
The first task is:
pandas various reader functions have many parameters allowing you to do things like skipping lines of the file, parsing dates, or specifying how to handle NA/NULL datapoints.
Writing to CSV
There's also a set of writer functions for writing to a variety of formats (CSVs, HTML tables, JSON).  They function exactly as you'd expect and are typically called to_format:
python
my_dataframe.to_csv('path_to_file.csv')
Take a look at the IO documentation to familiarize yourself with file reading/writing functionality.
2.3 From/To Excel
Know who hates VBA? Me. I bet you do, too. Thankfully, pandas allows you to read and write Excel files, so you can easily read from Excel, write your code in Python, and then write back out to Excel - no need for VBA.
Reading Excel files requires the xlrd library. You can install it via pip (pip install xlrd).
Let's first write a DataFrame to Excel.
Can you write Python code for it?

# this is the DataFrame we created from a dictionary earlier
print football.head()

# since our index on the football DataFrame is meaningless, let's not write it
football.to_excel('data/football.xlsx', index=False)

# command line : list .xlsx files
!ls -l data/*.xlsx

# delete the DataFrame
del football

# read from Excel
football = pd.read_excel('data/football.xlsx')
print football


I am working on the file "Pandas II - Working with DataFrames.ipynb".
The first task is:
.ix() method
When you change the indexing of a DataFrame to a specific column, you use the default pandas 0-based index.

Use .ix() method for row selection based on the new index.
Let's set the index to the user_id using the .set_index() method.

NB: By default, .set_index() returns a new DataFrame, so you'll have to specify if you'd like the changes to occur in place.
Can you write Python code for it?

# Change index column (new DataFrame)
new_df_users = df_users.set_index('user_id')
print new_df_users.head(3)

# Change index column (inplace)
df_users.set_index('user_id', inplace=True)
print df_users.head(3)

# Select users using their respective user_id
print df_users.ix[99], '\n'
print df_users.ix[[1, 50, 300]]


I am working on the file "Pandas II - Working with DataFrames.ipynb".
The first task is:
3. Sort
a) .sort() for DataFrames
Use .sort() method to sort DataFrames. Returns a new instance of a Dataframe. (See DOC)

column : column name to base the sorting on (list for nested sorting / tuple for multi-index sorting)
ascending (True) : sort ascending vs. descending (specify list for multiple sort orders)
inplace (False): result is a new instance of DataFrame
Can you write Python code for it?

# Oldest techicians
df_users.sort('age', ascending=False, inplace=True)
print df_users[df_users.occupation == "technician"][:5]


I am working on the file "Pandas II - Working with DataFrames.ipynb".
The first task is:
4. Operations
a) Descriptive Stats
A large number of methods for computing descriptive statistics and other related operations on Series, DataFrame, and Panel. For DataFrames these methods take an axis argument:

axis=0 : compute over indexes
axis=1 : compute over columns

Most methods produce a lower-dimensional result (aka aggregate functions) :
- .count(): number of NOT NULL values
- .nunique(): number of unique NOT NULL values
- .size() : number of values
- .min(): minimum
- .max(): maximum
- .sum(): sum of values
- .prod(): product of values
- .median(): arithmetic median of values
- .quantile(): sample quantile (value at %)
- .mean(): mean of values
- .std(): unbiased standard deviation
- .var(): unbiased variance
- .mad(): mean absolute deviation
- .sem(): unbiased standard error of the mean
- .skew(): unbiased skewness (3rd moment)
- .kurt(): unbiased kurtosis (4th moment)
Some methods produce an object of the same size :
- .rank(): compute data rank (1 through n)
- .mode(): mode
- .abs(): absolute value
- .cumsum(): cumulative sum
- .cumprod(): cumulative product
- .cummax(): cumulative maximum
- .cummin(): cumulative minimum
b) Apply
To apply your own or another library’s functions to pandas objects, you should be aware of the three methods below. The appropriate method to use depends on whether your function expects to operate on an entire DataFrame or Series, row- or column-wise, or elementwise.

Tablewise Function Application: .pipe()
Row or Column-wise Function Application: .apply()
Elementwise function application: .applymap() or .map()

.pipe()
Use .pipe() for method chaining over a DataFrame. (See DOC)<br>
The following two are equivalent :
    - f(g(h(df), arg1=1), arg2=2, arg3=3)
    - df.pipe(h).pipe(g, arg1=1).pipe(f, arg2=2, arg3=3)
The pipe method is inspired by unix pipes and more recently dplyr and magrittr, which have introduced the popular (%>%) (read pipe) operator for R.
.apply()
Use .apply() to apply a function along the axes of a DataFrame, like the descriptive statistics...
Can you write Python code for it?


I am working on the file "Pandas II - Working with DataFrames.ipynb".
The first task is:
5. Split-Apply-Combine
Use .groupby() method to execute the split-apply-combine strategy for data analysis :
1. Split the DataFrame into groups based on some criteria (DataFrameGroupBy or SeriesGroupBy)
2. Apply a function to each group independently
3. Combine the results into a data structure (DataFrame or Series)

DataFrameGroupBy/SeriesGroupBy Methods (See Doc)
- .apply(): apply your own or another library's function or list of functions
- .agg(): aggregate using input function or dict of {column: function}
- .transform(): transform
- .filter(): return a copy of a DataFrame excluding elements from groups
<br>
In the apply step, we might wish to do one of the following:
- Aggregation: computing a summary statistic (or statistics) about each group. Some examples:
    - Compute group columns sums and means :
        - gby.agg([np.sum, np.mean])
    - Compute group sizes and counts :
        - gby.agg([np.size, np.mean])
- Transformation: perform some group-specific computations on every data point. Some examples:
    - Standardizing data (zscore) within group :
        - gby.transform(lambda x: (x - x.mean()) / x.std())
    - Filling NAs within groups with a value derived from each group
        - gby.fillna(x.mean())
- Filtration: discard some groups, according to a group-wise computation that evaluates True or False. Some examples:
    - Discarding data that belongs to groups with only a few members :
        - gby.filter(lambda x: x.size() > 100)
    - Discarding data based on the group sum or mean
        - gby.filter(lambda x: x['A'].sum() + x['B'].sum() > 0)
    - Discarding data for missing data
        - gby.dropna(axis=0)
City of Chicago salaries
The City of Chicago is kind enough to publish all city employee salaries to its open data portal. Let's go through some basic groupby examples using this data.
Can you write Python code for it?

!head -n 3 data/city-of-chicago-salaries.csv


I am working on the file "Pandas II - Working with DataFrames.ipynb".
The first task is:
Multiple columns selection<br>
To select multiple columns, simply pass a list of column names to the DataFrame, the output of which will be a DataFrame.
Can you write Python code for it?

list_of_cols = ['occupation', 'sex']
print df_users[list_of_cols].head()


I am working on the file "Pandas II - Working with DataFrames.ipynb".
The first task is:
Notice user_id was included since it's numeric. Since this is an ID value, the stats for it don't really matter.
We can quickly see the average age of our users is just above 34 years old, with the youngest being 7 and the oldest being 73. The median age is 31, with the youngest quartile of users being 25 or younger, and the oldest quartile being at least 43.
c) .head(), tail(), [i:j]
By default, .head() displays the first five records of the DataFrame, while .tail() displays the last five.<br>
Alternatively, Python's regular slicing [i:j] syntax works as well.
Can you write Python code for it?

print df_users.head()

print df_users.tail(3)

print df_users[20:22]


I am working on the file "Pandas II - Working with DataFrames.ipynb".
The first task is:
Use the .reset_index() method to reset the default index (the same rule apply for inplace).
Can you write Python code for it?


I am working on the file "Pandas II - Working with DataFrames.ipynb".
The first task is:
What department pays best on average ?

Split DataFrame into groups by departement => DataFrameGroupBy
Apply .mean() method
(Combine into DataFrame)
Sort resulting DataFrame according to the salary (NB: .order() is for Series, .sort() is for DataFrames)
Can you write Python code for it?

print df_chicago.groupby('department').mean().sort('salary', ascending=False).head()

print df_chicago.groupby('department').agg({'salary': [np.size, np.mean]}).sort(('salary', 'mean'), ascending=False).head()


I am working on the file "Pandas II - Working with DataFrames.ipynb".
The first task is:
What departments have the most number of distinct title positions ?

Split DataFrame into groups by departement, keep only title column => SeriesGroupBy
Apply .nunique() method
(Combine into Serie)
Order resulting Serie (NB: .order() is for Series, .sort() is for DataFrames)
Can you write Python code for it?


I am working on the file "Pandas II - Working with DataFrames.ipynb".
The first task is:
Who is the highest paid employee of each department ?

Split DataFrame into groups by departement, keep only salary column => SeriesGroupBy
Apply .rank() method
(Combine into Serie)
Assign the resulting Serie to a new column of the DataFrame
Sort DataFrame according to salary (NB: .order() is for Series, .sort() is for DataFrames)
Display only first rankers

For the .rank() method, use attributes:
- ascending=False : to rank high (1) to low (N)
- method='first' : so that equally high paid people within a department don't get the same rank .
Can you write Python code for it?

df_chicago['dept_rank'] = df_chicago.groupby('department')['salary'].rank(method='first', ascending=False)
df_chicago.sort('salary', ascending=False, inplace=True)
print df_chicago[df_chicago['dept_rank'] == 1].head()

print df_chicago[df_chicago['department'] == 'MAYOR\'S OFFICE'].tail(10)


I am working on the file "Pandas II - Working with DataFrames.ipynb".
The first task is:
b) .order() for Series
Use .order() method to sort Series. Returns a new instance of a Dataframe.

ascending (True) : sort ascending vs. descending (specify list for multiple sort orders)
inplace (False): result is a new instance of DataFrame
Can you write Python code for it?


I am working on the file "Pandas II - Working with DataFrames.ipynb".
The first task is:
d) Histograms
Use .value_counts() Series method to return the counts of unique values (ie frequency). (See DOC)
Can you write Python code for it?


I am working on the file "Pandas III - Datetime Series.ipynb".
The first task is:
We'll be using the MovieLens dataset in many examples going forward. The dataset contains 100,000 ratings made by 943 users on 1,682 movies.
Can you write Python code for it?

# pass in column names for each CSV
u_cols = ['user_id', 'age', 'sex', 'occupation', 'zip_code']
df_users = pd.read_csv('data/MovieLens-100k/u.user', sep='|', names=u_cols)

r_cols = ['user_id', 'movie_id', 'rating', 'unix_timestamp']
df_ratings = pd.read_csv('data/MovieLens-100k/u.data', sep='\t', names=r_cols)

m_cols = ['movie_id', 'title', 'release_date', 'video_release_date', 'imdb_url']
df_movies = pd.read_csv('data/MovieLens-100k/u.item', sep='|', names=m_cols, usecols=range(5))# only load the first five columns


I am working on the file "Pandas Introduction/.ipynb_checkpoints/Untitled-checkpoint.ipynb".
The first task is:
Así de simple es crear un data frame. Ahora veamos lo que todos estaban esperando!! Crear data frames a partir de datos que esten en hojas de excel o en csv o txt. Que justamente son los que se generan en los experimentos que algunas de ustedes lleguen a hacer o que otras personas hagan para que ustedes los analicen.
Vamos a ver un ejemplo con un set de datos pequeño.
Can you write Python code for it?

mi_df = pd.read_csv('FreeFattyAcids.csv', header=0)
mi_df


I am working on the file "Pandas Introduction/.ipynb_checkpoints/Untitled-checkpoint.ipynb".
The first task is:
En pandas una de las cosas más confusas (especialmente si tienes tiempo trabajando con numpy) es sacar subgrupos de datos. La forma en la que pandas está diseñado para esto es a través de índices y encabezados.
Can you write Python code for it?

mi_df["Sample"]

#o algo padre que también es posible en pandas es acceder a columnas con puntos
mi_df.Sample


I am working on the file "Pandas Introduction/.ipynb_checkpoints/Untitled-checkpoint.ipynb".
The first task is:
Pregunta 1
¿Qué diferencias notas entre la lista y la serie?
Ajá!! Exactemente!! esos pequeños numerillos a la izquierda, ¿Qué te imaginas que son?
Pues esos pequeños se llaman índices y son un concepto muy útil que hay que entender porque pandas se maneja principalmente con índices. Ahora veamos como podemos nombrar como queramos a esos índices.
Can you write Python code for it?

serie2 = pd.Series(lista, index=['H', 'm', 'n', 'e', 'p', '0'])
serie2


I am working on the file "Pandas Introduction/.ipynb_checkpoints/Untitled-checkpoint.ipynb".
The first task is:
Selección por índice
Para seleccionas sólo por índice tenemos que agregar la terminación .loc después del nombre de nuestro data frame y luego poner los índices que queramos con los nombres de las columnas que nos interesen.
Esta sección es de la documentación de pandas y dice lo siguiente acerca del atributo .loc.

Este atributos es el método primario de acceso (a los datos). Los siguientes son inputs válidos:
1. Una etiqueta (que es estrictamente de los índices)
2. Una lista o arreglo de etiquetas
3. Un slice de etiquetas (nota, al contrario de lo que se hace en python, el número inicial y final están incluídos!)
4. Un arreglo booleano
5. Un "callable"
Can you write Python code for it?

mi_df.loc[0:4,'Sample']

#¿qué pasa si sólo pongo el nombre de la columna?
mi_df.loc['Sample']

mi_df.loc[:, 'Sample']

mi_df.loc[mi_df.Sample=='C']#Esto es un slicing dado un arreglo booleano


I am working on the file "Pandas Introduction/.ipynb_checkpoints/Untitled-checkpoint.ipynb".
The first task is:
Te recuerda esto a alguna estructura de datos que ya hemos visto?
ejem, ejem, dic..., ejem, ejem...
Pues si se podría pensar en una serie como una especie de diccionarios, de hecho para crear una serie también puedes partir de un diccionario...
Can you write Python code for it?

diccionario = {'Ciudad de México':'Tacos al pastor', 'Puebla':'Chile en nogada', 'Yucatán':'Cochinita'}

serie_dic = pd.Series(diccionario)
serie_dic

serie_dic.loc['Puebla':'Yucatán']

lista2 = [1,1,1,1,1,1]


I am working on the file "Pandas Introduction/Untitled.ipynb".
The first task is:
Ahora que ya tenemos pandas podemos empezar a explorar que cosas podemos hacer con esta librería.
En pandas hay dos tipos nuevos de estructura de datos. Pregunta rápida, recuerdas los otros tipos de datos que hemos visto??
De regreso al punto :P los nuevos tipos de datos se separan en dos grupos: las series y los "data frames".
Las series son el elemento más simple en pandas ya que es una arreglo de n x 1 es decir n número de filas con una sola columna. Veamos un ejemplo:
Can you write Python code for it?

lista = ['hola', 'mi', 'nombre', 'es', 'pandas',0.18 ]
serie = pd.Series(lista)
print(lista)
print(serie)


I am working on the file "Pandas Introduction/Untitled.ipynb".
The first task is:
Así de simple es crear un data frame. Ahora veamos lo que todos estaban esperando!! Crear data frames a partir de datos que esten en hojas de excel o en csv o txt. Que justamente son los que se generan en los experimentos que algunas de ustedes lleguen a hacer o que otras personas hagan para que ustedes los analicen.
Vamos a ver un ejemplo con un set de datos pequeño.
Can you write Python code for it?


I am working on the file "Pandas Introduction/Untitled.ipynb".
The first task is:
Ejercicio 3
Cuáles son los tres samples con mayor número de ácidos grasos libres totales
Can you write Python code for it?


I am working on the file "Pandas Introduction/Untitled.ipynb".
The first task is:
En pandas una de las cosas más confusas (especialmente si tienes tiempo trabajando con numpy) es sacar subgrupos de datos. La forma en la que pandas está diseñado para esto es a través de índices y encabezados.
Can you write Python code for it?


I am working on the file "Pandas Introduction/Untitled.ipynb".
The first task is:
Este set de datos es de un perfil lipídico obtenido con espectrometría de masas. Son tres condiciones, una control y dos tratamientos, para cada una de ellas se puede ver la concentración de ácidos grasos de cadenas de 12 hasta 26 carbonos.
Por suerte este data frame es pequeño y podemos visualizar casi todo, pero imaginemos que tenemos 1000000 de columnas, en realidad no necesitamos verlo todo, pero queremos saber más o menos como está organizado nuestro data frame. Para esto podemos acceder a un método de mi_df para ver nada más la "cabeza" (o la "cola") de nuestros datos.
Can you write Python code for it?

mi_df.head()

#Por default nos va a mostras las primeras 5 filas después del encabezado,
#pero podemos cambiarlo pasando un argumento a head()
mi_df.head(3)

#Para ver la "cola"
mi_df.tail(3)


I am working on the file "Pandas Introduction/Untitled.ipynb".
The first task is:
Pero si queremos seleccionar múltiples columnas separadas podemos hacer lo siguiente
Can you write Python code for it?

mi_df[["Sample","16:0"]][0:4].plot()

import matplotlib.pyplot as plt

plt.show()

mi_df[0:3]


I am working on the file "Pandas Introduction/Untitled.ipynb".
The first task is:
Pero una vez que tenemos los datos ¿qué hacemos con ellos? Pues bien en realidad eso depende mucho de las preguntas que queramos responder empecemos con unas preguntas fáciles.
Pregunta 3
Digamos que a mí sólo me interesa saber cómo cambia el total, ¿Cómo puedo hacer que en un data frame sólo me muestre la columna "Sample" y la columna "Total"?
Para responder esta cuestión, necesitamos saber como hacer "slicing" de los data frames, esto es, sacar subgrupos de un data frame para analizarlo más fácilmente. Alguien tiene alguna idea que podamos probar??
Can you write Python code for it?


I am working on the file "Pandas Introduction/Untitled.ipynb".
The first task is:
Pregunta 1
¿Qué diferencias notas entre la lista y la serie?
Ajá!! Exactemente!! esos pequeños numerillos a la izquierda, ¿Qué te imaginas que son?
Pues esos pequeños se llaman índices y son un concepto muy útil que hay que entender porque pandas se maneja principalmente con índices. Ahora veamos como podemos nombrar como queramos a esos índices.
Can you write Python code for it?


I am working on the file "Pandas Introduction/Untitled.ipynb".
The first task is:
Selección por posición.
Es la más parecida a como opera numpy, y aquí sólo tienes que darle las posiciones que quieres (en números) de filas y columnas. Para esto usamos el atributo .iloc después de nuestro data frame.
Aquí los argumentos válidos son los siguientes:

integer e.g. 5
Lista o arreglo de enteros [4, 3, 0]
Un objeto slice con ints 1:7
Un arreglo booleano
Un "callable"
Can you write Python code for it?


I am working on the file "Pandas Introduction/Untitled.ipynb".
The first task is:
Si quieres saber mas detalles de lo que la función random.randn de numpy hace, puedes checar esta página
Así de simple se puede generar un data frame, pero también podemos hacerlo con diccionarios...
Can you write Python code for it?

df_diccionario = pd.DataFrame({'A':list(range(5)),
                              'B': np.random.random_sample(5),
                              'C':'Jueves'})

df_diccionario


I am working on the file "Pandas Sparklines Demo.ipynb".
The first task is:
Detailed Formatting
Return only sparklines, format the line, fill and marker.
Can you write Python code for it?

df_out = pd.DataFrame()
df_out['sparkline'] = sparklines.create(data=a,
                                        color='#1b470a',
                                        fill_color='#99a894',
                                        fill_alpha=0.2,
                                        point_color='blue',
                                        point_fill='none',
                                        point_marker='*',
                                        point_size=3,
                                        figsize=(6, 0.25))
sparklines.show(df_out[['sparkline']])


I am working on the file "Pandas Sparklines Demo.ipynb".
The first task is:
Export to HTML
Inline Jupyter Notebook
Can you write Python code for it?


I am working on the file "Pandas Sparklines Demo.ipynb".
The first task is:
HTML text for rendering elsewhere
Can you write Python code for it?


I am working on the file "Pandas Sparklines Demo.ipynb".
The first task is:
Insert Sparklines into source DataFrame
Can you write Python code for it?

df['sparkline'] = sparklines.create(data=a)
sparklines.show(df[['function', 'sparkline']])


I am working on the file "Pandas Sparklines Demo.ipynb".
The first task is:
Output to new DataFrame of Sparklines
Can you write Python code for it?

df_out = pd.DataFrame()
df_out['sparkline'] = sparklines.create(data=a)
sparklines.show(df_out[['sparkline']])


I am working on the file "Pandas and Scikit-Learn.ipynb".
The first task is:
Finally, we convert the processed training data from a Pandas dataframe into a numerical (Numpy) array.
Can you write Python code for it?


I am working on the file "Pandas and Scikit-Learn.ipynb".
The first task is:
For convenience, we move the column Survived to the left-most column. We note that the left-most column is indexed as 0.
Can you write Python code for it?

cols = [cols[1]] + cols[0:1] + cols[2:]
df = df[cols]


I am working on the file "Pandas and Scikit-Learn.ipynb".
The first task is:
In our final review of our training data, we check that (1) the column Survived is the left-most column (2) there are no NA values, and (3) all the values are in numerical form.
Can you write Python code for it?


I am working on the file "Pandas and Scikit-Learn.ipynb".
The first task is:
Pandas - Cleaning data
We then review a selection of the data.
Can you write Python code for it?


I am working on the file "Pandas and Scikit-Learn.ipynb".
The first task is:
Question

If you were to fill in the missing values, with what values would you fill them with? Why?

Scikit-learn only takes numerical arrays as inputs. As such, we would need to convert the categorical columns Sex and Embarked into numerical ones. We first review the range of values for the column Sex, and create a new column that represents the data as numbers.
Can you write Python code for it?

df['Sex'].unique()

df['Gender'] = df['Sex'].map({'female': 0, 'male':1}).astype(int)


I am working on the file "Pandas and Scikit-Learn.ipynb".
The first task is:
Scikit-learn - Making predictions
We first load the test data.
Can you write Python code for it?


I am working on the file "Pandas and Scikit-Learn.ipynb".
The first task is:
Similarly for Embarked, we review the range of values and create a new column called Port that represents, as a numerical value, where each passenger embarks from.
Can you write Python code for it?

df['Embarked'].unique()

df['Port'] = df['Embarked'].map({'C':1, 'S':2, 'Q':3}).astype(int)


I am working on the file "Pandas and Scikit-Learn.ipynb".
The first task is:
We notice that the columns describe features of the Titanic passengers, such as age, sex, and class. Of particular interest is the column Survived, which describes whether or not the passenger survived. When training our model, what we are essentially doing is assessing how each feature impacts whether or not the passenger survived (or if the feature makes an impact at all).
Exercise:
- Write the code to review the tail-end section of the data.
We observe that the columns Name and Cabin are, for our current purposes, irrelevant. We proceed to remove them from our data set.
Can you write Python code for it?

df = df.drop(['Name', 'Ticket', 'Cabin'], axis=1)


I am working on the file "Pandas and Scikit-Learn.ipynb".
The first task is:
We review the columns our final, processed data set.
Can you write Python code for it?


I am working on the file "Pandas como interfaz SQL.ipynb".
The first task is:
Ahora que ya tenemos una conexión a la base de datos Pandas se puede encargar del trabajo sucio de 'hablar' con la base de datos y ayudarnos a interactuar directamente con los datos de la forma habitual y potente de Pandas.
Vamos a crear un DataFrame que usaremos como una tabla para insertar en la base de datos. Este DataFrame tendrá una columna de fechas, una de medidas de temperatura promedio diaria (inventada), una de precipitación acumulada en 24h (inventada), una columna con el tipo de sensor que midió la temperatura ese día y una última con el sensor que midió la precipitación.
Can you write Python code for it?

# fechas para cada día del 2014
fechas = pd.date_range(dt.datetime(2014, 1, 1), dt.datetime(2014, 12, 31))
# Temperatura media diaria durante 2014 en algún lugar del hemisferio Norte
tmed = (np.random.randint(-5, 5, size = 365) +
        20 * np.cos(np.arange(0 - 180, 365 - 180) * 2 * np.pi / 365)
        + 10)
# Precipitación acumulada en 24h
prec = (20 * np.abs(np.random.randn(365) *
        np.cos(np.arange(0, 365) * 2 * np.pi / 365)))
# Sensor que midió la temperatura
marcaT = np.random.choice(['marcaT1', 'marcaT2', 'marcaT3'], size = 365)
# Sensor midió la precipitación
marcaP = np.random.choice(['marcaP1', 'marcaP2'], size = 365)

# Creamos el dataframe y lo guardamos en una tabla llamada 'datos'
df = pd.DataFrame(
        np.array([fechas.values, tmed, prec, marcaT, marcaP]).T,
        columns = ['fecha', 'tmedia','precipitacion','sensorT','sensorP'])
df['fecha'] = pd.to_datetime(df['fecha'])
df.to_sql('datos', con = conn, dtype = {'time': 'TIMESTAMP'})


I am working on the file "Pandas como interfaz SQL.ipynb".
The first task is:
Consultando la base de datos
Dentro de SQL tenemos comandos que se pueden integrar en diferentes categorías. Ahora vamos a usar dql (data query language) que se usa para hacer consultas a la base de datos. Mirad en esta chuleta para conocer más.
Como SQL no hace distinción de mayúsculas y minúsculas usaré mayúsculas para las palabras clave de SQL y así las distinguiremos un poco mejor. Para hacer una petición se usa SELECT. Veamos como es una consulta para conocer las tablas que existen en la base de datos (esta consulta es específica para SQLite):
Can you write Python code for it?

# Esto es específico para sqlite
print(pd.read_sql("SELECT name FROM sqlite_master WHERE type='table';", conn))
# Para otras BBDD puedes buscar en internet :-)


I am working on the file "Pandas como interfaz SQL.ipynb".
The first task is:
En la consulta anterior hemos usados varias palabras clave: SELECT, FROM, WHERE. Este tipo de consultas serán de lo más habitual. Vamos a explotar lo que ya sabemos.
Quiero los datos de todas las columnas de la tabla sensores (y pandas me lo meterá en un DataFrame, maravilloso!!!)
Can you write Python code for it?

df = pd.read_sql("SELECT * FROM sensores;", conn)

print(df)


I am working on the file "Pandas como interfaz SQL.ipynb".
The first task is:
Podemos decirle que nos pase solo una serie de valores. Por ejemplo, solo quiero los tres valores más altos de precipitación de diciembre:
Can you write Python code for it?

q = """
SELECT
    fecha, precipitacion
FROM
    datos
WHERE
    fecha > '2014-11-30'
ORDER BY
    precipitacion DESC
LIMIT
    3"""

df = pd.read_sql(q, conn)

print(df)


I am working on the file "Pandas como interfaz SQL.ipynb".
The first task is:
Si os fijáis, en la consulta anterior he usado datos.sensorT (tabla.columna). Esto solo será necesario si estamos trabajando con varias tablas y una columna se puede llamar igual en varias tablas. De esta forma no hay posibilidad de equivocarse ya que usamos 'nombres y apellidos'.
Vamos a hacer una consulta que nos dé el mismo resultado pero que enlaza varias consultas (un SELECT dentro de un SELECT). También la represento de forma un poco más estructurada para que sea más legible:
Can you write Python code for it?

# La misma consulta de antes pero sin saber la precisión de cada uno de los sensores a priori
q = """
SELECT
    fecha, tmedia
FROM
    datos
WHERE
    datos.sensorT =
    (SELECT
        sensores
    FROM
        sensores
    WHERE
        precision < 0.5);"""

df = pd.read_sql(q, conn)

print(df)


I am working on the file "Pandas from the Inside (PyData DC).ipynb".
The first task is:
Download sample data
The sample data here is historical Australian Rules Football (AFL) game results, going back to the start of the first competition in 1897.
This data can be downloaded from the web site http://www.afltables.com. The site's focus is week/game-level HTML tables formatted for human footy enthusiasts to read. We want something easier for pandas to consume, so instead grab the large text file bg3.txt.
Can you write Python code for it?

def download_sample_data(names=('bg3.txt', 'bg7.txt')):
    '''
    Download results and attendance stats for every AFL match
    since 1897 from www.afltables.com into files
    'bg3.txt' and 'bg7.txt' in the current directory.
    '''
    import urllib.request

    base_url = 'http://afltables.com/afl/stats/biglists/'
    for filename in names:
        url = base_url + filename
        print("Downloading from %s" % url)
        txt = urllib.request.urlopen(url).read()
        with open(filename, 'wb') as f:
            f.write(txt)
            print("Wrote %d bytes to %s" % (len(txt), filename))

name = 'bg3.txt'
if not os.path.exists(name):
    download_sample_data([name])


I am working on the file "Pandas from the Inside (PyData DC).ipynb".
The first task is:
Here is the complete code to produce the ladder.
Note there is a bug in pandas 0.18.0's multidimensional slicing which we have to work around.
Can you write Python code for it?

def calc_team_ladder(scores_df, year=2016):
    '''
    DataFrame with championship ladder with round-robin games for the given year.
    Wins, draws and losses are worth 4, 2 and 0 points respectively.
    '''
    # Select a subset of the rows
    # df.loc[] matches dates as strings like '20160506' or '2016'.
    # Note here rounds are simple strings so sort with R1 < R10 < R2 < .. < R9
    #      (we could change this with a CategoricalIndex)
    if pd.__version__ > '0.18.0':
        # MultiIndex slicing works ok
        scores2 = scores_df.sort_index()
        x = scores2.loc(axis=0)[str(year), :, 'R1':'R9', :]
    else:
        # pandas 0.18.0 has a bug with .loc on MultiIndexes
        # if dates are the first level. It works as expected if we
        # move the dates to the end before slicing
        scores2 = scores_df.reorder_levels([1, 2, 3, 0]).sort_index()
        x = scores2.loc(axis=0)[:, 'R1':'R9', :, str(year):str(year)]
        # Don't need to put levels back in order as we are about to drop 3 of them
        # x = x.reorder_levels([3, 0, 1, 2]).sort_index()

    # Just keep Team. This does a copy too, avoiding SettingWithCopy warning
    y = x.reset_index(['Date', 'Venue', 'Round'], drop=True)

    # Add cols with 0/1 for number of games played, won, drawn and lost
    y['P'] = 1
    y['W'] = (y['F'] > y['A']).astype(int)
    y['D'] = 0
    y.loc[y['F'] == y['A'], 'D'] = 1
    y.eval('L = 1*(A>F)', inplace=True)
    #print(y)

    # Subtotal by team and then sort by Points/Percentage
    t = y.groupby(level='Team').sum()
    t['PCT'] = 100.0 * t.F / t.A
    t['PTS'] = 4 * t['W'] + 2 * t['D']
    ladder = t.sort_values(['PTS', 'PCT'], ascending=False)

    # Add ladder position (note: assumes no ties!)
    ladder['Pos'] = pd.RangeIndex(1, len(ladder) + 1)
    #print(ladder)

    return ladder

ladder_df = calc_team_ladder(scores_df, 2015)
ladder_df


I am working on the file "Pandas from the Inside (PyData DC).ipynb".
The first task is:
Note I prepared the slides in today's tutorial earlier in the year for PyData London, when the 2016 season was only part-way through.
If you download the data now, it includes results for the whole of the 2016 season. The last match is the Grand Final on Saturday 1 October, where the Western Bulldogs beat the Sydney Swans 13.11.89 to 10.7.67.
Load raw data into a pandas DataFrame
The pandas function pd.read_csv() is surprisingly powerful. Here we use its ability to split text into columns using a regular expression. This is a little tricky because most columns are separated by two or more spaces. The exception is the row numbers, where the separator is a fot (from the row number) plus a single space, once the row numbers are above 99,999,
Can you write Python code for it?

def load_data(name='bg3.txt'):
    '''
    Pandas DataFrames from loading csv files bg3.txt (games) or
    bg7.txt (attendance) csvs downloaded from www.afltables.com.
    '''
    if name == 'bg3.txt':
        # Scores with rounds
        # - GameNum ends with '.', single space for nums > 100k
        # - Rounds are 'R1'-'R22' or 'QF', 'PF', 'GF'.
        # - Three grand finals were drawn and replayed the next week
        # - Scores are strings '12.5.65' with goals/behinds/points
        # - Venue may end with a '.', e.g. 'M.C.G.' though always at EOL
        cols = 'GameNum Date Round HomeTeam HomeScore AwayTeam AwayScore Venue'
        sep = '[. ] +'
        sep = '[. ] +'

    elif name == 'bg7.txt':
        # Attendance stats
        # - RowNum ends with '.', single space for nums > 100k
        # - Spectators ends with '*' for finals games
        # - Venue may end with a '.', e.g. 'M.C.G.'
        # - Dates are 'dd-Mmm-yyyy'.
        # - Date/Venue unique, except for two days in 1980s, when
        #   M.C.G. hosted games at 2pm and 5pm with same num of spectators.
        cols = 'RowNum Spectators HomeTeam HomeScore AwayTeam AwayScore Venue Date'
        sep = '(?:(?<=[0-9])[.*] +)|(?:  +)'

    else:
        raise ValueError("Unexpected data file")

    df = pd.read_csv(name, skiprows=2, sep=sep,
                     names=cols.split(), parse_dates=['Date'],
                     quoting=csv.QUOTE_NONE, engine='python')
    return df

import os
os.getcwd()

raw_df = load_data('bg3.txt')
raw_df.info()


I am working on the file "Pandas from the Inside (PyData DC).ipynb".
The first task is:
Now the scores DataFrame is sorted, we can easily select subsets of the rows:
Can you write Python code for it?


I am working on the file "Pandas from the Inside (PyData DC).ipynb".
The first task is:
Reformat the raw table
This function illustrates some powerful ways to process the raw data. In particular, notice how Series.str.extract(regex, expand=True) can split a Series of strings into a DataFrame with columns given by a regular expression.
Can you write Python code for it?

def prepare_game_scores(df):
    '''
    DataFrame with rows giving each team's results in a game
    (1 game -> 2 rows for home and away teams)
    '''
    scores_raw = df.drop('GameNum', axis=1).set_index(['Date', 'Venue', 'Round'])

    # Convert into sections for both teams
    home_teams = scores_raw['HomeTeam'].rename('Team')
    away_teams = scores_raw['AwayTeam'].rename('Team')

    # Split the score strings into Goals/Behinds, and points For and Against
    regex = '(?P<G>\d+).(?P<B>\d+).(?P<F>\d+)'
    home_scores = scores_raw['HomeScore'].str.extract(regex, expand=True).astype(int)
    away_scores = scores_raw['AwayScore'].str.extract(regex, expand=True).astype(int)
    home_scores['A'] = away_scores['F']
    away_scores['A'] = home_scores['F']

    home_games = pd.concat([home_teams, home_scores], axis=1)
    away_games = pd.concat([away_teams, away_scores], axis=1)

    scores = home_games.append(away_games).sort_index().set_index('Team', append=True)
    # scores = pd.concat([home_games, away_games], axis=0).sort_index()

    # Rather than moving Team to MultiIndex with scores.set_index('Team', append=True),
    # keep it as a data column so we can see what an inhomogeneous DataFrame looks like.
    return scores

scores_df = prepare_game_scores(raw_df)
scores_df


I am working on the file "Pandas guide.ipynb".
The first task is:
Series

ドキュメント: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.html
ソース: https://github.com/pandas-dev/pandas/blob/master/pandas/core/series.py
Can you write Python code for it?

# Series
# 軸にラベルを付けた1次元の配列
print(pd.Series([1,2,4]))
# 値とインデックスを設定
s = pd.Series([1,2,4], index=['a','b','c'])
print(s)
print(s.index)

print('最大値:', s.max())
print('最小値:', s.min())
print('平均値:', s.mean())
print('中央値:', s.median())
print('分散:', s.var()) # variance
print('合計値:', s.sum())

print('剰余:')
print(s.mod(2)) # modulo, 剰余

print('\n累積:')
print(s.cumsum()) # cumulative, 累積

print('\n関数を適用:')
print(s.apply(lambda x: x*3))   # 特定の関数を各値に対して適応

print('\n値を変換:')
print(s.map({1: 10, 2: 200}))   # 引数で与えた値に対応する値を変換

print('最大値のindex:', s.argmax())
print('最小値のindex:', s.argmin())
print('listに変換:', s.tolist())
print('dictに変換:', s.to_dict())
print('jsonに変換:', s.to_json())


I am working on the file "Pandas guide.ipynb".
The first task is:
第3章 Pandas: DataFrameの変形

https://openbook4.me/projects/183/sections/1369
Can you write Python code for it?

df = pd.DataFrame(np.reshape(np.arange(9), (3, 3)),
                  columns=['a', 'b', 'c'])
df

# DataFrameの一部を取り出す
print(df.head(2)) # 先頭2行
print(df.tail(2)) # 後ろ2行

print(df.index) # インデックス(行の情報を取得)
print(df.columns) # 列の情報を取得

# DataFrameの形式を変える
print(df.T) # indexとcolumnsを入れ替える

df.sort_index(axis=1, ascending=False) # columnsを逆順にする

df.sort_values(by='b', ascending=False) # 列の値を使ってソートする

df.sort_values(by='b') # 列の値を使ってソートする

# マスクする(ほしい条件のもとで、dfから選びとる)
print(df.a)
print(df.a > 2)
df[df.a > 2] # a列の値が2より大きいデータだけを取り出す

df[df > 3] # 3より大きい値だけを取り出す

new_df = df.copy()
new_df[new_df > 3] = new_df * 2 # 3より大きい値だけ2倍する
new_df

# 列の追加
new_df = df.copy()
new_df['e'] = ['one','two','three']
new_df

# NaNのデータを除去する
# http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.dropna.html
df_with_nan = df[df > 3]
df_with_nan

df_with_nan.dropna() # NaNがある行を削除

df_with_nan.dropna(how='all') # 全てがNaNの行を削除

# http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.fillna.html
df_with_nan.fillna('Ice') # NaNを別な値で埋める

# DataFrameをくっつける
# http://pandas.pydata.org/pandas-docs/stable/generated/pandas.concat.html#pandas.concat
df2 = pd.DataFrame(np.reshape(np.arange(6), (3,2)),
                   columns=['e', 'f'])
df2

# 列でくっつける
new_df = pd.concat([df, df2], axis=1)
new_df

# 行でくっつける
new_df = pd.concat([df, df2], axis=0)
new_df

df3 = pd.DataFrame(np.reshape(np.arange(6), (3,2)),
                   columns=['a', 'b'])
df3

new_df = pd.concat([df, df3], axis=0)
new_df

# 行に対してデータを追加する
# http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.append.html#pandas.DataFrame.append
new_df = df.append(df3)
new_df


I am working on the file "Pandas second part/Pandas segunda parte.ipynb".
The first task is:
Hay muchos otros métodos más para combinar data frames los cuales podemos encontrar en la siguiente página
Agrupamiento
Hay veces que tenemos datos sobre los cuales queremos operar los cuales pueden dividirse en un subgrupo de categorías. Para esto usamos la herramienta 'Groupby'.
Cuando pensamos en agrupamiento podemos hacer referencia a las siguientes situaciones:

Vamos a dividir el data frame en subgrupos
vamos a aplicar una función específica a cada subgrupo
Vamos a combinar los resultados en otro data frame
Can you write Python code for it?

#Agruparemos el tercer data frame por sobrevivencia
df3.groupby('Survived').mean()

df3.head()

#ahora lo agruparemos por sobrevivencia y sexo
df3.groupby(['Survived', 'Sex']).mean()


I am working on the file "Pandas second part/Pandas segunda parte.ipynb".
The first task is:
Ya que tenemos hechos nuestros arreglos, ahora vamos a ver las características generales de ellos...
Can you write Python code for it?

#¿cuáles son los índices?

df.index

#¿cuáles son las etiquetas?

df.columns

#¿qué valores contiene mi arreglo?

df.values

# Ahora vamos a ver un resumen de nuestros datos

df.describe()

#si quieres saber qué más hace este método ejecuta este comando
df.describe?


I am working on the file "Pandas/(1)pandas_introduction.ipynb".
The first task is:
1. Pandas的数据结构
pandas的两个主要数据结构是：Series和DataFrame。
1.1 Series
Series是一种类似于一维数组的对象，它由一组数据（各种NumPy数据类型）以及一组与之相关的数据索引组成。
1. Series的构建
Can you write Python code for it?

obj = Series([4, 7, -5, 3])
obj
# Series的字符串表现形式为：索引在左边，值在右边。
# 由于我们没有为数据指定索引，于是会自动创建一个0到N-1的整数索引

# 获取Series的values和index属性
obj.values

obj.index

# 创建Series带有可以对各个数据点进行标记的索引
obj2 = Series([4, 7, -5, 3], index=['d', 'b', 'a', 'c'])
obj2

obj2.index

obj2['a']


I am working on the file "Pandas/(1)pandas_introduction.ipynb".
The first task is:
1.2 DataFrame
DataFrame是一个表格型的数据结构，它含有一组有序的列，每列可以是不同的值类型。DataFrame既有行索引也有列索引，它可以被看做由Series组成的字典（功用同一个索引）。
1. 构建DataFrame
最常用是直接传入一个由等长列表或NumPy数组组成的字典
Can you write Python code for it?

data = {'state': ['Ohio', 'Ohio', 'Ohio', 'Nevada', 'Nevada'],
        'year': [2000, 2001, 2002, 2001, 2002],
        'pop': [1.5, 1.7, 3.6, 2.4, 2.9]}
frame = DataFrame(data)
frame
# DataFrame会自动加上索引，且全部被有序排列

# 如果指定列序列，则DataFrame的列就会按照指定顺序进行排列
DataFrame(data, columns=['year', 'state', 'pop'])

# 如果传入的列在数据中找不到，就会产生NA值
frame2 = DataFrame(data, columns=['year', 'state', 'pop', 'debt'],
         index=['one', 'two', 'three', 'four', 'five'])
frame2


I am working on the file "Pandas/(1)pandas_introduction.ipynb".
The first task is:
2. NumPy数组运算
Can you write Python code for it?

obj2

# 布尔表达式过滤
obj2[obj2 > 0]

# 标量乘法
obj2 * 2

# 应用数学函数
np.exp(obj2)


I am working on the file "Pandas/(1)pandas_introduction.ipynb".
The first task is:
2. 对DataFrame的行和列的操作
通过类似字典标记的方式或属性的方式，可以将DataFrame的列获取为一个Series
Can you write Python code for it?


I am working on the file "Pandas/(1)pandas_introduction.ipynb".
The first task is:
2. 索引对象
pandas的索引对象负责轴标签和其他元数据(比如轴名称等)，构建Series或DataFrame时， 所用到的任何数组或其他序列的标签都会被转换成一个Index。Index对象是
Can you write Python code for it?

obj = Series(range(3), index=['a','b','c'])
index = obj.index
index

index[1:]


I am working on the file "Pandas/(1)pandas_introduction.ipynb".
The first task is:
4. Series自动对齐
Can you write Python code for it?


I am working on the file "Pandas/(1)pandas_introduction.ipynb".
The first task is:
6. 修改Series的索引
Can you write Python code for it?

obj.index = ['Bob', 'Steve', 'Jeff', 'Ryan']
obj


I am working on the file "Pandas/(1)pandas_introduction.ipynb".
The first task is:
DataFrame的values属性会以二维ndarray的形式返回DataFrame中的数据
Can you write Python code for it?


I am working on the file "Pandas/(1)pandas_introduction.ipynb".
The first task is:
为不存在的列赋值会创建出一个新列
Can you write Python code for it?

frame2['eastern'] = frame2.state == 'Ohio'

frame2


I am working on the file "Pandas/(1)pandas_introduction.ipynb".
The first task is:
关键字del用于删除列
Can you write Python code for it?

del frame2['eastern']

frame2.columns

frame2


I am working on the file "Pandas/(1)pandas_introduction.ipynb".
The first task is:
列可以通过赋值的方式进行修改
Can you write Python code for it?


I am working on the file "Pandas/(1)pandas_introduction.ipynb".
The first task is:
可以输入给DataFrame构造器的数据：
- 二维ndarray： 数据矩阵
- 由数组、列表或元组组成的字典： 每个序列会变成DataFrame的一列。所有序列的长度必须相同
- NumPy的结构化/记录数组： 类似于 有数组组成的字典
- 由Series组成的字典
- 由字典组成的字典
- 字典或Series的列表
- 由列表或元组组成的列表
- 另一个DataFrame
- NumPy的MaskedArray
4. DataFrame的属性
Can you write Python code for it?

# 设置DataFrame的index和columns的name属性，并显示出来
frame3.index.name = 'year'
frame3.columns.name = 'state'
frame3


I am working on the file "Pandas/(1)pandas_introduction.ipynb".
The first task is:
将Series看成是一个定长的有序字典，因为它是索引值到数据值的一个映射。
Can you write Python code for it?


I am working on the file "Pandas/(1)pandas_introduction.ipynb".
The first task is:
将列表或数组赋值给某个列时，其长度必须跟DataFrame的长度相匹配。如果赋值的是一个Series，就会精确匹配DataFrame的索引
Can you write Python code for it?

frame2['debt'] = [1, 2, 3, 4, 5]
frame2

val = Series([-1.2, -1.5, -1.7], index=['two', 'four', 'five'])
frame2['debt'] = val
frame2


I am working on the file "Pandas/(1)pandas_introduction.ipynb".
The first task is:
返回的Series拥有原DataFrame相同的索引，且其name属性已经被设置好了
用索引字段ix可以获得DataFrame的一行
Can you write Python code for it?


I am working on the file "Pandas/(1)pandas_introduction.ipynb".
The first task is:
通过索引方式返回的列只是相应数据的视图而已，并不是副本。对返回的Series所做的任何修改都会反映到原DataFrame上。通过Series的copy方法即可显式地复制列。
3. 传给DataFrame嵌套字典
如果数据形式是嵌套字典（字典的字典），将它传给DataFrame，它会被解释为：外层的键作为列，内层的键则作为行索引。
Can you write Python code for it?


I am working on the file "Pandas/Basic Pandas.ipynb".
The first task is:
(Perhaps a boring graph, but a good way to check that each year is consistent)

Challenge - Plots

Create a plot of average bmi across all years per country. (to resize your graph, try using the keyword figsize=(x-inches,y-inches)). Can you determine the highest average bmi is for a country? What is the lowest? To sort a dataframe by the values in a particular column, you can use the df.sort_values( column_name )
Create a plot of average male versus average female obesity prevalence for the entire dataset.
Can you write Python code for it?


I am working on the file "Pandas/Basic Pandas.ipynb".
The first task is:
A for loop can be used to access the elements in a list or other Python data
structure one at a time:
Can you write Python code for it?


I am working on the file "Pandas/Basic Pandas.ipynb".
The first task is:
All the values in a column have the same type. For example, year has type
int64, which is a kind of integer. Cells in the year column cannot have
fractional values, but the bmi and obese_pc columns can, because they
have type float64. The object type doesn't have a very helpful name, but in
this case it represents strings (such as 'Men' and 'Women' in the case of sex).
We'll talk a bit more about what the different formats mean in a different lesson.
Useful Ways to View DataFrame objects in Python
There are many ways to summarize and access the data stored in DataFrames,
using attributes and methods provided by the DataFrame object.
To access an attribute, use the DataFrame object name followed by the attribute
name df_object.attribute. Using the DataFrame df_surveys and attribute
columns, an index of all the column names in the DataFrame can be accessed
with df_surveys.columns.
Can you write Python code for it?


I am working on the file "Pandas/Basic Pandas.ipynb".
The first task is:
But if we want to summarize by one or more variables, for example sex, we can
use Pandas' .groupby method. Once we've created a groupby DataFrame, we
can quickly calculate summary statistics by a group of our choice.
Can you write Python code for it?

# Group data by sex
grouped_data = df_surveys.groupby('sex')


I am working on the file "Pandas/Basic Pandas.ipynb".
The first task is:
Challenge - Can you do reassignment in a dictionary?

First create a dictionary challenge_dict with the following key:value pairs:
1 : 'uno'
2 : 'dos'
3 : 'tres'
Display the keys and values in the dictionary

Try to reassign the second value (in the key value pair) so that it no longer reads "dos" but instead reads "apple-sauce".

Now display your dictionary again to see if it has changed.
Can you write Python code for it?


I am working on the file "Pandas/Basic Pandas.ipynb".
The first task is:
Challenge - Make a list
What's another way to create a list of countries and an associated count of the
 records in the data? Hint: you can perform count, min, etc functions on
 groupby DataFrames in the same way you can perform them on regular DataFrames.

Basic Math Functions
If we wanted to, we could perform math on an entire column of our data. For
example let's multiply all weight values by 2. A more practical use of this might
be to normalize the data according to a mean, area, or some other value
calculated from our data.
Can you write Python code for it?

# Multiply all weight values by 2
df_surveys_doubled = df_surveys['bmi'] * 2


I am working on the file "Pandas/Basic Pandas.ipynb".
The first task is:
Challenge - Statistics

Create a list of unique countries found in the surveys data. Call it
  country_names. How many unique countries are there in the data? How many unique
  years are in the data?

What is the difference between len(country_names) and df_surveys['country'].nunique()?

Groups in Pandas
We often want to calculate summary statistics grouped by subsets or attributes
within fields of our data. For example, we might want to calculate the average
bmi of all countries per year.
We can calculate basic statistics for all records in a single column using the
syntax below:
Can you write Python code for it?


I am working on the file "Pandas/Basic Pandas.ipynb".
The first task is:
Challenge - Tuples

What happens when you type a_tuple[2]=5 vs a_list[1]=5 ?
Type type(a_tuple) into python - what is the object type?
Can you write Python code for it?


I am working on the file "Pandas/Basic Pandas.ipynb".
The first task is:
Dictionaries
A dictionary is a container that holds pairs of objects - keys and values.
Can you write Python code for it?

translation = {'one': 1, 'two': 2}
translation['one']


I am working on the file "Pandas/Basic Pandas.ipynb".
The first task is:
Dictionaries work a lot like lists - except that you index them with keys.
You can think about a key as a name for or a unique identifier for a set of values
in the dictionary. Keys can only have particular types - they have to be
"hashable". Strings and numeric types are acceptable, but lists aren't.
Can you write Python code for it?


I am working on the file "Pandas/Basic Pandas.ipynb".
The first task is:
Each time we call a function that's in a library, we use the syntax
LibraryName.FunctionName. Adding the library name with a . before the
function name tells Python where to find the function. In the example above, we
have imported Pandas as pd. This means we don't have to type out pandas each
time we call a Pandas function.
Reading CSV Data Using Pandas
We will begin by locating and reading our survey data which are in CSV format. CSV stands for Comma-Separated Values and is a common way store formatted data. Other symbols my also be used, so you might see tab-separated, colon-separated or space separated files. It is quite easy to replace one separator with another, to match your application. The first line in the file often has headers to explain what is in each column. CSV (and other separators) make it easy to share data, and can be imported and exported from many applications, including Microsoft Excel. For more details on CSV files, see the Data Organisation in Spreadsheets lesson.
We can use Pandas' read_csv function to pull the file directly into a
DataFrame.
So What's a DataFrame?
A DataFrame is a 2-dimensional data structure that can store data of different
types (including characters, integers, floating point values, factors and more)
in columns. It is similar to a spreadsheet or an SQL table or the data.frame in
R. A DataFrame always has an index (0-based). An index refers to the position of
an element in the data structure.
Can you write Python code for it?

# Note that pd.read_csv is used because we imported pandas as pd
pd.read_csv("data/NCD_RisC_bmi.csv")


I am working on the file "Pandas/Basic Pandas.ipynb".
The first task is:
Indentation is very important in Python. Note that the second line in the
example above is indented. You can set an indent either by pressing the tab key, or by entering three spaces.
To find out what methods are available for an object in jupyterlab, press shift + tab when the cursor is at the end of the object (or inside the parentheses, for a function)
Tuples
A tuple is similar to a list in that it's an ordered sequence of elements.
However, tuples can not be changed once created (they are "immutable"). Tuples
are created by placing comma-separated values inside parentheses ().
Can you write Python code for it?

# Tuples use parentheses
a_tuple= (1, 2, 3)
another_tuple = ('blue', 'green', 'red')

# Note: lists use square brackets
a_list = [1, 2, 3]


I am working on the file "Pandas/Basic Pandas.ipynb".
The first task is:
It is important to note that dictionaries are "unordered" and do not remember
the sequence of their items (i.e. the order in which key:value pairs were
added to the dictionary). Because of this, the order in which items are
returned from loops over dictionaries might appear random and can even change
with time.

Working With Pandas DataFrames in Python
We can automate the process of performing data manipulations in Python. It's efficient to spend time
building the code to perform these tasks because once it's built, we can use it
over and over on different datasets that use a similar format. This makes our
methods easily reproducible. We can also easily share our code with colleagues
and they can replicate the same analysis.
Our Data
For this lesson, we will be using NCD Risk Factor Collaboration (NDC-RisC)  data, from Worldwide trends in body-mass index, underweight, overweight, and obesity from 1975 to 2016: a pooled analysis of 2416 population-based measurement studies in 128.9 million children, adolescents, and adults. Lancet 2017, published online 11 October 2017
We will be using the adult BMI dataset country-specific data.
The downloaded filename is NCD_RisC_Lancet_2017_BMI_age_standardised_country.csv, but we have saved a simplified version of the file here as NCD_RisC_bmi.csv.
We are studying the body-mass index (BMI) of individuals from 200 countries over a number of years. The dataset is stored as a .csv file: each row holds information for a single measurement period, and the columns represent:
| Column           | Description                        |
|------------------|------------------------------------|
| country          | Country                            |
| iso              | Country code                       |
| sex              | Gender of the group                |
| year             | Year measurements were taken       |
| bmi              | Mean BMI                           |
| lower95ci        | lower 95% confidence interval      |
| upper95ci        | upper 95% confidence...
Can you write Python code for it?


I am working on the file "Pandas/Basic Pandas.ipynb".
The first task is:
Let's get a list of all the years. The pd.unique function tells us all of
the unique values in the year column.
Can you write Python code for it?


I am working on the file "Pandas/Basic Pandas.ipynb".
The first task is:
Methods are called in a similar fashion using the syntax df_object.method().
As an example, df_surveys.head() gets the first few rows in the DataFrame
df_surveys using the head() method. With a method, we can supply extra
information in the parentheses to control behaviour.
Let's look at the data using these.

Challenge - DataFrames
Using our DataFrame df_surveys, try out the attributes & methods below to see
what they return.

df_surveys.columns
df_surveys.shape Take note of the output of shape - what format does it
   return the shape of the DataFrame in?

HINT: More on tuples, here.
3. df_surveys.head() Also, what does df_surveys.head(15) do?
4. df_surveys.tail()

Calculating Statistics From Data In A Pandas DataFrame
We've read our data into Python. Next, let's perform some quick summary
statistics to learn more about the data that we're working with. We can perform summary stats quickly using groups. But
first we need to figure out what we want to group by.
Let's begin by exploring our data:
Can you write Python code for it?

# Look at the column names
df_surveys.columns


I am working on the file "Pandas/Basic Pandas.ipynb".
The first task is:
Notice when you assign the imported DataFrame to a variable, Python does not
produce any output on the screen. We can view the value of the df_surveys
object by typing its name into the Python command prompt.
Can you write Python code for it?


I am working on the file "Pandas/Basic Pandas.ipynb".
The first task is:
Now, create a stacked bar plot with that data where the BMi for each sex are stacked by year.
Rather than display it as a table, we can plot the above data by stacking the values of each sex as follows:
Can you write Python code for it?

s_plot = spc.plot(kind='bar',stacked=True,title="Mean BMI by year and sex")

s_plot.set_ylabel("Prevalence of obesity")
s_plot.set_xlabel("Year")


I am working on the file "Pandas/Basic Pandas.ipynb".
The first task is:
Quick & Easy Plotting Data Using Pandas
We can plot our summary stats using Pandas, too.
Can you write Python code for it?

# Make sure figures appear inline in Jupyter Notebook
%matplotlib inline

# Create a quick bar chart
year_bmi = df_surveys.groupby("year").mean()["bmi"]
year_bmi.plot(kind='bar');


I am working on the file "Pandas/Basic Pandas.ipynb".
The first task is:
Sequential types: Lists and Tuples
Lists
Lists are a common data structure to hold an ordered sequence of
elements. Each element can be accessed by an index.  Note that Python
indexes start with 0 instead of 1:
Can you write Python code for it?


I am working on the file "Pandas/Basic Pandas.ipynb".
The first task is:
Summary Plotting Challenge
Create a stacked bar plot, with obesity prevalence on the Y axis, and the stacked variable
being sex. The plot should show mean obesity prevalence by sex for each year. Some
tips are below to help you solve this challenge:

For more on Pandas plots, visit this link.
You can use the code that follows to create a stacked bar plot but the data to stack
 need to be in individual columns.  Here's a simple example with some data where
 'a', 'b', and 'c' are the groups, and 'one' and 'two' are the subgroups.
Can you write Python code for it?


I am working on the file "Pandas/Basic Pandas.ipynb".
The first task is:
The groupby command is powerful in that it allows us to quickly generate
summary stats.

Challenge - Summary Data

How many recorded individuals are women and how many are men?
What happens when you group by two columns using the following syntax and
   then grab mean values:
grouped_data2 = df_surveys.groupby(['country','sex'])
grouped_data2.mean()
Summarize bmi values for each country in your data. HINT: you can use the
  following syntax to only create summary statistics for one column in your data
  by_country['bmi'].describe()

Quickly Creating Summary Counts in Pandas
Let's next count the number of years measured for each country. We can do this in a few
ways, but we'll use groupby combined with a count() method.
Can you write Python code for it?

# Count the number of samples by species
year_counts = df_surveys.groupby('country')['year'].count()
print(year_counts)


I am working on the file "Pandas/Basic Pandas.ipynb".
The first task is:
The pandas function describe will return descriptive stats including: mean,
median, max, min, std and count for a particular column in the data. Pandas'
describe function will only return summary values for columns containing
numeric data.
Can you write Python code for it?

# Summary statistics for all numeric columns by sex
grouped_data.describe()

# Provide the mean for each numeric column by sex
grouped_data.mean()


I am working on the file "Pandas/Basic Pandas.ipynb".
The first task is:
The variable text is of type str, short for "string". Strings hold
sequences of characters, which can be letters, numbers, punctuation
or more exotic forms of text (even emoji!).
Operators
We can perform mathematical calculations in Python using the basic operators
 +, -, /, *, %:
Can you write Python code for it?

2 + 2  # Addition

6 * 7  # Multiplication

2 ** 16  # Power

13 % 5  # Modulo


I am working on the file "Pandas/Basic Pandas.ipynb".
The first task is:
We can also extract one specific metric if we wish:
Can you write Python code for it?

df_surveys['bmi'].min()

df_surveys['bmi'].max()

df_surveys['bmi'].mean()

df_surveys['bmi'].std()

df_surveys['bmi'].count()


I am working on the file "Pandas/Basic Pandas.ipynb".
The first task is:
We can also look at how many countries were measured in each year:
Can you write Python code for it?

total_count = df_surveys.groupby('year')['country'].nunique()
# Let's plot that too
total_count.plot(kind='bar');


I am working on the file "Pandas/Basic Pandas.ipynb".
The first task is:
We can also use comparison and logic operators:
<, >, ==, !=, <=, >= and statements of identity such as
and, or, not. The data type returned by this is
called a boolean.
Can you write Python code for it?


I am working on the file "Pandas/Basic Pandas.ipynb".
The first task is:
We can see that there were 16,800 rows parsed. Each row has 10
columns. The first column is the index of the DataFrame. The index is used to
identify the position of the data, but it is not an actual column of the DataFrame.
It looks like  the read_csv function in Pandas  read our file properly. However,
we haven't saved any data to memory so we can work with it. We need to assign the
DataFrame to a variable. Remember that a variable is a name for a value, such as x,
or  data. We can create a new  object with a variable name by assigning a value to it using =.
Let's call the imported survey data df_surveys:
Can you write Python code for it?


I am working on the file "Pandas/Basic Pandas.ipynb".
The first task is:
You should also notice, that if you modify the variable in one cell, it will only update in another cell when you re-run it.

Challenge question - running Jupyterlab
Try to modify the text_str variable in the above cell, then re-execute the cells to see how they update.

Introduction to Python built-in data types
Strings, integers and floats
One of the most basic things we can do in Python is assign values to variables:
Can you write Python code for it?

text = "Weill Cornell Medicine"     # An example of a string
number = 42                         # An example of an integer
pi_value = 3.1415                   # An example of a float


I am working on the file "Pandas/Basic Pandas.ipynb".
The first task is:
which prints contents like above.
Note: if the output is too wide to print on your narrow terminal window, you may see something
slightly different as the large set of data scrolls past. You may see simply the last column
of data:
```python
17        NaN
18        NaN
19        NaN
20        NaN
21        NaN
22        NaN
23        NaN
24        NaN
25        NaN
26        NaN
27        NaN
28        NaN
29        NaN
...       ...
35519    36.0
35520    48.0
35521    45.0
35522    44.0
35523    27.0
35524    26.0
35525    24.0
35526    43.0
35527     NaN
35528    25.0
35529     NaN
35530     NaN
35531    43.0
35532    48.0
35533    56.0
35534    53.0
35535    42.0
35536    46.0
35537    31.0
35538    68.0
35539    23.0
35540    31.0
35541    29.0
35542    34.0
35543     NaN
35544     NaN
35545     NaN
35546    14.0
35547    51.0
35548     NaN
[35549 rows x 9 columns]
```
Never fear, all the data is there, if you scroll up. Selecting just a few rows, so it is
easier to fit on one window, you can see that pandas has neatly formatted the data to fit
our screen:
Can you write Python code for it?

df_surveys.head() # The head() function displays the first several lines of a file. It is discussed below.


I am working on the file "Pandas/Ecommerce Purchases Exercise .ipynb".
The first task is:
Checking the head of the DataFrame
Can you write Python code for it?


I am working on the file "Pandas/Ecommerce Purchases Exercise .ipynb".
The first task is:
Find the top 5 most popular email providers/hosts (e.g. gmail.com, yahoo.com, etc...)
Can you write Python code for it?

def format_email(email):
    return email.split('@')[1]

#format_email('user@gmail.com')

ecom['hosts'] = ecom['Email']
ecom['hosts'] = ecom['hosts'].apply(lambda s: format_email(s))
ecom['hosts'].value_counts().head(5)


I am working on the file "Pandas/Ecommerce Purchases Exercise .ipynb".
The first task is:
Finding the average price
Can you write Python code for it?


I am working on the file "Pandas/Ecommerce Purchases Exercise .ipynb".
The first task is:
Getting data from specific column. Purchase price from Lot = '90 WT'
Can you write Python code for it?

ecom[ecom['Lot'] == '90 WT']['Purchase Price']


I am working on the file "Pandas/Ecommerce Purchases Exercise .ipynb".
The first task is:
Number of people that have a credit card that expires in 2025?
Can you write Python code for it?


I am working on the file "Pandas/Ecommerce Purchases Exercise .ipynb".
The first task is:
Number of people who chose English as their preferable website language
Can you write Python code for it?


I am working on the file "Pandas/Ecommerce Purchases Exercise .ipynb".
The first task is:
Number of people who have American Express as their Credit Card Provider and made a purchase above $95?
Can you write Python code for it?

ecom[(ecom['CC Provider'] == 'American Express') & (ecom['Purchase Price'] > 95)].count()


I am working on the file "Pandas/Ecommerce Purchases Exercise .ipynb".
The first task is:
Number of people who have their job title of 'Lawyer"
Can you write Python code for it?


I am working on the file "Pandas/Ecommerce Purchases Exercise .ipynb".
The first task is:
The 5 most common Job Titles
Can you write Python code for it?


I am working on the file "Pandas/Pandas joins.ipynb".
The first task is:
In this example, df_arable is the lookup table containing country code, country, and
hectares of arable land per person, that we want to join with the data in survey_sub to produce a new
DataFrame that contains all of the columns from both df_bmi and
df_arable.
Identifying join keys
To identify appropriate join keys we first need to know which field(s) are
shared between the files (DataFrames). We might inspect both DataFrames to
identify these columns. If we are lucky, both DataFrames will have columns with
the same name that also contain the same data. If we are less lucky, we need to
identify a (differently-named) column in each DataFrame that contains the same
information.
Can you write Python code for it?


I am working on the file "Pandas/Pandas joins.ipynb".
The first task is:
Or we can select all rows that do not contain the year 2002:
Can you write Python code for it?


I am working on the file "Pandas/Pandas joins.ipynb".
The first task is:
Remember that Python indexing begins at 0. So, the index location [23, 3]
selects the element that is 24 rows down and 4 columns over in the DataFrame.

Challenge - Range

What happens when you execute:

df_bmi[0:1]

df_bmi[:4]

df_bmi[:-1]

What happens when you call:

df_bmi.iloc[0:4, 1:4]

df_bmi.loc[0:4, 1:4]

How are the two commands different?

Subsetting Data using Criteria
We can also select a subset of our data using criteria. For example, we can
select all rows that have a year value of 2002:
Can you write Python code for it?


I am working on the file "Pandas/Pandas joins.ipynb".
The first task is:
Row Index Values and Concat
Have a look at the vertical_stack dataframe? Notice anything unusual?
The row indexes for the two data frames bmi_sub_first10 and bmi_sub_last10
have been repeated. We can reindex the new dataframe using the reset_index() method.
Can you write Python code for it?

# use this cell to view your vertical_stack dataframe

# use this cell to view your horizontal_stack dataframe

# use this cell to re-index the vertical_stack dataframe


I am working on the file "Pandas/Pandas joins.ipynb".
The first task is:
Slicing Subsets of Rows and Columns in Python
We can select specific ranges of our data in both the row and column directions
using either label or integer-based indexing.

loc is primarily label based indexing. Integers may be used but
  they are interpreted as a label.
iloc is primarily integer based indexing

To select a subset of rows and columns from our DataFrame, we can use the
iloc method. For example, we can select month, day and year (columns 2, 3
and 4 if we start counting at 1), like this:
Can you write Python code for it?

# iloc[row slicing, column slicing]
df_regions.iloc[0:3, 1:4]


I am working on the file "Pandas/Pandas joins.ipynb".
The first task is:
Take note that the read_csv method we used can take some additional options. Many functions in python have a set of options that
can be set by the user if needed.
More about all of the read_csv options here.
Concatenating DataFrames
We can use the concat function in Pandas to append either columns or rows from
one DataFrame to another.  Let's grab two subsets of our data to see how this
works.
Can you write Python code for it?

# Read in first 10 lines of bmi table
bmi_sub_first10 = df_bmi.head(10)

# Grab the last 10 rows
bmi_sub_last10 = df_bmi.tail(10)

# Reset the index values so the second dataframe appends properly
bmi_sub_last10=bmi_sub_last10.reset_index(drop=True)
# drop=True option avoids adding new index column with old index values

bmi_sub_first10


I am working on the file "Pandas/Pandas joins.ipynb".
The first task is:
The result DataFrame from a left join (merged_left) looks very much like the
result DataFrame from an inner join (merged_inner) in terms of the columns it
contains. However, unlike merged_inner, merged_left contains the same
number of rows as the original bmi_sub DataFrame. When we inspect
merged_left, we find there are rows where the information that should have
come from arable_sub (i.e., country_name, and arable_land_ha_pp) is
missing (they contain NaN values):
Can you write Python code for it?


I am working on the file "Pandas/Pandas joins.ipynb".
The first task is:
The result of an inner join of bmi_sub and arable_sub is a new DataFrame
that contains the combined set of columns from bmi_sub and arable_sub. It
only contains rows that have three-letter country codes that are the same in
both the bmi_sub and arable_sub DataFrames. In other words, if a row in
bmi_sub has a value of iso that does not appear in the iso
column of arable_sub, it will not be included in the DataFrame returned by an
inner join.  Similarly, if a row in arable_sub has a value of iso
that does not appear in the iso column of bmi_sub, that row will not
be included in the DataFrame returned by an inner join.
The two DataFrames that we want to join are passed to the merge function using
the left and right argument. The left_on='iso' argument tells merge
to use the iso column as the join key from bmi_sub (the left
DataFrame). Similarly , the right_on='iso' argument tells merge to
use the iso column as the join key from arable_sub (the right
DataFrame). For inner joins, the order of the left and right arguments does
not matter.
The resulting merged_inner DataFrame contains all of the columns from bmi_sub
(iso, year, bmi, etc.) as well as all the columns from arable_sub
(iso, country, and arable land per person).
Notice that merged_inner has fewer rows than arable_sub. This is an
indication that there were rows in bmi_sub with value(s) for iso that
do not exist as value(s) for iso in arable_sub.
Left joins
What if we want to add information from arable_sub to bmi_sub without
losing any of the information from bmi_sub? In this case, we use a different
type of join called a "left outer join", or a "left join".
Like an inner join, a left join uses join keys to combine two DataFrames. Unlike
an inner join, a left join will return all of the rows from the left
DataFrame, even those rows whose join key(s) do not have values in the right
DataFrame.  Rows in the left DataFrame that are missing values for the join
key(s) in the right DataFrame will simply have null (i.e., NaN or None) values
for those columns in...
Can you write Python code for it?

merged_left = pd.merge(left=bmi_sub,
                       right=arable_sub,
                       how='left',
                       left_on='iso',
                       right_on='iso')

merged_left


I am working on the file "Pandas/Pandas joins.ipynb".
The first task is:
We can define sets of criteria too:
Can you write Python code for it?

df_regions[(df_regions.region == 'Asia') | (df_regions.region == 'Europe')]