I am working on the file "Python_2_7.ipynb".
The first task is:
Interessanter als der absolute Wert für das Trinkgeld ist die Höhe des Trinkgelds im Verhältnis zur Gesamtsumme. Fügen wir eine entsprechende Spalte unserem Datensatz hinzu:
Can you write Python code for it?

tips['tip_perc'] = tips.tip/(tips.total_bill / 100)

sns.jointplot(x="total_bill", y="tip_perc", data=tips, size=8);

g = sns.FacetGrid(tips, col="sex",  row="smoker", margin_titles=True, size=4)
g.map(sns.plt.scatter, "total_bill", "tip_perc" );

sns.jointplot(x="size", y="tip_perc", data=tips, size=8);


I am working on the file "Python_2_7.ipynb".
The first task is:
Leider funktionieren die Voreinstellungen hier nicht befriedigend: der Kreis wird als Elipse dargestellt, deshalb muss man die Gesamtgrafik (figure) explizit auf ein Quadrat setzen (mehr dazu unten). Außerdem können wir die Farben explizit setzen:
Can you write Python code for it?

election =  [38.3, 26.8, 11.0, 9.3, 4.0, 3.8, 6.8 ]
labels = ["CDU/CSU","SPD","Die Grünen","Die Linke","FDP","AfD","Sonstige"]
plt.figure(figsize=(7,7))
plt.pie(election,labels=labels)
plt.show()


I am working on the file "Python_2_7.ipynb".
The first task is:
Mit dem keyword explode können wir die Grafik außerdem etwas dynamischer gestalten; die einzelnen Kuchenstücke werden nun herausgerückt. Wir übergeben dem keyword eine Liste, die soviele Werte enthält, wie wir Kuchenstücke haben. Die Größe des Wertes zeigt an, um wieviel das Stück herausgeschoben ist. <br/>
Außerdem können wir mit dem keyword autopct festlegen, wie die Werte angezeigt werden sollen. Hier wird die Formatierungssprache verwendet, die auch im Python-Handbuch in  Abschnitt 4.7.2 printf-style formatting erläutert wird.
Can you write Python code for it?

explode = [0,0,0.1,0.1,0.2,0.2,0.3]
plt.figure(figsize=(7,7))
plt.pie(election,labels=labels,explode=explode, autopct='%1.1f%%')
plt.show()


I am working on the file "Python_2_7.ipynb".
The first task is:
Nun stellen wir unsere Wahlergebnisse noch einmal dar:
Can you write Python code for it?

import matplotlib.pyplot as plt
import numpy as np
election =  [38.3, 26.8, 11.0, 9.3, 4.5, 3.8, 6.8 ]
labels = ["CDU/CSU","SPD","Die Grünen","Die Linke","FDP","AfD","Sonstige"]
color = ["black","red","green","red","yellow","brown","grey"]
#die x-Werte sind hier einfache Zahlen, da wir eigentlich nur eine Reihe brauchen
#da die Labels relativ breit sind, machen wir auch die bars mit width etwas breiter (default=0.8)
plt.bar(range(len(election)), election, color=color, width=0.9)
#aber nun wollen wir unsere Beschriftung der x-Achse ändern: sie soll die Dateien
#da die Ticks immer unter der ersten Linie einer Säule anfangen, wir aber den Text in der Mitte stehen haben wollen,
#müssen wir die Position der Labels im ersten Parameter etwas nach rechts verschieben, indem wir 0.4 hinzu addieren (Vektoralgebra!)
plt.xticks(np.arange(len(labels))+0.4, labels)
plt.show()


I am working on the file "Python_2_7.ipynb".
The first task is:
Table of Contents

0.1  Datenanalyse III: Viszualisierung mit Matplotlib und Seaborn

0.1.1  Andere Chart-Typen: Pie Charts, Bar Charts, Histogramme, usw.

0.1.1.1  Pie Charts

0.1.1.2  Bar Charts

0.1.1.3  Histogramme

0.1.1.4  Scatter Plots

0.2  Seaborn

0.2.1  Links

## Datenanalyse III: Viszualisierung mit Matplotlib und Seaborn

Wir haben matplotlib bereits kennengelernt, als wir Graphen geplottet haben. Es handelt sich dabei um eine komplexe Grafikbibliothek, die es erlaubt, auch sehr komplexe Grafiken zu erstellen. Leider ist dies mnachmal mit einer gewissen Komplexität der Bibliothek selbst verbunden. Matplotlib kennt drei Modi:

pylab (eine an MATLAB orientierte sehr einfache Schnittstelle, die aus...
Can you write Python code for it?

import matplotlib.pyplot as plt
x = [10,20,30,40]
y = [5, 3, 7, 4]
plt.plot(x,y)
plt.show()


I am working on the file "Python_2_7.ipynb".
The first task is:
Und auch hier gilt, dass wir mehrere Elemente, z.B. Linien, zeichnen können, bevor wir uns die Gesamtgrafik anzeigen lassen. (Das liegt übrigens daran, dass plt.hold() auf True voreingestellt ist; wenn man den Wert False setzt, dann wird nach jedem plot-Befehl der Canvas zurückgesetzt):
Can you write Python code for it?

x = np.arange(0,6)
plt.plot(x, [0.75*xi for xi in x])
plt.plot(x, [2*xi for xi in x])
plt.plot(x, [3.2*xi for xi in x])
plt.show()

#oder das gleiche kürzer mit etwas Vektorenrechnung:
x = np.arange(0,6)
plt.plot(x, 0.75*x, x, 2*x, x, 3.2*x)
plt.show()


I am working on the file "Python_2_8.ipynb".
The first task is:
A tutorial on statistical-learning for scientific data processing
Can you write Python code for it?

from sklearn import datasets
iris = datasets.load_iris()
data = iris.data
data.shape

digits = datasets.load_digits()
digits.images.shape
import pylab as pl
pl.imshow(digits.images[-1], cmap=pl.cm.gray_r)
pl.show()

data = digits.images.reshape((digits.images.shape[0], -1))


I am working on the file "Python_2_8.ipynb".
The first task is:
Model selection
Can you write Python code for it?

from sklearn import datasets, svm
digits = datasets.load_digits()
X_digits = digits.data
y_digits = digits.target
svc = svm.SVC(C=1, kernel='linear')
svc.fit(X_digits[:-100], y_digits[:-100]).score(X_digits[-100:], y_digits[-100:])

import numpy as np
X_folds = np.array_split(X_digits, 3)
y_folds = np.array_split(y_digits, 3)
scores = list()
for k in range(3):
    # We use 'list' to copy, in order to 'pop' later on
    X_train = list(X_folds)
    X_test  = X_train.pop(k)
    X_train = np.concatenate(X_train)
    y_train = list(y_folds)
    y_test  = y_train.pop(k)
    y_train = np.concatenate(y_train)
    scores.append(svc.fit(X_train, y_train).score(X_test, y_test))
print(scores)

from sklearn import cross_validation
k_fold = cross_validation.KFold(n=6, n_folds=3)
for train_indices, test_indices in k_fold:
     print('Train: %s | test: %s' % (train_indices, test_indices))

>>> kfold = cross_validation.KFold(len(X_digits), n_folds=3)
>>> [svc.fit(X_digits[train], y_digits[train]).score(X_digits[test], y_digits[test])
...          for train, test in kfold]

cross_validation.cross_val_score(svc, X_digits, y_digits, cv=kfold, n_jobs=-1)

>>> from sklearn.grid_search import GridSearchCV
>>> gammas = np.logspace(-6, -1, 10)
>>> clf = GridSearchCV(estimator=svc, param_grid=dict(gamma=gammas),
...                    n_jobs=-1)
>>> clf.fit(X_digits[:1000], y_digits[:1000])

>>> clf.best_score_

clf.best_estimator_.gamma == 1e-6

>>> # Prediction performance on test set is not as good as on train set
>>> clf.score(X_digits[1000:], y_digits[1000:])

cross_validation.cross_val_score(clf, X_digits, y_digits)

>>> from sklearn import linear_model, datasets
>>> lasso = linear_model.LassoCV()
>>> diabetes = datasets.load_diabetes()
>>> X_diabetes = diabetes.data
>>> y_diabetes = diabetes.target
>>> lasso.fit(X_diabetes, y_diabetes)

>>> # The estimator chose automatically its lambda:
>>> lasso.alpha_


I am working on the file "Python_2_8.ipynb".
The first task is:
Nearest Neighbour
Can you write Python code for it?

import numpy as np
from sklearn import datasets
iris = datasets.load_iris()
iris_X = iris.data
iris_y = iris.target
np.unique(iris_y)

# Split iris data in train and test data
# A random permutation, to split the data randomly
np.random.seed(0)
indices = np.random.permutation(len(iris_X))
iris_X_train = iris_X[indices[:-10]]
iris_y_train = iris_y[indices[:-10]]
iris_X_test  = iris_X[indices[-10:]]
iris_y_test  = iris_y[indices[-10:]]
# Create and fit a nearest-neighbor classifier
from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier()
knn.fit(iris_X_train, iris_y_train)

knn.predict(iris_X_test)

iris_y_test

diabetes = datasets.load_diabetes()
diabetes_X_train = diabetes.data[:-20]
diabetes_X_test  = diabetes.data[-20:]
diabetes_y_train = diabetes.target[:-20]
diabetes_y_test  = diabetes.target[-20:]

from sklearn import linear_model
regr = linear_model.LinearRegression()
regr.fit(diabetes_X_train, diabetes_y_train)

print(regr.coef_)

np.mean((regr.predict(diabetes_X_test)-diabetes_y_test)**2)

regr.score(diabetes_X_test, diabetes_y_test)

X = np.c_[ .5, 1].T
y = [.5, 1]
test = np.c_[ 0, 2].T
regr = linear_model.LinearRegression()

import pylab as pl
pl.figure()

np.random.seed(0)
for _ in range(6):
   this_X = .1*np.random.normal(size=(2, 1)) + X
   regr.fit(this_X, y)
   pl.plot(test, regr.predict(test))
   pl.scatter(this_X, y, s=3)
pl.show()

regr = linear_model.Ridge(alpha=.1)

pl.figure()

np.random.seed(0)
for _ in range(6):
   this_X = .1*np.random.normal(size=(2, 1)) + X
   regr.fit(this_X, y)
   pl.plot(test, regr.predict(test))
   pl.scatter(this_X, y, s=3)
pl.show()

alphas = np.logspace(-4, -1, 6)
from __future__ import print_function
print([regr.set_params(alpha=alpha
            ).fit(diabetes_X_train, diabetes_y_train,
            ).score(diabetes_X_test, diabetes_y_test) for alpha in alphas])

regr = linear_model.Lasso()
scores = [regr.set_params(alpha=alpha
            ).fit(diabetes_X_train, diabetes_y_train
            ).score(diabetes_X_test,...


I am working on the file "Python_2_9.ipynb".
The first task is:
As the last output shows, lxml treats the fragments differently depending on the context. Conversion to a string results in a fragment. Applying another xpath expression works on the whole document. Any pointers how to handle this? Obviously I could convert the results to strings, build new xml trees and use the xpath expression on them, but that seems more like a workaround than a working solution. (And I know that in this case 2 xpath expressions are not needed, but my real life text is much more complicated.)
Solution to this problem is a full stop (more or less)!
Can you write Python code for it?

for i in r:
    print(etree.tostring(i))
    print(i.xpath("./b")[0].text)

print('\a')

import winsound

winsound.Beep(400, 180)
winsound.Beep(700, 180)


I am working on the file "Python_2_9.ipynb".
The first task is:
Attribute werden als dictionary gespeichert.
Can you write Python code for it?

c = tree.xpath("//faq")
print(c[0].tag)
print(c[0].text)
print(c[0].attrib)


I am working on the file "Python_2_9.ipynb".
The first task is:
Hier nun mit namespace:
Can you write Python code for it?

a = tree.xpath("//tei:title", namespaces={'tei': 'http://www.tei-c.org/ns/1.0'})
len(a)

a[1].text

a = tree.xpath("//tei:sourceDesc//tei:title", namespaces={'tei': 'http://www.tei-c.org/ns/1.0'})
a[0].text

a = tree.xpath("//tei:sourceDesc//tei:persName", namespaces={'tei': 'http://www.tei-c.org/ns/1.0'})
a[0].text

a[0].tag

a[0].attrib


I am working on the file "Python_2_9.ipynb".
The first task is:
Opening a file
Can you write Python code for it?

f = open("faq.xml", encoding="utf-8")
tree = etree.parse(f)
etree.tostring(tree)


I am working on the file "Python_2_9.ipynb".
The first task is:
Using Xpath
Wenn wir einen Xpath-Ausdruck verwenden, dann erhalten wir eine Liste als Antwortmenge. Wenn nichts gefunden wurde, dann ist die Liste leer.
Can you write Python code for it?


I am working on the file "Python_2_9.ipynb".
The first task is:
Wenn wir Informationen zu den einzelnen Elementen in der Liste haben wollen, dann können wir entsprechende Klassenattribute verwenden. Wichtig sind hier 'tag', 'text' und 'attrib'.
Can you write Python code for it?

a[0].tag

a[0].text

b = tree.xpath("//q")
len(b)

for i in b:
    print(i.text)


I am working on the file "Python_2_9.ipynb".
The first task is:
Wenn wir bei einem xml-Baum mit namespaces xpath verwenden, müssen wir das beim xpath-Ausdruck berücksichtigen, sonst erhalten wir keinen Fehler, sondern eine leere Antwortmenge.
Can you write Python code for it?


I am working on the file "Python_2_9.ipynb".
The first task is:
Wenn wir den Namen des Attributes kennen, dann können wir eine einfachere Schreibweise verwenden:
Can you write Python code for it?


I am working on the file "Python_2_9.ipynb".
The first task is:
xpath with namespaces
Can you write Python code for it?

import io
f = io.StringIO("""
<?xml-model href="http://www.tei-c.org/release/xml/tei/custom/schema/relaxng/tei_all.rng" type="application/xml"
	schematypens="http://purl.oclc.org/dsdl/schematron"?>
<TEI xmlns="http://www.tei-c.org/ns/1.0">
   <teiHeader>
      <!-- Angabe zur digitalen Version. -->
      <fileDesc>
         <titleStmt>
            <title>Herr von Sacken - digitalisiertes Novellenschatz-Korpus</title>
            <author>
               <persName>Willibald Alexis</persName>
               <birth>1798-06-29</birth> <death>1871-12-16</death>
               <addName type="realName">Georg Wilhelm Heinrich Härig</addName>
            </author>
            <funder>Digital Humanities Cooperation</funder>
            <principal>Prof. Dr. Thomas Weitin</principal>
         </titleStmt>
     </fileDesc>
     <sourceDesc>
        <biblFull>
           <titleStmt>
              <title>Herr von Sacken</title>
             <author>
                <persName sex="1">Willibald Alexis</persName>
                 <birth>1798-06-29</birth> <death>1871-12-16</death>
                 <addName type="realName">Georg Wilhelm Heinrich Härig</addName>
              </author>
           </titleStmt>
         </biblFull>
    </sourceDesc>
  </teiHeader>
</TEI>
""")
tree = etree.parse(f)


I am working on the file "Python_3/Jupyter/z_delay_estimation.ipynb".
The first task is:
Can we get a faster result using Numba?
Can you write Python code for it?

import sys,os
from numba import jit
sys.path.insert(0, os.path.join(sys.path[0], '..'))
import extnumba as xn

@jit(nopython=True)
def localCorr_numba(reference, match, winlen, nlag,lag,qual):
    hwin = winlen//2
    lags = 2*nlag+1
    ns = reference.shape[0]
    hxw = hwin-nlag
    cor = np.zeros(lags)
    refSSQ = np.zeros(ns)
    matSSQ = np.zeros(ns)
    xn.winSSQ(reference,2*hxw+1,refSSQ)
    xn.winSSQ(match,2*hxw+1,matSSQ)
    for ir in range(hwin,ns-hwin):
        rbeg = ir - hxw
        rend = ir + hxw + 1
        mbeg = rbeg - nlag
        mend = rend + nlag
        for il in range(lags):
            lbeg = rbeg + il - nlag
            lend = lbeg + 2 * hxw + 1
            sum = 0.0
            for iref,imat in zip(range(rbeg,rend),range(lbeg,lend)):
                sum += reference[iref]*match[imat]
            den = refSSQ[ir]*matSSQ[lbeg+hxw]
            if den== 0.0:
                cor[il] = 0.0
            else:
                cor[il] = sum/den
        pos = np.argmax(cor)
        if pos>0 and pos<lags-1:
            cp = (cor[pos-1]-cor[pos+1])/(2.*cor[pos-1]-4.*cor[pos]+2.*cor[pos+1])
            lag[ir] = pos-nlag+cp
            qual[ir] = cor[pos]
        else:
            lag[ir]=0.0
            qual[ir]=0.0

localCorr_numba(res,shifted,51,15,lag,qual)
fig = plt.figure(figsize=(12,2))
plt.plot(lag)
plt.plot(qual)

%timeit -o localCorr_numba(res,shifted,51,15,lag,qual)


I am working on the file "Python_Astroplan.ipynb".
The first task is:
Accurate Time - UT1
AstroPy calculates the times of events to a very high accuracy. To do this, is has to account for the fact that  Earth's rotation period is constantly changing due to tidal forces and changes in the Earth's moment of inertia.
To do this, AstroPy uses a time convention called UT1. This system is tied to the rotation of the Earth with repect to the positions of distant quasars. Since the Earth's rotation is constantly changing, the time system UT1 is constanly changing with repect to UTC.
The orientation of the Earth, which must be measured continuously to keep UT1 accurate. This measurement is logged by the International Earth Rotation and Reference Systems Service (IERS). They publish a "bulletin" with the most recent measurements of the Earth's orientation. This bulletin is constantly being updated.
You will run into occasions when you will get a warning that your dates are out of range of the IERS bulletin. To update the bulletin, run the follow block of code:
Can you write Python code for it?

from astroplan import download_IERS_A

download_IERS_A()


I am working on the file "Python_Astroplan.ipynb".
The first task is:
Airmass < 2, you are good to go.
Planning observation is easier with plots
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt

from astroplan import time_grid_from_range

from astroplan.plots import plot_sky, plot_airmass

plot_sky(my_target, mro, midnight_mro);

start_time = astro_set
end_time = astro_rise

observing_range = [astro_set, astro_rise]

time_grid = time_grid_from_range(observing_range)

plot_sky(my_target, mro, time_grid);


I am working on the file "Python_Astroplan.ipynb".
The first task is:
Can you see the object?
Can you write Python code for it?


I am working on the file "Python_Astroplan.ipynb".
The first task is:
Different Date Formats
Can you write Python code for it?

print("The current Julian Date is {0:.2f}".format(current_time.jd))

print("The current Modified Julian Date is {0:.2f}".format(current_time.mjd))

print("The current unix Epoch is {0:.2f}".format(current_time.unix))  # Seconds since (Jan 01, 1970 00:00:00 UTC)

print("The current fraction of a year is {0:.2f}".format(current_time.decimalyear))


I am working on the file "Python_Astroplan.ipynb".
The first task is:
Information at your location
Can you write Python code for it?

sunset_here = astrolab.sun_set_time(current_time, which='nearest')

print("Sunset will be at {0.iso} UTC".format(sunset_here))

print("Sunset will be at {0} local time".format(sunset_here.to_datetime(my_timezone)))


I am working on the file "Python_Astroplan.ipynb".
The first task is:
Local Siderial Time (LST) will tell you the Right Ascension on the meridian right now.

You can use a star chart to find what constellations are visible now.
Can you write Python code for it?

midnight_mro = mro.midnight(current_time, which='next')

mro.local_sidereal_time(midnight_mro)


I am working on the file "Python_Astroplan.ipynb".
The first task is:
MRO
The Manastash Ridge Observatory (MRO) is operated by the Astronomy Department of the University of Washington, for the training of graduate and undergraduate students, as well as for astronomical research.
Can you write Python code for it?

mro = Observer.at_site('mro')  # many observatories are built-in to astroplan

mro

sunset_mro = mro.sun_set_time(current_time, which='nearest')

print("Sunset at MRO will be at {0} local time".format(sunset_mro.to_datetime(my_timezone)))

(sunset_here - sunset_mro).to(u.min)


I am working on the file "Python_Astroplan.ipynb".
The first task is:
Math with Time and Dates
Can you write Python code for it?

print("In 1 hour and 25 minutes it will be {0} UTC".format(current_time + 1*u.h + 25*u.min))

Christmas = Time("2017-12-25 00:00:00", format='iso')

delta_xmas = Christmas - current_time

print("It is {0:.2f} days until christmas, which is {1:.2f} fortnights, or {2:.2f} seconds"
      .format(delta_xmas.to(u.d),delta_xmas.to(u.fortnight),delta_xmas.to(u.s)))


I am working on the file "Python_Astroplan.ipynb".
The first task is:
Most targets can be defined by name
Can you write Python code for it?

my_target = FixedTarget.from_name("m31")

my_target.coord

my_target.ra.hms


I am working on the file "Python_Astroplan.ipynb".
The first task is:
Nice high altitude - looking good
Can you write Python code for it?

# You can find the airmass by using the .secz method

altaz_my_target.secz


I am working on the file "Python_Astroplan.ipynb".
The first task is:
Not looking good
Can you write Python code for it?

# astroplan sets the default limits of the airmass plot to [3,0].
# If you want to see a target at a higher airmass you have to set the limits yourself.

fig,ax = plt.subplots(1,1)

plot_airmass(low_target, mro, time_grid)
ax.set_ylim([15,0]);


I am working on the file "Python_Astroplan.ipynb".
The first task is:
Object is up at midnight at MRO - good
Can you write Python code for it?

altaz_my_target = astrolab.altaz(midnight_mro, my_target)

altaz_my_target.alt, altaz_my_target.az


I am working on the file "Python_Astroplan.ipynb".
The first task is:
Places
Setting your location - Observer
Can you write Python code for it?

astrolab = Observer(longitude = -122.311473 * u.deg,
                    latitude = 47 * u.deg + 39 * u.arcmin + 15 * u.arcsec,
                    elevation = 63.4 * u.m,
                    timezone = 'US/Pacific',
                    name = "Astrolab"
                    )

astrolab


I am working on the file "Python_Astroplan.ipynb".
The first task is:
Plot the airmass of the target over the night
Can you write Python code for it?


I am working on the file "Python_Astroplan.ipynb".
The first task is:
Things
Objects in the sky - FixedTarget
You can define targets by coordinates
Can you write Python code for it?

coords = SkyCoord('02h19m00.0s', '+57d07m042s', frame='icrs')

ngc869 = FixedTarget(name='NGC869', coord=coords)

ngc869.ra

ngc869.ra.hms


I am working on the file "Python_Astroplan.ipynb".
The first task is:
This is good target for observation at MRO for most of the night
Not all targets can (or should) be observed at all locations
Can you write Python code for it?

low_target = FixedTarget.from_name("Diphda")

mro.target_is_up(midnight_mro, low_target)

plot_sky(low_target, mro, time_grid);

plot_airmass(low_target, mro, time_grid);


I am working on the file "Python_Astroplan.ipynb".
The first task is:
Time and Dates

All dates and times in are UTC: Coordinated Universal Time .

All Time calculation assume that the time is UTC.
UTC is related to  Greenwich Mean Time (GMT) but does not change with a change of seasons.
Time will default to 00:00:00 UTC.
Can you write Python code for it?

my_date1 = Time("2017-11-06 13:40:15", format='iso')

my_date1

my_date1.value

my_date2 = Time("2017-11-06", format='iso')

my_date2


I am working on the file "Python_Astroplan.ipynb".
The first task is:
When can you observe?
Astronomical twilight is when the Sun is 18 degrees below the horizon
Can you write Python code for it?

astro_set = mro.twilight_evening_astronomical(current_time, which='nearest')
astro_rise = mro.twilight_morning_astronomical(current_time, which='next')

print("Astronomical Evening Twilight starts at {0.iso} UTC".format(astro_set))
print("Astronomical Midnight is at {0.iso} UTC".format(midnight_mro))
print("Astronomical Morning Twilight starts at {0.iso} UTC".format(astro_rise))

observing_length = (astro_rise - astro_set).to(u.h)

print("You can observe for {0:.1f} at MRO tonight".format(observing_length))

# Local Times

print("Tonight's observing at MRO starts at {0},\n peaks at {1} and,\n ends at {2} local time"
      .format(astro_set.to_datetime(my_timezone),
              midnight_mro.to_datetime(my_timezone),
              astro_rise.to_datetime(my_timezone)))


I am working on the file "Python_Astroplan.ipynb".
The first task is:
Where can you see it a midnight?
Can you write Python code for it?

where_to_look = mro.altaz(midnight_mro, ngc869)

where_to_look.alt

where_to_look.az


I am working on the file "Python_Astroplan.ipynb".
The first task is:
Working with timezones (local time)

The python package pytz is used to try to deal with local timezones
Timezone List
Only use timezone conversions for printouts, NEVER calculations!
Working with tomezones is a quick path to maddness!
Can you write Python code for it?

my_timezone = pytz.timezone('US/Pacific')

local_now = current_time.to_datetime(my_timezone)

print("The current local time is {0}".format(local_now))

# Nepal is in a strange timezone!

everest_timezone = pytz.timezone('Asia/Kathmandu')

everest_local_now = current_time.to_datetime(everest_timezone)

print("The current local time on Mt. Everest is {0}".format(everest_local_now))


I am working on the file "Python_Astroplan.ipynb".
The first task is:
You can turn solar system objects into pseudo FixedTarget objects for observational planning
Can you write Python code for it?

moon_midnight = FixedTarget(name='Moon', coord=get_body('moon',midnight_mro))

moon_midnight


I am working on the file "Python_Astroplan_Constraints.ipynb".
The first task is:
Additional Constraints
from astroplan import CONSTRAINT

AtNightConstraint() - Constrain the Sun to be below horizon.
MoonIlluminationConstraint(min, max) - Constrain the fractional illumination of the Moon.
MoonSeparationConstraint(min, max) - Constrain the separation between the Moon and some targets.
SunSeparationConstraint(min, max) - Constrain the separation between the Sun and some targets.
Can you write Python code for it?

from astroplan import moon_illumination

moon_illumination(observe_start)

from astroplan import MoonSeparationConstraint

constraints.append(MoonSeparationConstraint(45*u.deg))

observing_table = observability_table(constraints, my_location, targets, time_range=observing_range)

print(observing_table)

fig,ax = plt.subplots(1,1)
fig.set_size_inches(10,5)

fig.tight_layout()

for i, my_object in enumerate(targets):

    if observing_table['ever observable'][i]:
        ax = plot_airmass(my_object, my_location, time_grid)

ax.legend(loc=0,shadow=True);


I am working on the file "Python_Astroplan_Constraints.ipynb".
The first task is:
But, you get to observe in Hawaii
Can you write Python code for it?

my_timezone = pytz.timezone('US/Hawaii')

my_location = Observer.at_site('gemini_north')

observe_start = my_location.sun_set_time(observe_date, which='nearest')
observe_end = my_location.sun_rise_time(observe_date, which='next')

print("Observing starts at {0.iso} UTC".format(observe_start))
print("Observing ends at {0.iso} UTC".format(observe_end))
print("Observing starts at {0} local".format(observe_start.to_datetime(my_timezone)))
print("Observing ends at {0} local".format(observe_end.to_datetime(my_timezone)))

# A complete list of built-in observatories can be found by:

#EarthLocation.get_site_names()

observing_length = (observe_end - observe_start).to(u.h)

print("You can observe for {0:.1f} tonight".format(observing_length))

observing_range = [observe_start, observe_end]


I am working on the file "Python_Astroplan_Constraints.ipynb".
The first task is:
Let us add another constraint
Can you write Python code for it?

constraints.append(AirmassConstraint(2))

observing_table = observability_table(constraints, my_location, targets, time_range=observing_range)

print(observing_table)


I am working on the file "Python_Astroplan_Constraints.ipynb".
The first task is:
Now to observations
Let us start with a external list of target objects:
Can you write Python code for it?

target_table = QTable.read('ObjectList.csv', format='ascii.csv')

target_table

targets = [FixedTarget(coord=SkyCoord(ra = RA*u.hourangle, dec = DEC*u.deg), name=Name)
           for Name, RA, DEC in target_table]

targets


I am working on the file "Python_Astroplan_Constraints.ipynb".
The first task is:
Observing Night
You are the most junior member of the team, so you get stuck observing on New Years Eve
Can you write Python code for it?

observe_date = Time("2018-01-01", format='iso')


I am working on the file "Python_Astroplan_Constraints.ipynb".
The first task is:
Plot the objects
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt

from astroplan import time_grid_from_range
from astroplan.plots import plot_sky, plot_airmass

time_grid = time_grid_from_range(observing_range)

fig,ax = plt.subplots(1,1)
fig.set_size_inches(10,10)

fig.tight_layout()

for my_object in targets:
    ax = plot_sky(my_object, my_location, time_grid)

ax.legend(loc=0,shadow=True);

fig,ax = plt.subplots(1,1)
fig.set_size_inches(10,5)

fig.tight_layout()

for my_object in targets:
    ax = plot_airmass(my_object, my_location, time_grid)

ax.legend(loc=0,shadow=True);


I am working on the file "Python_Astroplan_Constraints.ipynb".
The first task is:
You can do the same thing with:
Can you write Python code for it?

even_squares = [x**2 for x in range(10) if (x % 2 == 0)]

even_squares


I am working on the file "Python_Astroplan_Constraints.ipynb".
The first task is:
You can include if statements:
Can you write Python code for it?

even_squares = []

for x in range(10):

    if (x % 2 == 0):
        even_squares.append(x**2)

even_squares


I am working on the file "Python_Basics.ipynb".
The first task is:
Here we defined a list of words as word_list. Each of these words is automatically assigned an index with the first word "high" being at index 0. We also tried something new in the way we defined our range.
In the previous example, we defined our range to be 5 with range(5). In this example, we used range(len(word_list)). What we have is a function within a function. Layered functions are evaluated from the inside out, so len() is evaluated before range. len() is a function that takes a list as an argument and returns the length of the list. In this case, len(word_list) evaluates to 4. So we then are left with range(4) which looks more familiar.
Better yet, Python gives us a shortcut to iterate over lists of data:
Can you write Python code for it?

for single_word in word_list:
    print(single_word)


I am working on the file "Python_Basics.ipynb".
The first task is:
Looking over those functions, we've got some new terms and symbols as well as specific formatting. Since both functions are almost identical, we'll break down the first one:
+ The keyword def tells the interpretor that we are defining a new function.
+ area_rectangle is the name we chose for our function.
+ Directly after our function name is the familiar set of parentheses with names we chose for our arguments, length and width.
+ A colon at the end of our function definition line tells the interpretor that the function starts next. The rest of the function is indented 4 spaces.
+ We then create the variable area and define it in terms of the supplied arguments, length and width.
+ The keyword return ends the function and in this case sends the value stored in area that we calculated.
To use the function, we simply call it by name while supplying arguments for length and width. Since we called on our function from within the built-in function print you'll see two sets of embedded parentheses.
Now you give it a try. By following the format of the above code, write a function called square that takes one argument and returns the square of the argument. Remember, you will have to use the menu to create a new cell to type your code into.
Libraries and the Dot Operator
The best thing about functions is that we can make them once and then call on them as many times as we'd like. We wrote useful functions for figuring the area and perimeter of a rectangle, and then you wrote a function that gives you the square of a number.
Multiple functions can be grouped together and saved as a file called a library that we can call on later from whatever program we are writing. For example, it might be useful to take the square function you wrote and write some more functions such as sin, cos and square_root...
Can you write Python code for it?


I am working on the file "Python_Basics.ipynb".
The first task is:
Looping
A useful tool in programming is looping. This allows us to accomplish a task that must be done multiple times with less code. Python provides the for loop as one way to accomplish this:
Can you write Python code for it?

print("Hello!")

for i in range(5):
    print("Hello again!")

print("Okay, I'll be quiet now.")


I am working on the file "Python_Basics.ipynb".
The first task is:
Notice that each print statement prints on a new line.
Okay, we've gotten intimate with the print function and sending arguments to it. Let's create our own functions for area and perimeter.
Create Your Own Function
Can you write Python code for it?

def area_rectangle(length, width):
    area = length * width
    return area

def perimeter_rectangle(length, width):
    perimeter = 2 * length + 2 * width
    return perimeter

print(area_rectangle(8, 4))
print(perimeter_rectangle(8, 4))


I am working on the file "Python_Basics.ipynb".
The first task is:
The number 2 was placed into the variable x. We verified this by printing x.
The variable name x isn't very informative except in very specific cases. Using words as variable names will help with the readability of the code. In the following example, we'll also do some arithmetic manipulation with variables.
Can you write Python code for it?

length = 10
width = 5
area = length * width
perimeter = 2 * length + 2 * width

print("The area of the rectangle is")
print(area)
print("The perimeter of the rectangle is")
print(perimeter)


I am working on the file "Python_Basics.ipynb".
The first task is:
The variable i is named by us. It is typical to use i as it stands for "iterator" but the name is arbitrary. Notice the familiar colon and indention pattern to show the statements that are to be executed inside the loop. Any number of statements can be included.
Now lets iterate over an existing set of data. First, we'll create a variable. But instead of this variable being a placeholder for one piece of data, it'll hold a list of information. We'll use the length of the list as our range for the loop:
Can you write Python code for it?

word_list=["high", "energy", "proton", "collision"]

for i in range(len(word_list)):
    print(word_list[i])


I am working on the file "Python_Images.ipynb".
The first task is:
All of the normal numpy commands work with matrices (of any dimension)
Can you write Python code for it?

my_matrix.mean()        # mean of the whole matrix


I am working on the file "Python_Images.ipynb".
The first task is:
Colormap reference: http://matplotlib.org/examples/color/colormaps_reference.html
Can you write Python code for it?

fig, ax = plt.subplots(1,5,sharey=True)

fig.set_size_inches(12,6)

fig.tight_layout()

ax[0].imshow(test_image, cmap=plt.cm.viridis)
ax[0].set_xlabel('viridis')

ax[1].imshow(test_image, cmap=plt.cm.hot)
ax[1].set_xlabel('hot')

ax[2].imshow(test_image, cmap=plt.cm.magma)
ax[2].set_xlabel('magma')

ax[3].imshow(test_image, cmap=plt.cm.spectral)
ax[3].set_xlabel('spectral')

ax[4].imshow(test_image, cmap=plt.cm.gray)
ax[4].set_xlabel('gray');


I am working on the file "Python_Images.ipynb".
The first task is:
Creating images from math
Can you write Python code for it?

X = np.linspace(-5, 5, 500)
Y = np.linspace(-5, 5, 500)

X, Y = np.meshgrid(X, Y)     # turns two 1-d arrays (X, Y) into one 2-d grid

Z = np.sqrt(X**2+Y**2)+np.sin(X**2+Y**2)

Z.min(), Z.max(), Z.mean()


I am working on the file "Python_Images.ipynb".
The first task is:
Images are just arrays that can be sliced.

For common image formats the origin is the upper left hand corner
Can you write Python code for it?

fig, ax = plt.subplots(1,4)
fig.set_size_inches(12,6)

fig.tight_layout()

# You can show just slices of the image - Rememeber: The origin is the upper left corner

ax[0].imshow(my_doctor, cmap=plt.cm.gray)
ax[0].set_xlabel('Original')

ax[1].imshow(my_doctor[0:300,0:100], cmap=plt.cm.gray)
ax[1].set_xlabel('[0:300,0:100]')                 # 300 rows, 100 columns

ax[2].imshow(my_doctor[:,0:100], cmap=plt.cm.gray)       # ":" = whole range
ax[2].set_xlabel('[:,0:100]')                     # all rows, 100 columns

ax[3].imshow(my_doctor[:,::-1], cmap=plt.cm.gray);
ax[3].set_xlabel('[:,::-1]') ;                     # reverse the columns

fig, ax = plt.subplots(1,2)
fig.set_size_inches(12,6)

fig.tight_layout()

CutLine = 300

ax[0].imshow(my_doctor, cmap=plt.cm.gray)
ax[0].hlines(CutLine, 0, 194, color='b', linewidth=3)

ax[1].plot(my_doctor[CutLine,:], color='b', linewidth=3)
ax[1].set_xlabel("X Value")
ax[1].set_ylabel("Pixel Value")


I am working on the file "Python_Images.ipynb".
The first task is:
Math on images applies to every value (pixel)
Can you write Python code for it?

another_test_image = test_image + 8

print("The minimum value of the other image is {0:.2f}".format(another_test_image.min()))
print("The maximum value of the other image is {0:.2f}".format(another_test_image.max()))
print("The mean value of the other image is {0:.2f}".format(another_test_image.mean()))
print("The standard deviation of the other image is {0:.2f}".format(another_test_image.std()))


I am working on the file "Python_Images.ipynb".
The first task is:
Show the image represenation of test_image with a colorbar
Can you write Python code for it?


I am working on the file "Python_Images.ipynb".
The first task is:
Simple image manipulation
Can you write Python code for it?

from scipy import ndimage

fig, ax = plt.subplots(1,5)
fig.set_size_inches(14,6)

fig.tight_layout()

ax[0].imshow(my_doctor, cmap=plt.cm.gray)

my_doctor_2 = ndimage.rotate(my_doctor,45,cval=0.75)               # cval is the value to set pixels outside of image
ax[1].imshow(my_doctor_2, cmap=plt.cm.gray)                 # Rotate and reshape

my_doctor_3 = ndimage.rotate(my_doctor,45,reshape=False,cval=0.75) # Rotate and do not reshape
ax[2].imshow(my_doctor_3, cmap=plt.cm.gray)

my_doctor_4 = ndimage.shift(my_doctor,(10,30),cval=0.75)           # Shift image
ax[3].imshow(my_doctor_4, cmap=plt.cm.gray)

my_doctor_5 = ndimage.gaussian_filter(my_doctor,5)                # Blur image
ax[4].imshow(my_doctor_5, cmap=plt.cm.gray);


I am working on the file "Python_Images.ipynb".
The first task is:
WARNING! Common image formats DO NOT preserve dynamic range of original data!!

Common image formats: jpg, gif, png, tiff
Common image formats will re-scale your data values to [0:1]
Common image formats are NOT suitable for scientific data!
Can you write Python code for it?

plt.imsave('Splash.png', test_image, cmap=plt.cm.gray)     # Write the array I to a PNG file

my_png = plt.imread('Splash.png')                          # Read in the PNG file

print("The original data has a min = {0:.2f} and a max = {1:.2f}".format(test_image.min(), test_image.max()))

print("The PNG file has a min = {0:.2f} and a max = {1:.2f}".format(my_png.min(), my_png.max()))


I am working on the file "Python_Images.ipynb".
The first task is:
We just want to read in one of the three channels
Can you write Python code for it?

redfilter = plt.imread("./MyData/sphereR.jpg")[:,:,0]

redfilter.shape,redfilter.dtype

plt.imshow(redfilter,cmap=plt.cm.gray);

greenfilter = plt.imread("./MyData/sphereG.jpg")[:,:,0]
bluefilter = plt.imread("./MyData/sphereB.jpg")[:,:,0]

fig, ax = plt.subplots(1,3)
fig.set_size_inches(12,3)

fig.tight_layout()

ax[0].set_title("Red Filter")
ax[1].set_title("Green Filter")
ax[2].set_title("Blue Filter")

ax[0].imshow(redfilter,cmap=plt.cm.gray)
ax[1].imshow(greenfilter,cmap=plt.cm.gray)
ax[2].imshow(bluefilter,cmap=plt.cm.gray);


I am working on the file "Python_Images.ipynb".
The first task is:
You can work over just the rows or columns of the matrix

axis = 0 (COLUMNS)
axis = 1 (ROWS)
Can you write Python code for it?

my_matrix.mean(axis=0)   # mean of the columns

my_matrix.mean(axis=0)[0]  # mean of the 0th column

np.mean(my_matrix, axis=0)  # alternative

my_matrix.mean(axis=1)   # mean of the rows

my_matrix.flatten()      # convert to 1D (useful for some plotting)


I am working on the file "Python_Images_FITS.ipynb".
The first task is:
Accepted Data
Can you write Python code for it?


I am working on the file "Python_Images_FITS.ipynb".
The first task is:
FITS Tables - An astronomical example

Stellar spectra data from the ESO Library of Stellar Spectra
Can you write Python code for it?

my_spectra_file = './MyData/Star_G5.fits'

my_spectra_fits = fits.open(my_spectra_file)

my_spectra_fits.info()

spectra_data = my_spectra_fits[0].data
spectra_header = my_spectra_fits[0].header

spectra_header[0:15]

# The FITS header has the information to make an array of wavelengths

start_wavelength = spectra_header['CRVAL1']
num_of_points = spectra_header['NAXIS1']
width_of_points  = spectra_header['CDELT1']

end_wavelength = start_wavelength + (num_of_points * width_of_points)

wavelength_range = np.arange(start_wavelength,end_wavelength,width_of_points)

fig, ax = plt.subplots(2,1)
fig.set_size_inches(11,8.5)

fig.tight_layout()

# Full spectra

ax[0].plot(wavelength_range, spectra_data, color='b')
ax[0].set_ylabel("Flux")
ax[0].set_xlabel("Wavelength [angstroms]")

# Just the visible range with the hydrogen Balmer lines

ax[1].set_xlim(4000,7000)
ax[1].set_ylim(0.6,1.2)
ax[1].plot(wavelength_range, spectra_data, color='b')
ax[1].set_ylabel("Flux")
ax[1].set_xlabel("Wavelength [angstroms]")

H_Balmer = [6563,4861,4341,4102,3970,3889,3835,3646]

ax[1].vlines(H_Balmer,0,2, color='r', linewidth=3, alpha = 0.25);


I am working on the file "Python_Images_FITS.ipynb".
The first task is:
FITS files consist of at least two parts - Header and Data

A FITS file is comprised of segments called Header/Data Units (HDUs), where the first HDU is called the Primary HDU.
The primary data array can contain a 1-D spectrum, a 2-D image, or a 3-D data cube.
Every HDU consists of an ASCII formatted Header Unit.
Each header unit contains a sequence of fixed-length 80-character keyword (Cards)
Can you write Python code for it?

data_file = "./MyData/bsg01.fits"

my_fits_file = fits.open(data_file)

my_fits_file.info()

image_data = my_fits_file[0].data
image_header = my_fits_file[0].header

image_header

print("The image has a shape [height,width] of {0}".format(image_data.shape))
print("The image is made up of data of type {0}".format(image_data.dtype))


I am working on the file "Python_Images_FITS.ipynb".
The first task is:
FITS format preserves the full dynamic range of data
Can you write Python code for it?

print("The image has a maximum value of {0}".format(image_data.max()))
print("The image has a minimum value of {0}".format(image_data.min()))

fig, ax = plt.subplots(1,2)

fig.set_size_inches(12,6)

fig.tight_layout()

ax[0].imshow(image_data,cmap=plt.cm.gray)

ax[1].hist(image_data.flatten(),bins=20);


I am working on the file "Python_Images_FITS.ipynb".
The first task is:
Notice the origin is in the upper left corner (the image is upside down)
Can you write Python code for it?


I am working on the file "Python_Images_FITS.ipynb".
The first task is:
RGB - Pseudocolor Images
Can you write Python code for it?

from astropy.visualization import make_lupton_rgb

red_img = fits.getdata("./MyData/m51_IR.fits").astype(float)
green_img = fits.getdata("./MyData/m51_red.fits").astype(float)
blue_img= fits.getdata("./MyData/m51_blue.fits").astype(float)

clean_r =  red_img - np.median(red_img)
clean_g =  green_img - np.median(green_img)
clean_b =  blue_img - np.median(blue_img)

fig, ax = plt.subplots(1,3)
fig.set_size_inches(12,6)

fig.tight_layout()

ax[0].set_title("R")
ax[1].set_title("G")
ax[2].set_title("B")

ax[0].imshow(clean_r, origin='lower', cmap=plt.cm.gray)
ax[1].imshow(clean_g, origin='lower', cmap=plt.cm.gray);
ax[2].imshow(clean_b, origin='lower', cmap=plt.cm.gray);

image = make_lupton_rgb(clean_r, clean_g, clean_b, stretch = 5000)

fig, ax = plt.subplots(1,1)
fig.set_size_inches(6,6)

fig.tight_layout()

ax.imshow(image, origin='lower', cmap=plt.cm.gray);


I am working on the file "Python_Images_FITS.ipynb".
The first task is:
Rejected Data
Can you write Python code for it?


I am working on the file "Python_Images_FITS.ipynb".
The first task is:
Sigma clipping an image
Can you write Python code for it?

star_data.mean(), star_data.std(), star_data.max(), star_data.min()

clip_star = stats.sigma_clip(star_data, sigma=8, iters=5)

copy_data = np.copy(star_data)        # make a copy of the data to work with

copy_data[~clip_star.mask] = star_data.min()

fig, ax = plt.subplots(1,2)

fig.set_size_inches(12,6)

fig.tight_layout()

ax[0].imshow(star_data, origin='lower', cmap=plt.cm.gray)
ax[1].imshow(copy_data, origin='lower', cmap=plt.cm.gray);


I am working on the file "Python_Images_FITS.ipynb".
The first task is:
Stellar spectral classes
Can you write Python code for it?

import glob

star_list = glob.glob('./MyData/Star_*5.fits')

star_list

fig, ax = plt.subplots(1,1)
fig.set_size_inches(9,5)

fig.tight_layout()

# Full spectra

ax.set_ylabel("Flux")
ax.set_xlabel("Wavelength [angstroms]")
ax.set_ylim(0.0, 1.05)

for file in star_list:

    spectra = fits.getdata(file)

    spectra_normalized = spectra / spectra.max()

    ax.plot(wavelength_range, spectra_normalized, label=file)

ax.legend(loc=0,shadow=True);


I am working on the file "Python_Images_FITS.ipynb".
The first task is:
You can add and subtract images
Can you write Python code for it?

another_image_file = "./MyData/bsg02.fits"

another_image_data = fits.getdata(another_image_file)     # a quick way to just get the data

fig, ax = plt.subplots(1,2)
fig.set_size_inches(12,6)

fig.tight_layout()

ax[0].set_title("Original Image")
ax[1].set_title("Another Image")

ax[0].imshow(image_data, cmap=plt.cm.gray)
ax[1].imshow(another_image_data, cmap=plt.cm.gray);


I am working on the file "Python_Images_FITS.ipynb".
The first task is:
You can use masks on images
Can you write Python code for it?

copy_data = np.copy(image_data)        # make a copy of the data to work with

cut_off = 40

mask = np.where(copy_data > cut_off)
copy_data[mask] = 60                   # You can not just throw data away, you have to set it to something.

print("The cropped image has a maximum value of {0}".format(copy_data.max()))
print("The cropped image has a minimum value of {0}".format(copy_data.min()))


I am working on the file "Python_Images_FITS.ipynb".
The first task is:
You can use specific bins for histograms
Can you write Python code for it?

my_bins = np.arange(0,110,5)

my_bins

fig, ax = plt.subplots(1,2)

fig.set_size_inches(12,6)

fig.tight_layout()

ax[0].imshow(copy_data,cmap=plt.cm.gray)

ax[1].hist(image_data.flatten(),bins=my_bins)
ax[1].hist(copy_data.flatten(),bins=my_bins, alpha = 0.40);


I am working on the file "Python_Introduction.ipynb".
The first task is:
Create a variable
Can you write Python code for it?


I am working on the file "Python_Introduction.ipynb".
The first task is:
Creating Arrays
Numpy has a wide variety of ways of creating arrays: Array creation routines
Can you write Python code for it?

# a new array filled with zeros

array_0 = np.zeros(10)

array_0

# a new array filled with ones

array_1 = np.ones(10)

array_1

# a new array filled with evenly spaced values within a given interval

array_2 = np.arange(10,20)

array_2

# a new array filled with evenly spaced numbers over a specified interval (start, stop, num)

array_3 = np.linspace(10,20,5)

array_3

# a new array filled with evenly spaced numbers over a log scale. (start, stop, num, base)

array_4 = np.logspace(1,2,5,10)

array_4


I am working on the file "Python_Introduction.ipynb".
The first task is:
Datatypes
In computer programming, a data type is a classification identifying one of various types that data
can have.
The most common data type we will see in this class are:

Integers (int): Integers are the classic cardinal numbers: ... -3, -2, -1, 0, 1, 2, 3, 4, ...

Floating Point (float): Floating Point are numbers with a decimal point: 1.2, 34.98, -67,23354435, ...

Floating point values can also be expressed in scientific notation: 1e3 = 1000

Booleans (bool): Booleans types can only have one of two values: True or False. In many languages 0 is considered False, and any other value is considered True.

Strings (str): Strings can be composed of one or more characters: ’a’, ’spam’, ’spam spam eggs and spam’. Usually quotes (’) are used to specify a string. For example ’12’ would refer to the string, not the integer.

Collections of Data Types

Scalar: A single value of any data type.

List: A collection of values. May be mixed data types. (1, 2.34, ’Spam’, True) including lists of lists: (1, (1,2,3), (3,4))

Array: A collection of values. Must be same data type. [1,2,3,4] or [1.2, 4.5, 2.6] or [True, False, False] or [’Spam’, ’Eggs’, ’Spam’]

Matrix: A multi-dimensional array: [[1,2], [3,4]] (an array of arrays).
Can you write Python code for it?

a = 1
b = 2.3
c = 2.3e4
d = True
e = "Spam"

type(a), type(b), type(c), type(d), type(e)

a + b, type(a + b)

c + d, type(c + d)    # True = 1

a + e

str(a) + e


I am working on the file "Python_Introduction.ipynb".
The first task is:
Fancy masking
Can you write Python code for it?

mask3 = np.where(x >= 8)
x[mask3]

# Set all values of x that match mask3 to 0

x[mask3] = 0
x

mask4 = np.where(x != 0)
mask4

#Add 10 to every value of x that matches mask4:

x[mask4] += 100
x


I am working on the file "Python_Introduction.ipynb".
The first task is:
For loops are different in python.
You do not need to specify the beginning and end values of the loop
Can you write Python code for it?

z

for value in z:
    print(value)

for idx,val in enumerate(z):
    print(idx,val)

for idx,val in enumerate(z):
    if (val > 5):
        z[idx] = 0

for idx,val in enumerate(z):
    print(idx,val)


I am working on the file "Python_Introduction.ipynb".
The first task is:
Functions
In computer science, a function (also called a procedure, method, subroutine, or routine) is a portion
of code within a larger program that performs a specific task and is relatively independent of the
remaining code. The big advantage of a function is that it breaks a program into smaller, easier
to understand pieces. It also makes debugging easier. A function can also be reused in another
program.
The basic idea of a function is that it will take various values, do something with them, and return a result. The variables in a function are local. That means that they do not affect anything outside the function.
Below is a simple example of a function that solves the equation:
$ f(x,y) = x^2\ sin(y)$
In the example the name of the function is find_f (you can name functions what ever you want). The function find_f takes two arguments x and y, and returns the value of the equation to the main program. In the main program a variable named value_f is assigned the value returned by find_f. Notice that in the main program the function find_f is called using the arguments array_x and array_y. Since the variables in the function are local, you do not have name them x and y in the main program.
Can you write Python code for it?

def find_f(x,y):

    result = (x ** 2) * np.sin(y)           # assign the variable result the value of the function
    return result                           # return the value of the function to the main program

np.random.seed(42)

array_x = np.random.rand(10) * 10
array_y = np.random.rand(10) * 2.0 * np.pi

array_x, array_y

value_f = find_f(array_x,array_y)

value_f


I am working on the file "Python_Introduction.ipynb".
The first task is:
Help about a function:
Can you write Python code for it?


I am working on the file "Python_Introduction.ipynb".
The first task is:
Here is a link to all Numpy math functions.
Arrays

Each element of the array has a Value
The position of each Value is called its Index

Our basic unit will be the NumPy array
Can you write Python code for it?

np.random.seed(42)                 # set the seed - everyone gets the same random numbers
x = np.random.randint(1,10,20)     # 20 random ints between 1 and 10
x


I am working on the file "Python_Introduction.ipynb".
The first task is:
Loops are slow in Python. Do not use them if you do not have to!
Can you write Python code for it?

np.random.seed(42)
BigZ = np.random.random(10000)    # 10,000 value array
BigZ[:10]

# This is slow!

for Idx,Val in enumerate(BigZ):
    if (Val > 0.5):
        BigZ[Idx] = 0

BigZ[:10]

%%timeit

for Idx,Val in enumerate(BigZ):
    if (Val > 0.5):
        BigZ[Idx] = 0

# Masks are MUCH faster

mask = np.where(BigZ>0.5)
BigZ[mask] = 0

BigZ[:10]

%%timeit -o

mask = np.where(BigZ>0.5)
BigZ[mask] = 0


I am working on the file "Python_Introduction.ipynb".
The first task is:
Masking - The key to fast programs
Can you write Python code for it?

mask1 = np.where(x>5)
x, mask1

x[mask1], y[mask1]

mask2 = np.where((x>3) & (x<7))
x[mask2]


I am working on the file "Python_Introduction.ipynb".
The first task is:
NumPy (Numerical Python) is the fundamental package for scientific computing with Python.
Load the numpy library:
Can you write Python code for it?


I am working on the file "Python_Introduction.ipynb".
The first task is:
NumPy math works over an entire array:
Can you write Python code for it?

y = x * 2
y

sin(x)     # need to Numpy's math functions

np.sin(x)


I am working on the file "Python_Introduction.ipynb".
The first task is:
Print out the value of the variable
Can you write Python code for it?


I am working on the file "Python_Introduction.ipynb".
The first task is:
Slices
x[start:stop:step]

start is the first Index that you want [default = first element]
stop  is the first Index that you do not want [default = last element]
step  defines size of step and whether you are moving forwards (positive) or backwards (negative) [default = 1]
Can you write Python code for it?

x

x[0:4]           # first 4 items

x[:4]            # same

x[0:4:2]         # first four item, step = 2

x[3::-1]         # first four items backwards, step = -1

x[::-1]          # Reverse the array x

print(x[-5:])    # last 5 elements of the array x


I am working on the file "Python_Introduction.ipynb".
The first task is:
Sorting
Can you write Python code for it?

np.random.seed(13)                 # set the seed - everyone gets the same random numbers
z = np.random.randint(1,10,20)     # 20 random ints between 1 and 10
z

np.sort(z)

np.sort(z)[0:4]

# Returns the indices that would sort an array

np.argsort(z)

z, z[np.argsort(z)]

maskS = np.argsort(z)

z, z[maskS]


I am working on the file "Python_Introduction.ipynb".
The first task is:
The results of one function can be used as the input to another function
Can you write Python code for it?

def find_g(z):

    result = z / np.e
    return result

find_g(value_f)

find_g(find_f(array_x,array_y))


I am working on the file "Python_Introduction.ipynb".
The first task is:
There are lots of different methods that can be applied to a NumPy array
Can you write Python code for it?

x.size                   # Number of elements in x

x.mean()                 # Average of the elements in x

x.sum()                  # Total of the elements in x

x[-5:].sum()              # Total of last 5 elements in x

x.cumsum()                # Cumulative sum

x.cumsum()/x.sum()        # Cumulative percentage

x.


I am working on the file "Python_Introduction.ipynb".
The first task is:
or even easier:
Can you write Python code for it?


I am working on the file "Python_Misc/TMWP_CSV_Manipulation/File_Fragmenter_TestBook.ipynb".
The first task is:
About These Files
These scripts make use of reusable objects.  The object code is contained within them but can be deployed on your systems so that updates and enhancements then flow into the final scripts you build.  If you analyze the code, you will notice that abstract objects then have child objects to do the actual work.  In these objects is a seemingly useless function with no definition.  It is there as a placeholder.
The code as written will split or fuse files.  These placeholder functions can then be modified in the child objects to allow you to perform operations on the inputs ahead of creating the outputs.  The intent is to set up inherently extensible code where the coder can focus on what needs to be done to the files and let the underlying objects handle spitting / fusing as designed.
Can you write Python code for it?


I am working on the file "Python_Misc/TMWP_CSV_Manipulation/File_Fragmenter_TestBook.ipynb".
The first task is:
Code Testing
In the cells that follow:

Python scripts are tested from the command line on the csv files listed above
this provides a demo of command line output messages the scripts produce
and shows how to use the arguments for the scripts
code is written and tested on the assumption that input file(s) will be csv either with no index
or with a simple index of one column at column zero (column 0 has unique values and now header in the csv)
Note the following about arguments which tell the code if input/output will have an index:
if index related arguments have any value, code valuates them to boolean value of True
if index related arguments are left off, they will default to boolean value of False
expected usage for the index related arguments is pass them True or leave them off (do not set to False)

Getting Help:
Can you write Python code for it?

!python file_fragmenter.py -h

!python file_fuser.py --help


I am working on the file "Python_Misc/TMWP_CSV_Manipulation/File_Fragmenter_TestBook.ipynb".
The first task is:
Final test involving a large file.  Lack of --numRows (or -rw) argument means code will default to 10,000 rows per file.
Can you write Python code for it?

!python file_fragmenter.py -i TestFileLarge.csv -o TestFileLrgFrag

!python file_fuser.py -i TestFileLrgFrag -o TestFileLarge_Out.csv

# The End


I am working on the file "Python_Misc/TMWP_CSV_Manipulation/File_Fragmenter_TestBook.ipynb".
The first task is:
In above tests:

First test splits a single file into files with the starting pattern: TestFSwi_i
Second test then merges these files back into a single output file named: TestFSwi_i_Out.csv
Third test, merging the files into output without an index, includes the file "TestFSwi_i_Out.csv" in its output since this too fits the input file pattern

Test Using No Index in The Original Input File
This next batch of tests starts from an input csv that does not have an index on it.
Can you write Python code for it?

!python file_fragmenter.py -i TestFileSmall-NoIndx.csv -o TestFSnoX_ --numRows 27

!python file_fuser.py -i TestFSnoX_ -o TstFSnoX_Out.csv

!python file_fuser.py -i TestFSnoX_ -o TestFSnoX_Out_i.csv --output_index True


I am working on the file "Python_Misc/TMWP_CSV_Manipulation/File_Fragmenter_TestBook.ipynb".
The first task is:
In the previous batch of tests, the original input file did not have an index.  The file is split and then merged back together.  Then finally, the inputs are merged again, but this time into a file with an index.  The index is auto-generated and sorted to run from 0, 1, 2, ... etc.
Testing Using Single Column File
This next section repeats all of the tests before it using a csv with only one column of data in it.  These tests were performed for completeness.  Early drafts of the code worked on a single code but not on multiple columns.  Once code was revised to work on multiple columns, these tests were performed again to show it still works on a single column.  They also illustrate more permutations of the input parameter syntax.
Can you write Python code for it?

!python file_fragmenter.py --input TestFile_SingleCol-NoIndx.csv --outputStart TestSCFL_noX_

!python file_fuser.py -i TestSCFL_noX_ -o TstSCFL_noX_Out.csv

!python file_fuser.py -i TestSCFL_noX_ -o TestSCFL_noX_Out.csv -odx True

!python file_fragmenter.py -i TestFile_SingleCol-wIndx.csv -o TestSCFL_idX_ -rw 11500 -idx True -odx True

!python file_fuser.py -i TestSCFL_idX_ -o TstSCFL_idX_Out.csv -idx True -odx True

!python file_fuser.py --inputStart TestSCFL_idX_ --output TestSCFL_idX_Out.csv --input_index True


I am working on the file "Python_Misc/TMWP_DFBuilder_OO_PY/testing_and_documentation/TMWP_DFBuilder_AbstractClass_Module_Testing.ipynb".
The first task is:
Add 5 Rows - Original Data Source
Using buildOutDF() to add to outDF inside the object.  This test will repeat the first 5 rows on the end of the DF
Can you write Python code for it?

tstDFbldr.buildOutDF(tst_lat_lon_df[0:5])  # add copy of 5 rows to the end (like adding more data later)
tstDFbldr.outDF.tail(10)                   # function added these new rows and fixed the index
                                           # this test is why the reset_index code was added

tstDFbldr.outDF.head(10)  ## quick check of the head of the DF

# tstDFbldr.outDF  ## uncomment to view whole DF


I am working on the file "Python_Misc/TMWP_DFBuilder_OO_PY/testing_and_documentation/TMWP_DFBuilder_AbstractClass_Module_Testing.ipynb".
The first task is:
Attempt To Replicate Earlier Problem
This problem was originally created in another Notebook without all the tests before it.  Strangely, removing some of the tests that preceded the one that was expected to work caused it to fail in an initial run of this notebook.  Code has since changed and these tests now show that the content works as expected (problem solved).
Can you write Python code for it?

## to illustrate:  we try creating a fresh object to see if we can show that problem in this NB
## build 3 here ...

tstDFbld3 = TstModification_DFBuilder(time_delay=1)
tstDFbld3.buildOutDF(tst_lat_lon_df_sample)          ## stop this test in middle for next set of tests

print(tstDFbld3.delay)      ## show delay used:  1 second
tstDFbld3.outDF.describe()  ## describe resulting DF ... it has only a fraction of the expected rows
                             ## because we stopped the code early during testing ...

tstDFbld3.outDF.tail()

tmpDF2 = tst_lat_lon_df_sample[-5:].copy(deep=True)
tmpDF2.reset_index(drop=True, inplace=True)
tmpDF2

tstDFbld3.buildOutDF(tmpDF2)   # starting with fresh object and fresh deep copy of the sample
tstDFbld3.outDF.tail()         # the problem recurs
                               # first attempt fails

tstDFbld3.buildOutDF(tmpDF2)
tstDFbld3.outDF.tail()          # second and subsequent attempts succeed

tstDFbld3.buildOutDF(tmpDF2)
tstDFbld3.outDF.tail()


I am working on the file "Python_Misc/TMWP_DFBuilder_OO_PY/testing_and_documentation/TMWP_DFBuilder_AbstractClass_Module_Testing.ipynb".
The first task is:
Code Testing
The abstract class which follows is intended to be the "work horse" of this code.  Intent is that it gets the developer to the point where all they need to think about is what their final subclass will do to enrich the data.  The parent class sets up a loop that can extract from a larger input DF, a small number of rows to be operated on in a temp DF and then be added to an outputDF.  In the event of something interrupting the process (a common event when dealing with web APIs), modified rows created before the incident are waiting in output DF and can be extracted. Then code can be restarted or continued to allow building up the rest of the Dataframe without losing previous work or having to go all the way back to the beginnin.
This test notebook sets up a subclass that will never be used in the real world.  There are more efficient ways to modify a DF with the example selected for this test.  The test's intent is simply to show that most of the core logic works before we test a subclass that is slower and more involved because it actually makes calls to a web API.
Can you write Python code for it?

who

from abc import ABCMeta, abstractmethod
import pandas as pd

class DFBuilder(object, metaclass=ABCMeta):       # sets up abstract class
    def __init__(self,endRw,time_delay):          # abstract classes can be subclassed
        self.endRow=endRw                         # but cannot be instantiated
        self.delay=time_delay
        self.tmpDF=pd.DataFrame()   # temp DF will be endRow rows in length
        self.outDF=pd.DataFrame()   # final DF build in sets of endRow rows so all is not lost in a failure
        self.lastIndex = None
        # self.start=0

    def __str__(self):
        return ("Global Settings for this object: \n" +
               "endRow: " + str(self.endRow) + "\n" +
               "delay:  " + str(self.delay) + "\n" +
               "Length of outDF: " + str(len(self.outDF)) + "\n" +
               "nextIndex: " + str(self.lastIndex))       # if continuing with last added table - index of next rec.

    @abstractmethod                               # abstract method definition in Python
    def _modifyTempDF_(): pass                    # This method will operate on TempDF inside the loop

    def buildOutDF(self, inputDF):
        '''Scans inputDF using self.endRow rows (default of 5) at a time to do it.  It then calls in logic
from _modifyTempDF()_ to make changes to each subset of rows and appends tiny tempDF onto an outDF.  When the
subclass is using a web API, self.time_delay tells it how much time to delay each iteration of the loop.  All
parameters are set during initialization of the object.  Should this function fail in the middle, outDF will
have all work up to the failure.  This can be saved out to a DF or csv.  The function can be run again on
a subset of the data (the records not encountered yet before the failure).'''

        lenDF = len(inputDF)
        print("Processing inputDF of length: ", lenDF)
        endIndx = 0

        i = 0
        while i < lenDF:
            # print("i: ", i)
            endIndx = i + self.endRow
        ...


I am working on the file "Python_Misc/TMWP_DFBuilder_OO_PY/testing_and_documentation/TMWP_DFBuilder_AbstractClass_Module_Testing.ipynb".
The first task is:
Documentation Tests
Can you write Python code for it?

# create new object to test the docstrings
testObj1 = TstModification_DFBuilder()

help(testObj1)

print(testObj1.__doc__)  # note: formatting is messed up if you do not use print() on the doc string

print(testObj1.buildOutDF.__doc__) # buildOutDF


I am working on the file "Python_Misc/TMWP_DFBuilder_OO_PY/testing_and_documentation/TMWP_DFBuilder_AbstractClass_Module_Testing.ipynb".
The first task is:
Simulated Interrupt
In the final object, it will be server timeouts from the web that may result in coding failing to complete.  The closest we can come to simulating this without introducing web API content is to set a time delay and interrupt the code run in the middle (manually) from within Jupyter.  That test as well as some other logic checks follows here.
Can you write Python code for it?

tstDFbldr2.buildOutDF(tst_lat_lon_df_sample)          ## stop this test in middle for next set of tests

print(tstDFbldr2.delay  )    ## show delay used:  1 second
tstDFbldr2.outDF.describe()  ## describe resulting DF ... it has only a fraction of the expected rows
                             ## because we stopped the code early during testing ...

tstDFbldr2.outDF.tail()  ## index values shown here were cleaned up by changing creation of sample DF
                         ## see comments in data preparation at start of NB

## work-around:  this code could be used to reset index with same options as buildOutDF()
# tstDFbldr2.reindex_OutDF()
# tstDFbldr2.outDF.tail()

tstDFbldr2.buildOutDF(tst_lat_lon_df_sample[-5:])   ## first attempt to add last 5 rows again from sample data

tstDFbldr2.outDF.describe()  # count was unchanged from previous in earlier iteration of the code

tstDFbldr2.outDF.tail(10)  # now it seems to work right

tstDFbldr2.buildOutDF(tst_lat_lon_df_sample[-5:])   ## Try it a second time

tstDFbldr2.outDF.describe()                         ## note: count is 5 more than before

tstDFbldr2.outDF.tail(10)    ## comparison of tail helps confirm new records were added
                             ## but clean index reset occurs this time (as it is supposed to)

## Another investigation
'''Idea: If in doubt as to whether the dataframe being passed in for the second run is mutating correctly or not,
try making a deep copy in steps and resetting the index on the copy as shown here. '''

## problems this was investigating now appear to be fixed

tmpDF1 = tst_lat_lon_df_sample[-5:].copy(deep=True)
tmpDF1.reset_index(drop=True, inplace=True)

tmpDF1

tstDFbldr2.buildOutDF(tmpDF1)  ## initial test seems promising

tstDFbldr2.outDF.tail(10)     ##  as shown here, multiple tests seem to add the new rows every time

tstDFbldr2.buildOutDF(tmpDF1)
tstDFbldr2.outDF.tail(10)

print(tstDFbldr2)  ## note: we added tmpDF1 which ended on index 4.  This is why "nextIndex" now reads 5
                   ## nextIndex...


I am working on the file "Python_Misc/TMWP_DFBuilder_OO_PY/testing_and_documentation/TMWP_DFBuilder_AbstractClass_Module_Testing.ipynb".
The first task is:
Test Loop Math Issue
This section was created to debug an issue with the loop.  Successful runs of these these tests show this problem was corrected.
Can you write Python code for it?

# del loopDFTst

# del tstDFbldr

who

loopDFTst = TstModification_DFBuilder()

print(loopDFTst)

loopDFTst.buildOutDF(tst_lat_lon_df)

print("Length of outDF: ", len(loopDFTst.outDF))
loopDFTst.outDF.tail()

tst_lat_lon_df.iloc[[0,1,2,1157,1158,1159]]  # looking at start and end of source data

loopDFTst.outDF.iloc[[0,1,2,1157,1158,1159]] # comparing to output rows

# reset the looptest object and check it again with two slices of original DF
del loopDFTst

loopDFTst = TstModification_DFBuilder()
print(loopDFTst)

loopDFTst.buildOutDF(tst_lat_lon_df[0:98])  # use number that does not divide evenly by 5 (endRow=5)
print(loopDFTst)

print("Length of outDF: ", len(loopDFTst.outDF))
loopDFTst.outDF.tail()

loopDFTst.buildOutDF(tst_lat_lon_df[98:100])  # test: missed few records scenario (in this case, adding just 2)

print("Length of outDF: ", len(loopDFTst.outDF))
loopDFTst.outDF.tail()

loopDFTst.buildOutDF(tst_lat_lon_df[100:105])  # test: add amount equal to internal self.endRow variable

print("Length of outDF: ", len(loopDFTst.outDF))
loopDFTst.outDF.tail()

loopDFTst.buildOutDF(tst_lat_lon_df[105:205])  # test: add 100 rows (which is divisble by 5)

print("Length of outDF: ", len(loopDFTst.outDF))
loopDFTst.outDF.tail()

loopDFTst.buildOutDF(tst_lat_lon_df[205:306])  # another add records test

print("Length of outDF: ", len(loopDFTst.outDF))
loopDFTst.outDF.tail()

loopDFTst.buildOutDF(tst_lat_lon_df[306:])  # get the rest added

print("Length of outDF: ", len(loopDFTst.outDF))
loopDFTst.outDF.tail()

## sanity checking:
loopDFTst.outDF.iloc[[0,1,2,1157,1158,1159]]

tst_lat_lon_df.iloc[[0,1,2,1157,1158,1159]]


I am working on the file "Python_Misc/TMWP_DFBuilder_OO_PY/testing_and_documentation/TMWP_DFBuilder_AbstractClass_Module_Testing.ipynb".
The first task is:
Test Sub Class
Before creating, testing, and debugging a subclass that uses a Google maps API to enrich the data, it is desirable to start simpler.  This test object makes use of all of the same logic except for the API calls out to the web to get new data.  For this test, the "delay" will be set to 0 since it is not needed.  This code just shows that we can loop through an original DF, copy rows 5 at a time to a temporary DF, add columns to them using logic that looks at existing rows, and output to our output DF.
Stopping the code in the middle may allow us to test what happens if the code halts, showing what is stored in the dataframe in the object when this happens.
Can you write Python code for it?

class TstModification_DFBuilder(DFBuilder):
    '''Test of ability to scan a dataframe x rows at a time and add data columns to it.
There are more efficient ways to round cols in a DF; this object is a test of base logic from the abstract class
ahead of creating a more complex subclass that interacts with the web during the loop.  It builds a copy of the
DF a small number of rows at a time and creates some new fields as it does so.  Input DF must have "lat" and
"lon" cols. lat=Latitude / lon = Longitude. Defaults set delay to 0 seconds and rows processed at a time to
5 for this test.'''

    def __init__(self, endRw=5,time_delay=0):
        super().__init__(endRw,time_delay)

    def roundValue(self, value, dec_places=4, rtn_null=False):
        '''Takes arguments: value, dec_places. Rounds value to dec_places specified (if not specified, default=4.)
rtn_null defaults to False.  If True, error handling should result in an empty string being returned.
If false __ErrType__ should be returned to help with debugging code and data by distinguishing why there is
no rounded answer returned.  Testing shows that while using round() throws errors if input is not a number,
applying it to a dataframe does not.  Try-except code left in for future research but does not seem to ever
get triggered as of this writing.'''

        try:
            rtnVal = round(value, dec_places)
        except TypeError as terr:
            print(type(terr))
            print(terr)
            rtnVal = "__NAN__"
        except Exception as eerr:
            print(type(eerr))
            print(eerr)
            rtnVal = "__ERR__"
        finally:
            if rtn_null==True:
                if isinstance(rtnVal, str):
                    return ""
                elif rtnVal is None:
                    return ""
                else:
                    return rtnVal
            else:
                return rtnVal

    def _modifyTempDF_(self, dec_places=4, rtn_null=False):
       ...


I am working on the file "Python_Misc/TMWP_DFBuilder_OO_PY/testing_and_documentation/TMWP_DFBuilder_GMapsSubClass_Module_Testing.ipynb".
The first task is:
Code Testing
The abstract class which follows is intended to be the "work horse" of this code.  Intent is that it gets the developer to the point where all they need to think about is what their final subclass will do to enrich the data.  The parent class sets up a loop that can extract from a larger input DF, a small number of rows to be operated on in a temp DF and then be added to an outputDF.  In the event of something interrupting the process (a common event when dealing with web APIs), modified rows created before the incident are waiting in output DF and can be extracted. Then code can be restarted or continued to allow building up the rest of the Dataframe without losing previous work or having to go all the way back to the beginnin.
Can you write Python code for it?

# note: gmtime() produced results in Grenwich Mean Time
#       localtime() seems to get the local time from the computer (in my case EST)

from time import localtime, strftime

def getNow():
    return strftime("%Y-%m-%d %H:%M:%S", localtime())

getNow()

from abc import ABCMeta, abstractmethod
import pandas as pd

class DFBuilder(object, metaclass=ABCMeta):       # sets up abstract class
    '''DataFrame Builder abstract class.  Sets up logic to be inherited by objects that need to loop over a DataFrame
and cache the results.  Original use case involves making API calls to the web which can get interrupted by
errors and server timeouts.  This object stores all the logic to build up and save a DataFrame a small number of
records at a time.  Then a subclass can define an abstract method in the base class as to what we want to do to
the input data.  Original use case added in content extracted form the web to a new column.  But subclasses can
be built to do more. Initialization argumens: endRw, time_delay. endRw = number of records to cache at a time
when building outDF.  time_delay is number of seconds delay between each cycle of the loop that builds outDF.'''
    def __init__(self,endRw,time_delay):          # abstract classes can be subclassed
        self.endRow=endRw                         # but cannot be instantiated
        self.delay=time_delay
        self.tmpDF=pd.DataFrame()   # temp DF will be endRow rows in length
        self.outDF=pd.DataFrame()   # final DF build in sets of endRow rows so all is not lost in a failure
        self.lastIndex = None
        self.statusMsgGrouping = 100

    def __str__(self):
        return ("Global Settings for this object: \n" +
                "endRow: " + str(self.endRow) + "\n" +
                "delay:  " + str(self.delay) + "\n" +
                "statusMsgGrouping: " + str(self.statusMsgGrouping) + "\n"
                "Length of outDF: " + str(len(self.outDF)) + "\n" +
                "nextIndex: " + str(self.lastIndex))
          ...


I am working on the file "Python_Misc/TMWP_DFBuilder_OO_PY/testing_and_documentation/TMWP_DFBuilder_GMapsSubClass_Module_Testing.ipynb".
The first task is:
Documentation Tests
Can you write Python code for it?

# create new object to test the docstrings and some more quick coding tweaks
testObjDocs = GMapsLoc_DFBuilder()
print(testObjDocs)

help(testObjDocs)

print(testObjDocs.__doc__)  # note: formatting is messed up if you do not use print() on the doc string

print(testObjDocs.buildOutDF.__doc__) # buildOutDF

help(DFBuilder)


I am working on the file "Python_Misc/TMWP_DFBuilder_OO_PY/testing_and_documentation/TMWP_DFBuilder_GMapsSubClass_Module_Testing.ipynb".
The first task is:
Test Data
Input Data Set up Here
Can you write Python code for it?

## Test code on a reasonably small DF
tst_lat_lon_df = pd.read_csv("testset_unique_lat_and_lon_vals.csv", index_col=0)

tst_lat_lon_df.describe()

tst_lat_lon_df.tail()


I am working on the file "Python_Misc/TMWP_DFBuilder_OO_PY/testing_and_documentation/TMWP_DFBuilder_GMapsSubClass_Module_Testing.ipynb".
The first task is:
Test of Other Internal Functions
These functions can be used in testing or to just get back a single value.  These examples might prove useful.
Can you write Python code for it?

gMapAddrDat.testConnection()  ## uses default test record to just ensure connection is working

gMapAddrDat.getGeoAddr(40.699100, -73.703697, test=True)  ## function called by buildOutDF()
                                                          ## use test mode to obtain more information

tstLoc1 = gMapAddrDat.getGeoAddr(40.699100, -73.703697, test=True)  ## use .raw on output during testing
print(type(tstLoc1))                                                ## to view JSON structure of Location obj
tstLoc1.raw

tstLoc1 = gMapAddrDat.getGeoAddr(40.699100, -73.703697)  ## default: test=False
print(type(tstLoc1))                                     ## when called internally to build the address field for
tstLoc1                                                  ## for outDF, it just returns an address string


I am working on the file "Python_Misc/TMWP_DFBuilder_OO_PY/testing_and_documentation/TMWP_DFBuilder_GMapsSubClass_Module_Testing.ipynb".
The first task is:
Testing of The Subclass
A different subclass was created in another notebook to test most if not all of the non-web related logic of the Abstract class.  This means testing in this notebook can focus on the code that produces final results and that interacts with the Google Maps API.
This section shows how the code can build up outDF() adding addresses obtained from the Google Maps API to the latitude and longitude provided in the input data. Tests show how errors are handled, both as text of the form "<errorTxt>" in the address field, or as empty strings if you set rtn_null to True.  Tests also show how data can be added to outDF by re-running the build function.  This allows adding of additional data to outDF, or of adding in data that was missed due to server timeout errors or other interruptions to the web process.
Test Main Logic with Error Handling Exposed
These tests were designed to show the error handling in action.  For the sake of brevity, earlier testing was deleted to just show later tests in which errors are expected (due to exceeding daily license allotment from Google).
Can you write Python code for it?

## build main object using the defaults
testObj = GMapsLoc_DFBuilder()

print(testObj)

testObj.buildOutDF(tst_lat_lon_df)  ## some tests not shown performed ahead of this run
                                    ## errors should be result of exceeding daily record allotment
                                    ## for free Google Maps API license
                                    ## this code tests basic functioning and default error handling

testObj.outDF.head()

testObj.outDF.tail()  ## this check shows default behavior
                      ## errors recorded in address field so user can find out why a particular location
                      ## failed to return results - in this case "too many requests" (for license allotment)
                      ## errors begin and end with "_" which an address will not.
                      ## a query or filter of the data for addresses starting with "_" can inform the user
                      ## which records need to be run again

## change error handling and a few other default parameters
testObj.rtn_null = True           ## change error handling: bad records will not simply get blank Address values
testObj.set_statusMsgGrouping(10) ## get status message about every 10 records (this will be a small test)
testObj.set_timeDelay(0)          ## remove time delay (this increases risk of errors)
                                  ## note: each set_ function outputs current state of variables
                                  ##       each output begins with "Global settings ..."
                                  ##       last one is what these settings look like going into the next test

testObj.buildOutDF(tst_lat_lon_df[-25:])  ## redo end of DF .. should be entirely blank since we're out of licenses
                                          ## rtn_null = False told code to return empty cell instead of error text
                                          ## in production, it may be easier to just search for the nulls to get
                                      ...


I am working on the file "Python_NRT.ipynb".
The first task is:
Blocks specified by indentation (not braces nor brackets)
First statement of a block ends in colon (:)
Can you write Python code for it?

def myfunction(x):
    pass
    if x > 10:
        pass
        pass
        return "bigger"
    else:
        pass
        pass
        return "smaller"

print "This number is: " + myfunction(5)


I am working on the file "Python_NRT.ipynb".
The first task is:
Built-in method
Can you write Python code for it?

df.mean() # operates by columns (axis=0)


I am working on the file "Python_NRT.ipynb".
The first task is:
Combined selection
Can you write Python code for it?

print df.loc['row1',['col1', 'col3']] # labels
print df.loc[['row1','row3'],'col1' : 'col3']

df.iloc[0:2,[0,2]]  # row position 0,1

print df.ix[0,['col2','col3']] # position & label
print df.ix['row1':'row3', :]


I am working on the file "Python_NRT.ipynb".
The first task is:
Conditionals (if, elif, else)
Can you write Python code for it?

united_kingdom = ["England", "Scotland", "Wales", "N Ireland"]
one = "France"

if one in united_kingdom:
    print "UK"
elif one == "France":
    print "Not UK. Bon jour!"
else:
    print "Not UK"


I am working on the file "Python_NRT.ipynb".
The first task is:
Control flow

Loops (while, for)
Can you write Python code for it?

for numbers in range(1,5):
    print numbers


I am working on the file "Python_NRT.ipynb".
The first task is:
Functions & Methods
Can you write Python code for it?

"""functions are pieces of code that you can
call/execute, they are defined with the def keyword"""

def hola_mundo():
    print "Hola Mundo!"

""" methods are attributes of an object that
you can call over the object with and "." """

s = "How are you"
print s.split(" ")


I am working on the file "Python_NRT.ipynb".
The first task is:
Help!
"house".len()?
len(house)?
Can you write Python code for it?


I am working on the file "Python_NRT.ipynb".
The first task is:
PANDAS LIBRARY
Open source library  providing high-performance structures and data analysis tools for the Python programming language.
Import
Can you write Python code for it?


I am working on the file "Python_NRT.ipynb".
The first task is:
Pandas Axis
| axis   |     axis       |  along            |  each          |
|--------|:--------------:|------------------:|---------------:|
| axis=1 | axis="columns" | along the columns |for each row    |
| axis=0 | axis="index"   | along the rows    |for each column |
Can you write Python code for it?

df2 = pd.DataFrame(
    data =[[1, 2], [4, 5], [7, 8]],
    columns=["A", "B"])
df2

df2.mean(axis=1) # mean for each row

df2 = pd.DataFrame(
    data =[[1, 2], [4, 5], [7, 8]],
    columns=["A", "B"])
df2

df2.drop("A", axis=1) # drop columns for each row


I am working on the file "Python_NRT.ipynb".
The first task is:
Pandas Dataframe
Can you write Python code for it?


I am working on the file "Python_NRT.ipynb".
The first task is:
Select rows
Can you write Python code for it?

df.loc['row1'] # by row using label

df.iloc[0]  # by row using position

df.ix['row1']   # by row, using label
print df.ix[0]  # by row, using position


I am working on the file "Python_NRT.ipynb".
The first task is:
Selection
Can you write Python code for it?

ss = pd.Series([1,2,3],
              index = ['a','b','c'])
print ss[0]       # as a list
print ss.iloc[0]  # by position, integer
print ss.loc['a'] # by label of the index
print ss.ix['a']  # label (priority)
print ss.ix[0]    # position if no label

"""Be careful with the slice operator
using positions or labels"""

print ss.iloc[0:2] # positions 0,1
print ss.loc['a':'c'] # labels 'a','b','c'


I am working on the file "Python_NRT.ipynb".
The first task is:
Selection
Select columns
Can you write Python code for it?

df['col1']  # one col => Series

df[['col1']] # list of cols => DataFrame


I am working on the file "Python_NRT.ipynb".
The first task is:
Should I use always .ix()?
.ix() selector gotcha!
Can you write Python code for it?


I am working on the file "Python_NRT.ipynb".
The first task is:
Structures
Pandas Series
Can you write Python code for it?


I am working on the file "Python_NRT.ipynb".
The first task is:
Tuples
Similar to lists, sequence of elements that conforms an immutable object.
Can you write Python code for it?

tup = ('physics', 'chemistry', 1997, 2000)
print tup[0]

print tup[1:3]


I am working on the file "Python_OO.ipynb".
The first task is:
Documentation
If you want to share your code with collaborators or with your future self, you should include documentation. We've neglected that above, so let's add in some docstrings!
Can you write Python code for it?

class SpaceRock(object):

    """Container for Asteroids"""

    def __init__(self, name = None, ab_mag = None, albedo = None,
                 semi_major= None, ecc = None):

        """
        Parameters
        ----------
        name : string
            Name of the target
        ab_mag : array-like
            Absolute Magnitude of each Asteroid
        albedo : array-like
            Albedo of each Asteroid
        semi_major : array-like
            Semi Major Axis of each Asteroid in AU
        ecc : array-like
            Eccentricity of each Asteroid
        """

        self.name = name
        self.ab_mag = ab_mag
        self.albedo = albedo
        self.semi_major = semi_major
        self.ecc = ecc

    def diameter(self):
        """
        Determine the diameter (in km) of the Asteroids
        """
        result = (1329.0 / np.sqrt(self.albedo)) * (10 ** (-0.2 * self.ab_mag))
        return result * u.km

    def two_diameter(self):
        """
        Determine twice the diameter (in km) of the Asteroids
        """
        result = 2.0 * self.diameter()
        return result

    def find_perihelion(self):
        """
        Determine the perihelion distance of the Asteroids in AU
        """
        result = self.semi_major * ( 1.0 - self.ecc )
        return result * u.AU

rocks = (SpaceRock(name=my_name, ecc = my_ecc, semi_major=my_semi_major,
                   ab_mag=my_ab_mag, albedo=my_albedo))


I am working on the file "Python_OO.ipynb".
The first task is:
Methods
So far this just looks like another way to store your data. It becomes more powerful when you write methods for your object.
Methods can be thought of as functions associated with an object.
You can now access the attributes of the object within methods by calling self.<attribute name>.
Let's make a simple method for the SpaceRock object, which determines the size of the asteroid.
Can you write Python code for it?

class SpaceRock(object):

    def __init__(self, name=None, ab_mag=None, albedo=None):
        self.name = name
        self.ab_mag = ab_mag
        self.albedo = albedo

    def diameter(self):
        result = (1329.0 / np.sqrt(self.albedo)) * (10 ** (-0.2 * self.ab_mag))
        return result * u.km


I am working on the file "Python_OO.ipynb".
The first task is:
Now you can see the documentation on the module within the Notebooks by typing
rocks?
...you can see the documentation for each method by typing
rocks.diameter?
Can you write Python code for it?


I am working on the file "Python_OO.ipynb".
The first task is:
To use a method you need to add () to the end of the method
Can you write Python code for it?

new_asteroid = SpaceRock(name=my_name, ab_mag=my_ab_mag, albedo=my_albedo)

new_asteroid.diameter()


I am working on the file "Python_Plotting.ipynb".
The first task is:
3D plots
Can you write Python code for it?

from mpl_toolkits.mplot3d import Axes3D

fig = plt.figure()
ax = fig.add_subplot(111,projection='3d')

fig.set_size_inches(9,9)

fig.tight_layout()

xx = np.cos(3*theta)
yy = np.sin(2*theta)

ax.plot(theta, xx, yy, c = "Maroon")
ax.scatter(theta, xx, yy, c = "Navy", s = 15);

ax.view_init(azim = -140, elev = 15)


I am working on the file "Python_Plotting.ipynb".
The first task is:
An Astronomical Example - Color Magnitude Diagrams
Can you write Python code for it?

T = QTable.read('M15_Bright.csv', format='ascii.csv')
T[0:3]

fig, ax = plt.subplots(1,1)                 # 1 row, 2 colums
fig.set_size_inches(15,10)

fig.tight_layout()

BV = T['Bmag'] - T['Vmag']
V = T['Vmag']

ax.set_xlim(-0.25,1.5)
ax.set_ylim(12,19)

ax.set_aspect(1/6)         # Make 1 unit in X = 6 units in Y
ax.invert_yaxis()          # Magnitudes increase to smaller values

ax.set_xlabel("B-V")
ax.set_ylabel("V")

ax.plot(BV,V,color="b",marker="o",linestyle="None",markersize=5);

# overplotting

mask_color = np.where((V < 16.25) & (BV < 0.55))

ax.plot(BV[mask_color], V[mask_color],color="r",marker="o",linestyle="None",markersize=4, alpha=0.5);


I am working on the file "Python_Plotting.ipynb".
The first task is:
Plotting from multiple external data files
Can you write Python code for it?

data_list = glob.glob('./MyData/12_data*.csv')

data_list

fig,ax = plt.subplots(1,1)                    # One window
fig.set_size_inches(11,8.5)                   # (width,height) - letter paper landscape

fig.tight_layout()                          # Make better use of space on plot

ax.set_xlim(0.0,80.0)
ax.set_ylim(15.0,100.0)

ax.set_xlabel("This is X")
ax.set_ylabel("This is Y")

for file in data_list:

    data = QTable.read(file, format='ascii.csv')
    ax.plot(data['x'], data['y'],marker="o",linestyle="None",markersize=7,label=file)

ax.legend(loc=0,shadow=True);


I am working on the file "Python_Plotting.ipynb".
The first task is:
Polar Plots
Can you write Python code for it?

theta = np.linspace(0,2*np.pi,1000)

fig = plt.figure()
ax = fig.add_subplot(111,projection='polar')

fig.set_size_inches(6,6)                   # (width,height) - letter paper landscape

fig.tight_layout()                          # Make better use of space on plot

ax.plot(theta,theta/5.0,label="spiral")
ax.plot(theta,np.cos(4*theta),label="flower")

ax.legend(loc=2, frameon=False);


I am working on the file "Python_Plotting.ipynb".
The first task is:
Simple Plotting
Can you write Python code for it?

t = np.linspace(0,2,100)               # 100 points linearly spaced between 0.0 and 2.0
s = np.cos(2*np.pi*t) * np.exp(-t)     # s if a function of t

plt.plot(t,s)


I am working on the file "Python_Plotting.ipynb".
The first task is:
Simple plotting - with style
The default style of matplotlib is a bit lacking in style. Some would term it ugly. The new version of matplotlib has added some new styles that you can use in place of the default. Changing the style will effect all of the rest of the plots on the notebook.
Examples of the various styles can be found here
Can you write Python code for it?

plt.style.available

plt.style.use('ggplot')

plt.plot(t,s)

plt.xlabel('time (s)')
plt.ylabel('voltage (mV)')
plt.title('This is a title')

plt.ylim(-1.5,1.5)

plt.plot(t, s, color='b', marker='None', linestyle='--');   # adding the ';' at then suppresses the Out[] line

mask1 = np.where((s>-0.4) & (s<0))

plt.plot(t, s, color='b', marker='None', linestyle='--')

plt.plot(t[mask1],s[mask1],color="g",marker="o",linestyle="None",markersize=8);


I am working on the file "Python_Plotting.ipynb".
The first task is:
You have better control of the plot with the object oriented interface.
While most plt functions translate directly to ax methods (such as plt.plot() → ax.plot(), plt.legend() → ax.legend(), etc.), this is not the case for all commands.
In particular, functions to set limits, labels, and titles are slightly modified.
For transitioning between matlab-style functions and object-oriented methods, make the following changes:

plt.xlabel()  → ax.set_xlabel()
plt.ylabel() → ax.set_ylabel()
plt.xlim()  → ax.set_xlim()
plt.ylim() → ax.set_ylim()
plt.title() → ax.set_title()
Can you write Python code for it?

fig,ax = plt.subplots(1,1)                    # One window
fig.set_size_inches(11,8.5)                   # (width,height) - letter paper landscape

fig.tight_layout()                          # Make better use of space on plot

ax.set_xlim(0.0,1.5)

ax.spines['bottom'].set_position('zero')    # Move the bottom axis line to x = 0

ax.set_xlabel("This is X")
ax.set_ylabel("This is Y")

ax.plot(t, s, color='b', marker='None', linestyle='--')

ax.text(0.8, 0.6, 'Bad Wolf', color='green', fontsize=36)            # You can place text on the plot

ax.vlines(0.4, -0.4, 0.8, color='m', linewidth=3)                    # vlines(x, ymin, ymax)
ax.hlines(0.8,  0.2, 0.6, color='y', linewidth=5)                    # hlines(y, xmin, xmax)

fig.savefig('fig1.png', bbox_inches='tight')

import glob

glob.glob('*.png')


I am working on the file "Python_ReadingData.ipynb".
The first task is:
Adding a column to the Table
Can you write Python code for it?

perihelion = planet_table['a'] * (1.0 - planet_table['ecc'])

perihelion

planet_table['Peri'] = perihelion

planet_table


perihelion = planet_table2['a'] * (1.0 - planet_table2['ecc'])

perihelion

planet_table2['Peri'] = perihelion

planet_table2


I am working on the file "Python_ReadingData.ipynb".
The first task is:
Deal with the missing data with fillna()
Can you write Python code for it?

messy_table['Name'].fillna("unknown", inplace=True)
messy_table['Size'].fillna(999.0, inplace=True)

messy_table


I am working on the file "Python_ReadingData.ipynb".
The first task is:
Masking
Can you write Python code for it?

planet_table.sort(['a'])    # re-sort our table

mask1 = np.where(planet_table['a'] > 5)

mask1

planet_table[mask1]

mask2 = ((planet_table['a'] > 5) &
         (planet_table['ecc'] < 0.05))

planet_table[mask2]


mask3 = planet_table['a'] > 5

mask3

planet_table2[mask3]

mask4 = ((planet_table2['a'] > 5) &
         (planet_table2['ecc'] < 0.05))

planet_table2[mask4]


I am working on the file "Python_ReadingData.ipynb".
The first task is:
Messy Data

Pandas is a good choice when working with messy data files.
In the "real world" all data is messy.
For example, here is the contents of the file Mess.csv:

This is not going to end well ...
Can you write Python code for it?


I am working on the file "Python_ReadingData.ipynb".
The first task is:
NaN = Not_A_Number, python's null value
Column names are messed up
Option 1 - Turn off the header
Can you write Python code for it?

messy_table = pd.read_csv('Mess.csv', skiprows = 6, header= None)

messy_table


I am working on the file "Python_ReadingData.ipynb".
The first task is:
Option 2 - Add the column names
Can you write Python code for it?

cols = ["Name", "Size"]

messy_table = pd.read_csv('Mess.csv', skiprows = 6, names = cols)

messy_table


I am working on the file "Python_ReadingData.ipynb".
The first task is:
Renaming columns
Can you write Python code for it?

planet_table2.rename(columns={'Unnamed: 2': 'ecc'}, inplace=True)

planet_table2

planet_table2['Name']

planet_table['Name'][0]


I am working on the file "Python_ReadingData.ipynb".
The first task is:
Saving a table
Can you write Python code for it?

planet_table.write('NewPlanets.csv', format='ascii.csv')

os.listdir()


I am working on the file "Python_ReadingData.ipynb".
The first task is:
Skip the header
Can you write Python code for it?

messy_table = pd.read_csv('Mess.csv', skiprows = 6)

messy_table


I am working on the file "Python_ReadingData.ipynb".
The first task is:
Sorting
Can you write Python code for it?

planet_table2.sort_values(['ecc'])

planet_table2

planet_table2.sort_values(['ecc'], ascending=False)


I am working on the file "Python_ReadingData.ipynb".
The first task is:
The AstroPy package - QTable
Can you write Python code for it?

import numpy as np

from astropy.table import QTable

os.listdir()

planet_table = QTable.read('Planets.csv', format='ascii.csv')

planet_table

print(planet_table)


I am working on the file "Python_ReadingData.ipynb".
The first task is:
The Pandas package - DataFrame
Can you write Python code for it?

import pandas as pd

planet_table2 = pd.read_csv('Planets.csv')

planet_table2

print(planet_table2)


I am working on the file "Python_Resources/SimulationOptimization.ipynb".
The first task is:
As you can see, the amount of time the unoptimized program requires to run only 3000 generations with a population of 1000 is 1 minute and 29 seconds.
Can you write Python code for it?

%%timeit
SSP(1000, 3*np.power(10,3), SternJudging, optimized=True)


I am working on the file "Python_Resources/SimulationOptimization.ipynb".
The first task is:
Investigate Cython speedup
Can you write Python code for it?

runs = 1
generations = 3*np.power(10, 2)
population_size = 10
socialnorm = np.array([[0, 0], [0, 1]])
mutation_rate = np.power(10.0*population_size, -1.0)

execution_error = 0.08
reputation_assignment_error = 0.01
private_assessment_error = 0.01
reputation_update_probability = 1.0
cost = 1
benefit = 5

%%timeit
sim_cyth.simulation_instance.run_instance(runs, generations, population_size, mutation_rate,
                              execution_error, reputation_assignment_error,
                              private_assessment_error, reputation_update_probability,
                              socialnorm, cost, benefit)

%%timeit
SimulationInstance.simulate(runs, generations, population_size, mutation_rate,
                            execution_error, reputation_assignment_error,
                            private_assessment_error, reputation_update_probability,
                            socialnorm, cost, benefit)

cython_time_objects = []
pure_python_time_objects = []
python_vectorized_time_objects = []
p_sizes = [5, 10, 20, 30, 40, 50, 60, 70, 90, 110, 130, 160, 180]
for p_size in p_sizes:
    population_size = p_size
    variables = InstanceVariables(runs, generations, population_size, mutation_rate,
                                  execution_error, reputation_assignment_error,
                                  private_assessment_error, reputation_update_probability,
                                  socialnorm, cost, benefit)
    cython_result = %timeit -r 1 -o sim_cyth.simulation_instance.run_instance(runs, generations, population_size, mutation_rate, \
                            execution_error, reputation_assignment_error, \
                            private_assessment_error, reputation_update_probability, \
                            socialnorm, cost, benefit)
    pure_python_result = %timeit -r 1 -o SimulationInstance.simulate(runs, generations, population_size, mutation_rate, \
                            execution_error, reputation_assignment_error, \
                           ...