I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 3 - Text networks and word embeddings.ipynb".
The first task is:
Finally, you can use the similarity method to return the similarity between two terms. In our trained model, "britain" and "france" are more similar to each other than "mexico" and "canada".
Can you write Python code for it?

bios_model.wv.similarity('republican','democrat')

bios_model.wv.similarity('mexico','canada')

bios_model.wv.similarity('britain','france')


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 3 - Text networks and word embeddings.ipynb".
The first task is:
For each word in top_words, we get its vector from bios_model and add it to the top_word_vectors list and cast this list back to a numpy array.
Can you write Python code for it?

top_word_vectors = []

for word in top_words:
    try:
        vector = bios_model.wv[word]
        top_word_vectors.append(vector)
    except KeyError:
        pass

top_word_vectors = np.array(top_word_vectors)


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 3 - Text networks and word embeddings.ipynb".
The first task is:
Isomap is an extension of MDS.
Can you write Python code for it?

# Step 1: Choose your model class(es)
from sklearn.manifold import Isomap

# Step 2: Instantiate your model class(es)
iso = Isomap(n_neighbors = 5, n_components = 2)

# Step 3: Arrange data into features matrices
# Done!

# Step 4: Fit the data and transform
X_iso = iso.fit_transform(bio_tfidf_dense)

# Plot the data
f,ax = plt.subplots(1,1,figsize=(10,10))
ax.scatter(X_iso[:,0],X_iso[:,1])

ax.set_title('IsoMap')

for i,txt in enumerate(bios.keys()):
    if txt == 'Barack Obama':
        ax.annotate(txt,(X_iso[i,0],X_iso[i,1]),color='blue',fontweight='bold')
    elif txt == 'Donald Trump':
        ax.annotate(txt,(X_iso[i,0],X_iso[i,1]),color='red',fontweight='bold')
    else:
        ax.annotate(txt,(X_iso[i,0],X_iso[i,1]))


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 3 - Text networks and word embeddings.ipynb".
The first task is:
Loading data
Load the data from disk into memory.
Can you write Python code for it?

with open('potus_wiki_bios_cleaned.json','r') as f:
    bios = json.load(f)


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 3 - Text networks and word embeddings.ipynb".
The first task is:
Multi-dimensional scaling is another common technique in the social sciences.
Can you write Python code for it?

# Step 1: Choose your model class(es)
from sklearn.manifold import MDS

# Step 2: Instantiate your model class(es)
mds = MDS(n_components=2,metric=False,n_jobs=-1)

# Step 3: Arrange data into features matrices
# Done!

# Step 4: Fit the data and transform
X_mds = mds.fit_transform(bio_tfidf_dense)

# Plot the data
f,ax = plt.subplots(1,1,figsize=(10,10))
ax.scatter(X_mds[:,0],X_mds[:,1])

ax.set_title('Multi-Dimensional Scaling')

for i,txt in enumerate(bios.keys()):
    if txt == 'Barack Obama':
        ax.annotate(txt,(X_mds[i,0],X_mds[i,1]),color='blue',fontweight='bold')
    elif txt == 'Donald Trump':
        ax.annotate(txt,(X_mds[i,0],X_mds[i,1]),color='red',fontweight='bold')
    else:
        ax.annotate(txt,(X_mds[i,0],X_mds[i,1]))


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 3 - Text networks and word embeddings.ipynb".
The first task is:
One of the most powerful implications of having these vectorized embeddings of word meanings is the ability to do operations similar arithmetic that recover or reveal interesting semantic meanings. The classic example is Man:Woman::King:Queen:

What are some examples of these vector similarities from our trained model?
republican - slavery = democrat - X
-(republican - slavery) + democrat = X
slavery + democrat - republican = X
Can you write Python code for it?


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 3 - Text networks and word embeddings.ipynb".
The first task is:
Other times the results are unexpected/interesting.
Can you write Python code for it?


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 3 - Text networks and word embeddings.ipynb".
The first task is:
Principal component analysis (PCA) is probably one of the most widely-used and efficient methods for dimensionality reduction.
Can you write Python code for it?

# Step 1: Choose a class of models
from sklearn.decomposition import PCA

# Step 2: Instantiate the model
pca = PCA(n_components=2)

# Step 3: Arrange the data into features matrices
# Already done

# Step 4: Fit the model to the data
pca.fit(bio_tfidf_dense)

# Step 5: Evaluate the model
X_pca = pca.transform(bio_tfidf_dense)

# Visualize
f,ax = plt.subplots(1,1,figsize=(10,10))
ax.scatter(X_pca[:,0],X_pca[:,1])

ax.set_title('PCA')

for i,txt in enumerate(bios.keys()):
    if txt == 'Barack Obama':
        ax.annotate(txt,(X_pca[i,0],X_pca[i,1]),color='blue',fontweight='bold')
    elif txt == 'Donald Trump':
        ax.annotate(txt,(X_pca[i,0],X_pca[i,1]),color='red',fontweight='bold')
    else:
        ax.annotate(txt,(X_pca[i,0],X_pca[i,1]))


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 3 - Text networks and word embeddings.ipynb".
The first task is:
Since this is a small and sparse network, we can try to use Matplotlib to visualize it instead. I would only use the nx.draw functionality for small networks like this one.
Can you write Python code for it?

# Plot the nodes as a spring layout

#g_pos = nx.layout.fruchterman_reingold_layout(g, k = 5, iterations=10000)
g_pos = nx.layout.kamada_kawai_layout(g)

# Draw the graph
f,ax = plt.subplots(1,1,figsize=(10,10))
nx.draw(G = g,
        ax = ax,
        pos = g_pos,
        with_labels = True,
        node_size = [dc*(len(g) - 1)*100 for dc in nx.degree_centrality(g).values()],
        font_size = 10,
        font_weight = 'bold',
        width = [d['weight']*10 for i,j,d in g.edges(data=True)],
        node_color = 'tomato',
        edge_color = 'grey'
       )


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 3 - Text networks and word embeddings.ipynb".
The first task is:
Step 2: Compute the TFIDF matrix for the S&P 500 companies.
Can you write Python code for it?

# Compute the word counts
sp500_counts =

# Compute the TF-IDF for the word counts from each biography
sp500_tfidf =

# Convert from sparse to dense array representation
sp500_tfidf_dense =


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 3 - Text networks and word embeddings.ipynb".
The first task is:
Step 3: Compute the cosine similarities.
Can you write Python code for it?

# Compute cosine similarity
company_company_df =

# Filter for edges in the 90th percentile or greater
company_company_filtered_df =

# Reshape and filter data
sp500_edgelist_df =
sp500_edgelist_df =

# Rename and replace data
sp500_edgelist_df.rename(columns={'level_0':'from','level_1':'to',0:'weight'},inplace=True)
sp500_edgelist_df.replace(dict(enumerate(sp500_articles.keys())),inplace=True)

# Inspect
sp500_edgelist_df.head()


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 3 - Text networks and word embeddings.ipynb".
The first task is:
Step 4: Visualize the resulting network.
Word2Vec
We used TF-IDF vectors of documents and cosine similarities between these document vectors as a way of representing the similarity in the networks above. However, TF-IDF score are simply (normalized) word frequencies: they do not capture semantic information. A vector space model like the popular Word2Vec represents each token (word) in a high-dimensional (here we'll use 100-dimensions) space that is trained from some (ideally) large corpus of documents. Ideally, tokens that are used in similar contexts are placed into similar locations in this high-dimensional space. Once we have vectorized words into this space, we're able to efficiently compute do a variety of other operations such as compute similarities between words or do transformations that can find analogies.
I lack the expertise and we lack the time to get into the math behind these methods, but here are some helpful tutorials I've found:
* Word embeddings: exploration, explanation, and exploitation
* Learning Word Embedding
* On word embeddings
* TensorFlow - Vector Representations of Words
We'll use the 44 Presidential biographies as a small and specific corpus. We start by training a bios_model from the list of biographies using hyperparamaters for the number of dimensions (size), the number of surrounding words to use as training (window), and the minimum number of times a word has to occur to be included in the model (min_count).
Can you write Python code for it?

from gensim.models import Word2Vec

bios_model = Word2Vec(bios.values(),size=100,window=10,min_count=8)


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 3 - Text networks and word embeddings.ipynb".
The first task is:
There's a doesnt_match method that predicts which word in a list doesn't match the other word senses in the list. Sometime the results are predictable/trivial.
Can you write Python code for it?


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 3 - Text networks and word embeddings.ipynb".
The first task is:
UMAP is faster and I think better, but you'll need to make sure this is installed on your system since it doesn't come with scikit-learn or Anaconda by default. Words like "nominee" and "campaign" or the names of the months cluster clearly together apart from the rest.
Can you write Python code for it?

# Step 1: Choose your model class(es)
from umap import UMAP

# Step 2: Instantiate your model class(es)
umap_ = UMAP(n_components=2, n_neighbors=5, random_state=42)

# Step 3: Arrange data into features matrices
# Done!

# Step 4: Fit the data and transform
X_w2v_umap = umap_.fit_transform(X_w2v)

# Plot the data
f,ax = plt.subplots(1,1,figsize=(10,10))
ax.scatter(X_w2v_umap[:,0],X_w2v_umap[:,1],s=3)

ax.set_title('UMAP')

for i,txt in enumerate(top_words):
    if i%10 == 0:
        ax.annotate(txt,(X_w2v_umap[i,0],X_w2v_umap[i,1]))

f.savefig('term_umap.pdf')


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 3 - Text networks and word embeddings.ipynb".
The first task is:
Uniform Maniford Approximation and Projection (UMAP) is a new and particularly fast dimensionality reduction method with some comparatively great documentation. Unfortunately, UMAP is so new that it hasn't been translated into scikit-learn yet, so you'll need to install it separately from the terminal:
conda install -c conda-forge umap-learn
Can you write Python code for it?

# Step 1: Choose your model class(es)
from umap import UMAP

# Step 2: Instantiate your model class(es)
umap_ = UMAP(n_components=2, n_neighbors=10, random_state=42)

# Step 3: Arrange data into features matrices
# Done!

# Step 4: Fit the data and transform
X_umap = umap_.fit_transform(bio_tfidf_dense)

# Plot the data
f,ax = plt.subplots(1,1,figsize=(10,10))
ax.scatter(X_umap[:,0],X_umap[:,1])

ax.set_title('UMAP')

for i,txt in enumerate(bios.keys()):
    if txt == 'Barack Obama':
        ax.annotate(txt,(X_umap[i,0],X_umap[i,1]),color='blue',fontweight='bold')
    elif txt == 'Donald Trump':
        ax.annotate(txt,(X_umap[i,0],X_umap[i,1]),color='red',fontweight='bold')
    else:
        ax.annotate(txt,(X_umap[i,0],X_umap[i,1]))


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 3 - Text networks and word embeddings.ipynb".
The first task is:
We can then use the dimensionality tools we just covered in the previous section to visualize the word similarities. PCA is fast but rarely does a great job with this extremely high-dimensional and sparse data: it's a cloud of points with no discernable structure.
Can you write Python code for it?

# Step 1: Choose your model class(es)
# from sklearn.decomposition import PCA

# Step 2: Instantiate the model
pca = PCA(n_components=2)

# Step 3: Arrange data into features matrices
X_w2v = top_word_vectors

# Step 4: Fit the data and transform
X_w2v_pca = pca.fit_transform(X_w2v)

# Plot the data
f,ax = plt.subplots(1,1,figsize=(10,10))
ax.scatter(X_w2v_pca[:,0],X_w2v_pca[:,1],s=3)

ax.set_title('PCA')

for i,txt in enumerate(top_words):
    if i%10 == 0:
        ax.annotate(txt,(X_w2v_pca[i,0],X_w2v_pca[i,1]))

f.savefig('term_pca.pdf')


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 3 - Text networks and word embeddings.ipynb".
The first task is:
We read this pandas edgelist into networkx using from_pandas_edgelist, report out some basic descriptives about the network, and write the graph object to file in case we want to visualize it in a dedicated network visualization package like Gephi.
Can you write Python code for it?

# Convert from edgelist to a graph object
g = nx.from_pandas_edgelist(edgelist_df,source='from',target='to',edge_attr=['weight'])

# Report out basic descriptives
print("There are {0:,} nodes and {1:,} edges in the network.".format(g.number_of_nodes(),g.number_of_edges()))

# Write graph object to disk for visualization
nx.write_gexf(g,'bio_similarity.gexf')


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 3 - Text networks and word embeddings.ipynb".
The first task is:
What's an example of a single biography? We access the dictionary by passing the key (President's name), which returns the value (the text of the biography).
Can you write Python code for it?

example = bios['Grover Cleveland']
print(example)


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 3 - Text networks and word embeddings.ipynb".
The first task is:
t-SNE was more-or-less engineered for precisely the task of visualizing word embeddings. It likely takes on the order of a minute or more for t-SNE to reduce the top_words embeddings to only two dimensions. Assuming our perplexity and other t-SNE hyperparameters are well-behaved, there should be relatively easy-to-discern clusters of words with similar meanings. You can also open the "term_sne.pdf" file and zoome to inspect.
Can you write Python code for it?

# Step 1: Choose your model class(es)
from sklearn.manifold import TSNE

# Step 2: Instantiate your model class(es)
tsne = TSNE(n_components = 2, init='pca', random_state=42, perplexity=25)

# Step 3: Arrange data into features matrices
X_w2v = top_word_vectors

# Step 4: Fit the data and transform
X_w2v_tsne = tsne.fit_transform(X_w2v)

# Plot the data
f,ax = plt.subplots(1,1,figsize=(10,10))
ax.scatter(X_w2v_tsne[:,0],X_w2v_tsne[:,1],s=3)

ax.set_title('t-SNE')

for i,txt in enumerate(top_words):
    if i%10 == 0:
        ax.annotate(txt,(X_w2v_tsne[i,0],X_w2v_tsne[i,1]))

f.savefig('term_tsne.pdf')


I am working on the file "2018/materials/boulder/day5-causal-inference/Day 5 - Case Study.ipynb".
The first task is:
Appendix 2: Get page outlinks
Load libraries and define two functions:

get_page_outlinks - Get all of the outlinks from the current version of the page.
get_pageviews - Get all of the pageviews for an article over a time range
Can you write Python code for it?

from datetime import datetime
import requests, json
from bs4 import BeautifulSoup
from urllib.parse import urlparse, quote, unquote
import networkx as nx

def get_page_outlinks(page_title,lang='en',redirects=1):
    """Takes a page title and returns a list of wiki-links on the page. The
    list may contain duplicates and the position in the list is approximately
    where the links occurred.

    page_title - a string with the title of the page on Wikipedia
    lang - a string (typically two letter ISO 639-1 code) for the language
        edition, defaults to "en"
    redirects - 1 or 0 for whether to follow page redirects, defaults to 1

    Returns:
    outlinks_per_lang - a dictionary keyed by language returning a dictionary
        keyed by page title returning a list of outlinks
    """

    # Replace spaces with underscores
    page_title = page_title.replace(' ','_')

    bad_titles = ['Special:','Wikipedia:','Help:','Template:','Category:','International Standard','Portal:','s:','File:','Digital object identifier','(page does not exist)']

    # Get the response from the API for a query
    # After passing a page title, the API returns the HTML markup of the current article version within a JSON payload
    req = requests.get('https://{2}.wikipedia.org/w/api.php?action=parse&format=json&page={0}&redirects={1}&prop=text&disableeditsection=1&disabletoc=1'.format(page_title,redirects,lang))

    # Read the response into JSON to parse and extract the HTML
    json_string = json.loads(req.text)

    # Initialize an empty list to store the links
    outlinks_list = []

    if 'parse' in json_string.keys():
        page_html = json_string['parse']['text']['*']

        # Parse the HTML into Beautiful Soup
        soup = BeautifulSoup(page_html,'lxml')

        # Remove sections at end
        bad_sections = ['See_also','Notes','References','Bibliography','External_links']
        sections = soup.find_all('h2')
        for section in sections:
            if...


I am working on the file "2018/materials/boulder/day5-causal-inference/Day 5 - Case Study.ipynb".
The first task is:
Case study: Wikipedia pageview dynamics
On November 8, 2016, California voters passed Proposition 64 legalizing recreational use of cannabis. On January 1, 2018, recreational sales began. The following two files capture the daily pageview data for the article Cannabis in California as well as the daily pageview for all the other pages it links to.
Can you write Python code for it?

ca2016 = pd.read_csv('wikipv_ca_2016.csv',encoding='utf8',parse_dates=['timestamp']).set_index('timestamp')
ca2018 = pd.read_csv('wikipv_ca_2018.csv',encoding='utf8',parse_dates=['timestamp']).set_index('timestamp')
ca2016.head()


I am working on the file "2018/materials/boulder/day5-causal-inference/Day 5 - Case Study.ipynb".
The first task is:
Changes in fatal accidents on 4/20?
There's another exogenous source of variation in this car crash data we can exploit: the unofficial cannabis enthusiast holiday of April 20. If consumption increases on this day compared to a week before or after (April 13 and 27), does this explain changes in fatal auto accidents?
Can you write Python code for it?

counts_df.head()

population_estimates.head()

# Select only data after 2004 in the month of April
april_df = counts_df.query('MONTH == 4 & YEAR > 2004').set_index(['STATE','HOUR','MONTH','DAY','YEAR'])

# Re-index the data to fill in missing dates
ix = pd.MultiIndex.from_product([state_codings.values(),range(0,24),[4],range(1,31),range(2005,2017)],
                                names = ['STATE','HOUR','MONTH','DAY','YEAR'])

april_df = april_df.reindex(ix).fillna(0)
april_df.reset_index(inplace=True)

# Add in population data
april_df = pd.merge(april_df,population_estimates,
                    left_on=['STATE','YEAR'],right_on=['State','Year'])
april_df = april_df[[i for i in april_df.columns if i not in ['Year','State']]]

# Inspect
april_df.head()

# Calculate whether day is a Friday, Saturday, or Sunday
april_df['Weekday'] = pd.to_datetime(april_df[['YEAR','MONTH','DAY']]).apply(lambda x:x.weekday())
april_df['Weekend'] = np.where(april_df['Weekday'] >= 4,1,0)

# Treated days are on April 20
april_df['Fourtwenty'] = np.where(april_df['DAY'] == 20,1,0)
april_df['Legal'] = np.where((april_df['STATE'].isin(['Colorado','Washington'])) & (april_df['YEAR'] >= 2014),1,0)

# Examine data for a week before and after April 20
april_df = april_df[april_df['DAY'].isin([13,20,27])]

# Inspect Colorado data
april_df.query('STATE == "Colorado"').sort_values(['YEAR','DAY'])


I am working on the file "2018/materials/boulder/day5-causal-inference/Day 5 - Case Study.ipynb".
The first task is:
Define a function cleanup_pageviews to make a rectangular DataFrame with dates as index, pages as columns, and pageviews as values.
Can you write Python code for it?

def cleanup_pageviews(pv_dict):
    _df = pd.concat(pv_dict.values())
    _df.reset_index(inplace=True)
    _df.columns = _df.columns.droplevel(0)
    _df.columns = ['timestamp','pageviews','page']
    _df = _df.set_index(['timestamp','page']).unstack('page')['pageviews']
    return _df

cleanup_pageviews(pvs_2016).to_csv('wikipv_ca_2016.csv',encoding='utf8')
cleanup_pageviews(pvs_2018).to_csv('wikipv_ca_2018.csv',encoding='utf8')


I am working on the file "2018/materials/boulder/day5-causal-inference/Day 5 - Case Study.ipynb".
The first task is:
Estimate the models. The Fourtwenty and Legal dummy variables (and their interaction) capture whether fatal accidents increased on April 20 compared to the week beforehand and afterwards, while controlling for state legality, year, whether it's a weekend, and state population. We do not observe a statistically signidicant increase in the number of incidents, alcohol-involved incidents, and total fatalities on April 20.
Can you write Python code for it?

m_cases_420 = smf.ols(formula = 'ST_CASE ~ Fourtwenty*Legal + YEAR + Weekend + Population',
                  data = april_df).fit()
print(m_cases_420.summary())

m_drunk_420 = smf.ols(formula = 'DRUNK_DR ~ Fourtwenty*Legal + YEAR + Weekend + Population',
                  data = april_df).fit()
print(m_drunk_420.summary())

m_fatal_420 = smf.ols(formula = 'FATALS ~ Fourtwenty*Legal + YEAR + Weekend + Population',
                  data = april_df).fit()
print(m_fatal_420.summary())


I am working on the file "2018/materials/boulder/day5-causal-inference/Day 5 - Case Study.ipynb".
The first task is:
Exploratory data analysis
Can you write Python code for it?

sb.factorplot(x='HOUR',y='ST_CASE',hue='WEEKDAY',data=counts_df,
              aspect=2,order=range(24),palette='nipy_spectral',dodge=.5)

sb.factorplot(x='MONTH',y='ST_CASE',hue='WEEKDAY',data=counts_df,
              aspect=2,order=range(1,13),palette='nipy_spectral',dodge=.5)


I am working on the file "2018/materials/boulder/day5-causal-inference/Day 5 - Case Study.ipynb".
The first task is:
Get all of the links from "Cannabis in California", and add the seed article itself.
Can you write Python code for it?

ca_links = get_page_outlinks('Cannabis in California') + ['Cannabis in California']

link_d = {}

for l in list(set(ca_links)):
    link_d[l] = get_page_outlinks(l)


I am working on the file "2018/materials/boulder/day5-causal-inference/Day 5 - Case Study.ipynb".
The first task is:
Groupby-aggregate the data by state, month, day, and year counting the number of cases, alcohol-involved deaths, and total fatalities. Save the data as "accident_counts.csv".
Can you write Python code for it?

gb_vars = ['STATE','HOUR','DAY','MONTH','YEAR']
agg_dict = {'ST_CASE':len,'DRUNK_DR':np.sum,'FATALS':np.sum}
counts_df = all_accident_df.groupby(gb_vars).agg(agg_dict).reset_index()

counts_df['STATE'] = counts_df['STATE'].map(state_codings)
counts_df = counts_df.query('YEAR > 1999')
counts_df.to_csv('accident_counts.csv',encoding='utf8',index=False)
counts_df.head()


I am working on the file "2018/materials/boulder/day5-causal-inference/Day 5 - Case Study.ipynb".
The first task is:
Here are two plots of the pageview dynamics for the seed "Cannabis in California" article.
Can you write Python code for it?

f,axs = plt.subplots(2,1,figsize=(10,5))
ca2016['Cannabis in California'].plot(ax=axs[0],color='red',lw=3)
ca2018['Cannabis in California'].plot(ax=axs[1],color='blue',lw=3)
axs[0].axvline(pd.Timestamp('2016-11-08'),c='k',ls='--')
axs[1].axvline(pd.Timestamp('2018-01-01'),c='k',ls='--')

for ax in axs:
    ax.set_ylabel('Pageviews')

f.tight_layout()


I am working on the file "2018/materials/boulder/day5-causal-inference/Day 5 - Case Study.ipynb".
The first task is:
Here is a visualization of the local hyperlink network around "Cannabis in California."
Can you write Python code for it?

g = nx.read_gexf('wikilinks_cannabis_in_california.gexf')

f,ax = plt.subplots(1,1,figsize=(10,10))

g_pos = nx.layout.kamada_kawai_layout(g)

nx.draw(G = g,
        ax = ax,
        pos = g_pos,
        with_labels = True,
        node_size = [dc*(len(g) - 1)*10 for dc in nx.degree_centrality(g).values()],
        font_size = 7.5,
        font_weight = 'bold',
        node_color = 'tomato',
        edge_color = 'grey'
       )


I am working on the file "2018/materials/boulder/day5-causal-inference/Day 5 - Case Study.ipynb".
The first task is:
Inspect the main effect of cases on April 20 are compared to the week before and after.
Can you write Python code for it?


I am working on the file "2018/materials/boulder/day5-causal-inference/Day 5 - Case Study.ipynb".
The first task is:
Make a network object of these hyperlinks among articles.
Can you write Python code for it?

g = nx.DiGraph()
seed_edges = [('Cannabis in California',l) for l in link_d['Cannabis in California']]
#g.add_edges_from(seed_edges)
for page,links in link_d.items():
    for link in links:
        if link in link_d['Cannabis in California']:
            g.add_edge(page,link)

print("There are {0:,} nodes and {1:,} edges in the network.".format(g.number_of_nodes(),g.number_of_edges()))
nx.write_gexf(g,'wikilinks_cannabis_in_california.gexf')


I am working on the file "2018/materials/boulder/day5-causal-inference/Day 5 - Case Study.ipynb".
The first task is:
What kinds of causal arguments could you make from these pageview data and the hyperlink networks?
Appendix 1: Cleaning NHTSA FARS Data
"accidents.csv" is a ~450MB file after concatenating the raw annual data from NHTSA FARS.
Can you write Python code for it?

all_accident_df = pd.read_csv('accidents.csv',encoding='utf8',index_col=0)
all_accident_df.head()


I am working on the file "2018/rate.am.ipynb".
The first task is:
old=""
new=" "
while new>old:
    old = browser.page_source
    button.send_keys(Keys.END)
    new = browser.page_source
Can you write Python code for it?

browser.get("https://www.bloomberg.com/")

browser.implicitly_wait(30)
browser.find_element_by_partial_link_text("S&P")

#EC(presense_of_element_located())


I am working on the file "2018/tf_tutorial/neural_network/Neural_network.ipynb".
The first task is:
Build model (DNN)
일반적으로 이미지의 학습은 CNN 으로 이루어지지만 기본적으로 DNN에 대해 알아야 하기때문에 DNN 으로 모델을 만든다.
Can you write Python code for it?

w_1 = tf.get_variable("weight1",shape=[img_width*img_height*channel, 256], initializer=tf.random_normal_initializer())
b_1 = tf.get_variable("bias1", shape=[256,] ,initializer=tf.zeros_initializer())
layer_1 = tf.nn.bias_add(tf.matmul(X_img,w_1),b_1)
layer_1 = tf.nn.relu(layer_1)

w_2 =  tf.get_variable("weight2",shape=[256, 512], initializer=tf.random_normal_initializer())
b_2 = tf.get_variable("bias2", shape=[512,] ,initializer=tf.zeros_initializer())
layer_2 = tf.nn.bias_add(tf.matmul(layer_1,w_2),b_2)
layer_2 = tf.nn.relu(layer_2)

w_3 =  tf.get_variable("weight3",shape=[512, 2], initializer=tf.random_normal_initializer())
b_3 = tf.get_variable("bias3", shape=[2,] ,initializer=tf.zeros_initializer())
layer_3 = tf.nn.bias_add(tf.matmul(layer_2,w_3),b_3)


I am working on the file "2018/tf_tutorial/neural_network/Neural_network.ipynb".
The first task is:
Data argumentation
우리가 이번에 사용하는 데이터는 식물의 잎 모양을 보고 분류하는 모델이다. 그렇다면 어떠한 전처리를 해야 효과적일까.

식물의 잎을 보고 판단을 내려야 한다. 식물의 잎외에 보여지는 이미지는 학습요소가 될 수 없으며 모델의 성능에 악영향을 끼칠것이다.

따라서 우리는 최대한 식물의 잎을 보는 이미지로 전처리 해야하며 이러한 과정은 좋은 모델을 쓰는 것만큼 매우! 중요하다.

데이터 전처리를 하기전 항상 데이터를 들여다보고 어떠한 처리를 해야할 지 생각해야한다. 가장 우선적으로 데이터셋을 살펴보자
Can you write Python code for it?

plt.imshow(img_matrix[0])
plt.title("Original Image")
print("Image size :", np.shape(img_matrix))


I am working on the file "2018/tf_tutorial/neural_network/Neural_network.ipynb".
The first task is:
One-hot encoding
라벨에 대한 정보를 수치화 해야하며 classification 의 경우 One-hot encoding 의 방식을 보편적으로 사용한다. <br>
one-hot encoding 이란 [개, 고양이, 고양이, 개] 의 형식을 [[0,1],[1,0],[1,0],[0,1]] 의 방식으로 표현한 방법이다.
Can you write Python code for it?


I am working on the file "2018/tf_tutorial/neural_network/Neural_network.ipynb".
The first task is:
Range of image
RGB 의 3 채널을 가지는 이미지이다. 여기서 각 pixel 들은 0~255 의 범위를 가지고 있다. 하지만 0~255의 범위로 데이터셋이 들어가게 되면 경계면의 pixel 차이가 커지게되고 그것은 weight 들의 element 들이 dense 하지 않는다. 그렇기 때문에 우리는 0~255의 범위를 0~1로 변환한다.
Can you write Python code for it?

img = np.array(img_matrix)/255
print("change the range of image : \n",img[0,0:3,0:3,0])
print(np.shape(img))


I am working on the file "2018/tf_tutorial/neural_network/Neural_network.ipynb".
The first task is:
Remove obstacle feature
MNIST 를 생각해보자. MNIST 는 고도로 정제된 데이터셋이다. 우리는 어떠한 객체를 인식하고 분류해야한다.

하지만 위의 데이터셋을 보았을 때 seed 이외의 자갈 등은 학습요소의 저하요인이다. 따라서 이러한 feature 들을 제거해야한다.

각각의 이미지는 RGB 채널로 이루어져 있다. 그렇다면 우리는 G channel 의 값이 충분이 높은 pixel 의 값은 살리고 나머지는 제거한다면 green 의 영역의 pixel 값들만 추출될 것이다.
Can you write Python code for it?

# RGB channel
for i in range(len(img)):
    row_img = img[i]
    red = row_img[:,:,0]
    green = row_img[:,:,1]
    blue = row_img[:,:,2]
    red_reshape = np.reshape(red, -1)
    green_reshape = np.reshape(green, -1)
    blue_reshape = np.reshape(blue, -1)
    for index, value in enumerate(zip(red_reshape,green_reshape,blue_reshape)):
        if value[1] < 0.20:
            img[i,index//256, index%256, :] = 0
        elif value[0] > 0.30:
            img[i,index//256, index%256, :] = 0
        elif value[2] > 0.20:
            img[i,index//256, index%256, :] = 0


I am working on the file "2018/tf_tutorial/neural_network/Neural_network.ipynb".
The first task is:
Set parameter
이제 전처리의 과정은 지났다. 모델을 학습시키기 위해서 파라메터 설정과 모델을 만들자
Can you write Python code for it?

img_width = np.shape(img)[1]
img_height = np.shape(img)[1]
channel = 3
batch_size = 64
learning_rate = 0.01
epoch = 7

X = tf.placeholder(tf.float32, [None, img_width, img_width, channel])
y = tf.placeholder(tf.float32, [None, 2])
X_img = tf.reshape(X,[-1, img_width*img_height*channel])


I am working on the file "20180129-cohpy-or-operator.ipynb".
The first task is:
It is too bad that there is not an or= operator.
C does not have a ||= operator either.
Can you write Python code for it?


I am working on the file "20180129-cohpy-or-operator.ipynb".
The first task is:
Maybe a better name for this presentation would be 'this' or 'that'.
Can you write Python code for it?

'this' or 'that'

'give me liberty' or 'give me death'


I am working on the file "20180129-cohpy-or-operator.ipynb".
The first task is:
Note that the reduce() evaluates all the elements of its second operand,
whereas a big long multiple "or" expression
is guaranteed to stop evaluating operands after the first true operand.
Then I thought the operator module should eliminate the need for the lamdba,
so I explored the operator module.
Can you write Python code for it?

import operator

[s for s in dir(operator) if 'or' in s]


I am working on the file "20180129-cohpy-or-operator.ipynb".
The first task is:
Now we get to how the "or" operator was used in fizzbuzz().
Can you write Python code for it?

'' or 1

'' or 2

'fizz' or 3

'buzz' or 5

'fizz' or 6

'fizzbuzz' or 15

'' or 16


I am working on the file "20180129-cohpy-or-operator.ipynb".
The first task is:
Pete Carswell asked about doing the above long expressions with a lamdba,
hence the following.
Can you write Python code for it?

from functools import reduce

a = (
    False,
    0,
    0j,
    0.0,
    [],
    {},
    'look ma no hands',
    set(),
    None,
    (),
)

reduce(lambda x, y: x or y, a)


I am working on the file "20180129-cohpy-or-operator.ipynb".
The first task is:
Python's concept of truthiness
Numerical values are false if zero and true if not zero.
None is false.
Sequences and collections are false if empty, and true if not empty.
Can you write Python code for it?


I am working on the file "20180129-cohpy-or-operator.ipynb".
The first task is:
Python's or
operator has some similarities with C's || operator.

Both always evaluate the first operand.
Both guarantee that the second operand is not evaluated
if the first operand is true.

C's || operator yields an integer: 0 (false) or 1 (true).
Python "or" operator yields one of the operands.
If the first operand is true, it is the result
and the second operand is guaranteed to not be evaluated.
If the first operand if false,
the second operand is evaluated and is the result.
Note that Python returns one of the operands, not merely 0 or 1.
Python's concept of truth
...
Here are most of the built-in objects considered false:

constants defined to be false: None and False.
zero of any numeric type: 0, 0.0, 0j, Decimal(0), Fraction(0, 1)
empty sequences and collections: '', (), [], {}, set(), range(0)

...
Objects that are not false are true.

constants defined to be true: True.
not zero of any numeric type: 1, -1, 0.01, 1j, Decimal(1), Fraction(-1, 1)
non-empty sequences and collections: 'hello', (0,), [0], {0: 0}, set([0]), range(1)

For each of the following cells, predict the output.
During the meeting, there was much disagreement in predictions,
especially in the first four cells below.
That led to real learning.
Can you write Python code for it?

False or False

0 or False

False or 0

0 or 0

False or True

True or False

True or True

True or 1

1 or True

1 or 1

1 > 2

3 < 4

# This kind of expression using the "or" operator is very typical,
# comprising the vast majority of use.

1 > 2 or 3 < 4

'hello' or 'world'

'' or 'world'

'hello' or ''

'' or ''

'' or None

False or 3.14

'False' or 3.14

bool('False' or 3.14)

[] or {}

'' or []

'' or {}

'' or (1, 3)

'' or 'False'

'' or 'True'

'' or True

'' or False


I am working on the file "20180129-cohpy-or-operator.ipynb".
The first task is:
Slowly scroll through the output of the following cell,
predicting the output of each value
before scrolling to reveal the actual output.
Can you write Python code for it?

for value in values:
    print(repr(value), type(value))
    print(bool(value))
    print()


I am working on the file "20180129-cohpy-or-operator.ipynb".
The first task is:
That can be fixed with a traditional "is None" test.
Can you write Python code for it?

def foo(p=None):
    if p is None:
        p = [1, 2, 3, 4]
    return p

foo()

foo(None)

foo([1])

foo([])

foo(0)


I am working on the file "20180129-cohpy-or-operator.ipynb".
The first task is:
There was some confusion and disbelief
that True and False are integers,
so that was played with.
Some folks also did not know about
the distinction between the / and // operators in Python 3,
so that was played with also.
Can you write Python code for it?

True + True

True / (True + True)

True // (True + True)


I am working on the file "20180129-cohpy-or-operator.ipynb".
The first task is:
Unfortunately, I was not able to find an equivalent to the "or" operator.

Zach brought up another use of the "or" operator
for handling default arguments.
Can you write Python code for it?

def foo(p=None):
    p = p or [1, 2, 3, 4]
    return p


I am working on the file "20180129-cohpy-or-operator.ipynb".
The first task is:
Zach prefers his code above to code below,
with the danger of its mutable default value.
Can you write Python code for it?

def foo(p=[1, 2, 3, 4]):
    return p

foo(3)

foo()

a = foo()
a[1] = 'hi mom'
a


I am working on the file "20180129-cohpy-or-operator.ipynb".
The first task is:
Zach's version does not suffer from the mutable default argument problem.
Can you write Python code for it?

def foo(p=None):
    p = p or [1, 2, 3, 4]
    return p

b = foo()
b

b[2] = 'this'
b

foo()


I am working on the file "20180320-twitter-commandline/Twitter command-line.ipynb".
The first task is:
Filter CSVs
Here I'm finding all of the tweets that are replies and getting the tweets to which they are replies.
Can you write Python code for it?

!csvgrep -c tweet_type -m reply gelmanlibrary.csv | csvcut -c id,in_reply_to_status_id > replies_to_in_reply_to.csv
!head -5 replies_to_in_reply_to.csv


I am working on the file "20180320-twitter-commandline/Twitter command-line.ipynb".
The first task is:
Select columns
Can you write Python code for it?

!csvcut -c id,created_at,text gelmanlibrary.csv > gelmanlibrary_cut.csv
!head -3 gelmanlibrary_cut.csv


I am working on the file "20180320-twitter-commandline/Twitter command-line.ipynb".
The first task is:
Tip when loading a tweet CSV into Excel: If you open up a tweet CSV with Excel, it will mishandle tweet and user ids. For example, 976161920133816322 will become 976161920133816000.
To correctly import tweet CSV into Excel, select Data > Get External Data > Import File. When given the option of selecting the data type for fields, select text for all id fields.
csvkit
csvkit supports a wide variety of operations for filtering and transforming CSV files. Here are a few highlights.
Can you write Python code for it?


I am working on the file "20180320-twitter-commandline/Twitter command-line.ipynb".
The first task is:
Uncompress to a stream of tweets
Can you write Python code for it?


I am working on the file "20180320-twitter-commandline/Twitter command-line.ipynb".
The first task is:
awk
Sample tweets
In the following example, I create a 1 in 5 sample.
Can you write Python code for it?

!gunzip -c *.jsonl.gz | awk 'NR % 5 == 0' > sample.json
!wc -l sample.json


I am working on the file "20180320-twitter-commandline/Twitter command-line.ipynb".
The first task is:
json2csv.py
json2csv.py is a utility that is part of Twarc.
Can you write Python code for it?

!git clone https://github.com/DocNow/twarc.git
!pip install -e twarc


I am working on the file "20180320-twitter-commandline/Twitter command-line.ipynb".
The first task is:
sort
Sort a list
-r reverses the sort order (see the example for uniq).
Can you write Python code for it?

!gunzip -c *.jsonl.gz | jq -r '.entities.user_mentions[].screen_name' | sort > sorted_screen_names.txt
!head -10 sorted_screen_names.txt


I am working on the file "20180320-twitter-commandline/Twitter command-line.ipynb".
The first task is:
split
Split tweets into files by number of tweets
Can you write Python code for it?

!gunzip -c *.jsonl.gz | split --lines=1000 -d --additional-suffix=.jsonl - tweets-
!wc -l tweets-*.jsonl


I am working on the file "20180320-twitter-commandline/Twitter command-line.ipynb".
The first task is:
tee
Read once, write twice
In this example, I write out a lists of tweet ids and user ids.
Can you write Python code for it?

%%bash
gunzip -c gwtweets.jsonl.gz | tee >(jq -r '.user.id_str' > gwtweets-user_ids.txt) | jq -r '.id_str' > gwtweets-tweet_ids.txt
head -5 gwtweets-tweet_ids.txt
head -5 gwtweets-user_ids.txt


I am working on the file "20180320-twitter-commandline/Twitter command-line.ipynb".
The first task is:
uniq
Get the unique values for a list
The list must be sorted first. Omit -c to remove counts.
Can you write Python code for it?

!gunzip -c *.jsonl.gz | jq -r '.entities.user_mentions[].screen_name' | sort | uniq -c | sort -r > unique_screen_names.txt
!head -10 unique_screen_names.txt


I am working on the file "20180320-twitter-commandline/Twitter command-line.ipynb".
The first task is:
wc gotcha: When counting many tweets, wc -l will output a partial total and then reset the count.
gzip
Because of the size of the datasets that I deal with, I usually gzip compress the tweets.
Compress a stream of tweets
Can you write Python code for it?

!cat gwtweets.jsonl | gzip > gwtweets.jsonl.gz
!cat gelmanlibrary.jsonl | gzip > gelmanlibrary.jsonl.gz
!ls *.jsonl.gz


I am working on the file "20180511-pyspark-elasticsearch/PySpark-ElasticSearch.ipynb".
The first task is:
Get the time zone
Can you write Python code for it?

tz_df = sqlContext.sql("SELECT user_time_zone, count(user_time_zone) FROM tweets group by user_time_zone order by count(user_time_zone) desc")
tz_df.show(10, truncate=False)


I am working on the file "20180511-pyspark-elasticsearch/PySpark-ElasticSearch.ipynb".
The first task is:
Get the top hashtags
Can you write Python code for it?

tweets_rdd.flatMap(lambda t: t[1]['hashtags']).map(lambda x: (x, 1)).reduceByKey(lambda x,y: x + y).sortBy(lambda x: x[1], ascending=False).take(10)


hashtags_df = sqlContext.sql("SELECT hashtag, count(hashtag) from (SELECT explode(hashtags) hashtag FROM tweets) group by hashtag order by count(hashtag) desc")
hashtags_df.show(10, truncate=False)


I am working on the file "20180511-pyspark-elasticsearch/PySpark-ElasticSearch.ipynb".
The first task is:
Get the top retweets
Can you write Python code for it?

rt_df = sqlContext.sql("SELECT CONCAT('https://twitter.com/', retweeted_quoted_screen_name, '/status/', retweet_quoted_status_id), count(retweet_quoted_status_id) FROM tweets group by retweet_quoted_status_id, retweeted_quoted_screen_name order by count(retweet_quoted_status_id) desc")
rt_df.show(10, truncate=False)


I am working on the file "20180511-pyspark-elasticsearch/PySpark-ElasticSearch.ipynb".
The first task is:
Get the top trigrams
Can you write Python code for it?

from pyspark.ml.feature import RegexTokenizer, NGram, StopWordsRemover
from pyspark.sql.functions import sort_array, udf, explode
from pyspark.sql.types import ArrayType, StringType

# Text (using distinct)
text_df = tweets_df.select(explode("text").alias("text")).distinct()

# Tokenize
tokenizer = RegexTokenizer(pattern="([:\.!?,]|'s|’s)*\\s+[‘]*", inputCol="text", outputCol="words")
tokenized_df = tokenizer.transform(text_df)

# Stopwords
stop_words = StopWordsRemover.loadDefaultStopWords('english')
stop_words.extend(['rt', ' ', '-', '&', 'it’s', '', 'may', 'see', 'want', 'i’m', 'us', 'make', "we've", "you're", "you've", "don't", "i’ve", 'it', 'they’re', 'don’t', 'lets', 'add'])
remover = StopWordsRemover(inputCol="words", outputCol="filtered_words", stopWords=stop_words)
filtered_df = remover.transform(tokenized_df)

# Remove hashtags and URLs and dupes
def clean(arr):
    new_arr = set()
    for item in arr:
        add_to_arr = True
        for startswith in ('#', 'http'):
            if item.startswith(startswith):
                add_to_arr = False
        if add_to_arr:
            new_arr.add(item)
    return list(new_arr)

clean_udf = udf(lambda arr: clean(arr), ArrayType(StringType()))
clean_df = filtered_df.withColumn("clean_words", clean_udf(filtered_df.filtered_words))

# Sort the words
sorted_df = clean_df.select(sort_array('clean_words').alias('sorted_words'))

ngram = NGram(n=3, inputCol="sorted_words", outputCol="ngrams")
ngram_df = ngram.transform(sorted_df).select(explode('ngrams').alias('ngrams'))
ngram_df.groupBy('ngrams').count().orderBy('count', ascending=False).show(20, truncate=False)


I am working on the file "20180511-pyspark-elasticsearch/PySpark-ElasticSearch.ipynb".
The first task is:
Get the top user languages
By parsing and extracting from each tweet since it is not already a field.
Can you write Python code for it?

import json

parsed_tweets_rdd = tweets_rdd.map(lambda x: json.loads(x[1]['tweet'])).persist()
parsed_tweets_rdd.map(lambda t: (t['user']['lang'], 1)).reduceByKey(lambda x,y: x + y).sortBy(lambda x: x[1], ascending=False).take(10)


I am working on the file "20180511-pyspark-elasticsearch/PySpark-ElasticSearch.ipynb".
The first task is:
Print schema
Can you write Python code for it?


I am working on the file "20180511-pyspark-elasticsearch/PySpark-ElasticSearch.ipynb".
The first task is:
Retrieve the first element from the RDD.
Can you write Python code for it?


I am working on the file "20180511-pyspark-elasticsearch/PySpark-ElasticSearch.ipynb".
The first task is:
Using a SQL table
Create SQL table
Can you write Python code for it?

from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)
es_conf['es.read.field.as.array.include'] = 'hashtags,text,urls'
tweets_df = sqlContext.read.format("org.elasticsearch.spark.sql").options(**es_conf).load()
tweets_df.createOrReplaceTempView("tweets")


I am working on the file "20181027-ccc-or-operator.ipynb".
The first task is:
By the way, Python objects can define their own truthiness
by defining the
_bool_()
or
_len_()
methods.
Joe Shaw: The 'wat' lightning talk by Gary Bernhardt.

2018-10-28 afterthoughts
Can you write Python code for it?

bool(True)

bool(-True)

bool(True + 1j)

bool(True - 1j)

bool(True + 1j**2)


I am working on the file "20181027-ccc-or-operator.ipynb".
The first task is:
Now we get to a more serious discussion of when it is good to use
the "or" operator
where the operands are not merely
the typical case of False or True.
The short answer is:
Use the "or" operator when it makes the code more readable.

But that begs the question.
When does using the "or" operator make the code more readable?

This is a question I have been struggling with.
Let's go back to the old code at hand.
if not output:
    return str(n)
return output

It returns either output or str(n).
When said that simply in English,
Neil's refactoring is the most readable code.
It says most simply and directly what we want.
return output or str(n)

The problem may be from the biases of experienced programmers like myself
who expect the "or" operator to yield only a true or false value,
like we expect from other languages
such as but not limited to C.
Inexperienced folks do not bring such baggage from other languages.
For myself,
I have decided to absorb and use the idiom like Neil showed us.
It is part of learning Python.

With a long string of "or"ed stuff,
the result is the first true operand.
If no operands are true, the result is the last operand.
Can you write Python code for it?

False or 0 or 0j or 0.0 or [] or {} or set() or None or ()

False or 0 or 0j or 0.0 or 'false' or [] or {} or set() or None or ()


I am working on the file "20181027-ccc-or-operator.ipynb".
The first task is:
Now we get to how the "or" operator was used in fizzbuzz().
Can you write Python code for it?

'' or 1

'' or 2

'fizz' or 3

'buzz' or 5

'fizz' or 6

'fizzbuzz' or 15

'' or 16


I am working on the file "20181027-ccc-or-operator.ipynb".
The first task is:
Python's concept of truthiness
Numerical values are false if zero and true if not zero.
None is false.
Sequences and collections are false if empty, and true if not empty.
Can you write Python code for it?


I am working on the file "20181027-ccc-or-operator.ipynb".
The first task is:
Python's or
operator has some similarities with C's || operator.

Both always evaluate the first operand.
Both guarantee that the second operand is not evaluated
if the first operand is true.

C's || operator yields an integer: 0 (false) or 1 (true).
Python "or" operator yields one of the operands.
If the first operand is true, it is the result
and the second operand is guaranteed to not be evaluated.
If the first operand if false,
the second operand is evaluated and is the result.
Note that Python returns one of the operands, not merely 0 or 1.
Python's concept of truth
...
Here are most of the built-in objects considered false:

constants defined to be false: None and False.
zero of any numeric type: 0, 0.0, 0j, Decimal(0), Fraction(0, 1)
empty sequences and collections: '', (), [], {}, set(), range(0)

...
Objects that are not false are true.

constants defined to be true: True.
not zero of any numeric type: 1, -1, 0.01, 1j, Decimal(1), Fraction(-1, 1)
non-empty sequences and collections: 'hello', (0,), [0], {0: 0}, set([0]), range(1)

For each of the following cells, predict the output.
During the meeting, there was much disagreement in predictions,
especially in the first four cells below.
That led to real learning.
Can you write Python code for it?

False or False

0 or False

False or 0

0 or 0

False or True

True or False

True or True

True or 1

1 or True

1 or 1

1 > 2

3 < 4

# This kind of expression using the "or" operator is very typical,
# comprising the vast majority of use.

1 > 2 or 3 < 4

'this' or 'that'

'' or 'that'

'this' or ''

'' or ''

'' or None

False or 3.14

'False' or 3.14

bool('False' or 3.14)

[] or {}

'' or []

'' or {}

'' or (1, 3)

'' or 'False'

'' or 'True'

'' or True

'' or False


I am working on the file "20181027-ccc-or-operator.ipynb".
The first task is:
Slowly scroll through the output of the following cell,
predicting the output of each value
before scrolling to reveal the actual output.
Can you write Python code for it?

for value in values:
    print(repr(value), type(value))
    print(bool(value))
    print()


I am working on the file "20181027-ccc-or-operator.ipynb".
The first task is:
There was some confusion and disbelief
that True and False are integers,
so that was played with.
Some folks also did not know about
the distinction between the / and // operators in Python 3,
so that was played with also.
Can you write Python code for it?

True + True

True / (True + True)

True // (True + True)


I am working on the file "20181027-ccc-or-operator.ipynb".
The first task is:
Zach reported that the "or" operator in Javascript works like Python.

This presentation concentrated on the "or" operator.
The "and"
operator works like you (should) expect.
Explore it on your own.
Can you write Python code for it?


I am working on the file "20181127-top-hashtags-json.ipynb".
The first task is:
Let's see how many hashtags we extracted:
Can you write Python code for it?


I am working on the file "20181127-top-hashtags-json.ipynb".
The first task is:
Personally, I have no idea what most of these hashtags are about, but this is apparently what people were tweeting about on October 31, 2018.
And as for how many unique hashtags are in this set:
Can you write Python code for it?


I am working on the file "2018_material/labs/2017-hw1.ipynb".
The first task is:
The response variable is quality.
Can you write Python code for it?

X = np.array(sample_data.iloc[:,range(1,5)])
y = np.array(sample_data.iloc[:,0])

def loo_risk(X,y,regmod):
    """
    Construct the leave-one-out square error risk for a regression model

    Input: design matrix, X, response vector, y, a regression model, regmod
    Output: scalar LOO risk
    """
    loo = LeaveOneOut()
    loo_losses = []
    for train_index, test_index in loo.split(X):
        X_train, X_test = X[train_index], X[test_index]
        y_train, y_test = y[train_index], y[test_index]
        regmod.fit(X_train,y_train)
        y_hat = regmod.predict(X_test)
        loss = np.sum((y_hat - y_test)**2)
        loo_losses.append(loss)
    return np.mean(loo_losses)

def emp_risk(X,y,regmod):
    """
    Return the empirical risk for square error loss

    Input: design matrix, X, response vector, y, a regression model, regmod
    Output: scalar empirical risk
    """
    regmod.fit(X,y)
    y_hat = regmod.predict(X)
    return np.mean((y_hat - y)**2)


I am working on the file "2018_material/labs/2017-hw2.ipynb".
The first task is:
Load the following medical dataset with 750 patients. The response variable is survival dates (Y), the predictors are 104 measurements measured at a specific time (numerical variables have been standardized).
Can you write Python code for it?

sample_data = pd.read_csv(data_dir+"/hw2.csv", delimiter=',')

sample_data.head()

sample_data.V1 = sample_data.V1.eq('Yes').mul(1)


I am working on the file "2018_material/labs/2017-hw2.ipynb".
The first task is:
The response variable is Y for 2.1-2.3 and Z for 2.4.
Can you write Python code for it?

X = np.array(sample_data.iloc[:,range(2,104)])
y = np.array(sample_data.iloc[:,0])
z = np.array(sample_data.iloc[:,1])


I am working on the file "2018_material/labs/lab1-soln.ipynb".
The first task is:
As you can see, the empirical risk is much less than the leave-one-out risk!  This can happen in more dimensions.
Nearest neighbor regression
Use the method described here: http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsRegressor.html
I have already imported the necessary module, so you just need to use the regression object (like we used LinearRegression)
Can you write Python code for it?

# knn = neighbors.KNeighborsRegressor(n_neighbors=5)


I am working on the file "2018_material/labs/lab1-soln.ipynb".
The first task is:
Exercise 1 For each k from 1 to 30 compute the nearest neighbors empirical risk and LOO risk. Plot these as a function of k and reflect on the bias-variance tradeoff here. (Hint: use the previously defined functions)
Can you write Python code for it?

LOOs = []
MSEs = []
K=30
Ks = range(1,K+1)
for k in Ks:
    knn = neighbors.KNeighborsRegressor(n_neighbors=k)
    LOOs.append(loo_risk(X,y,knn))
    MSEs.append(emp_risk(X,y,knn))

plt.plot(Ks,LOOs,'r',label="LOO risk")
plt.title("Risks for kNN Regression")
plt.plot(Ks,MSEs,'b',label="Emp risk")
plt.legend()
_ = plt.xlabel('k')


I am working on the file "2018_material/labs/lab1-soln.ipynb".
The first task is:
Here's the full dataset, and there are other columns.  I will subselect a few of them by hand.
Can you write Python code for it?

xvars = ['Self-reported health','Water quality','Quality of support network','GDP per capita']

X = np.array(full_country_stats[xvars])
y = np.array(full_country_stats['Life satisfaction'])


I am working on the file "2018_material/labs/lab1-soln.ipynb".
The first task is:
I decided to see what the performance is for k from 1 to 30.  We see that the bias does not dominate until k exceeds 17, the performance is somewhat better for k around 12. This demonstrates that you can't trust the Empirical risk, since it includes the training sample.  We can compare this LOO risk to that of linear regression (0.348) and see that it outperforms linear regression.
Exercise 2 Do the same but for the reduced predictor variables below...
Can you write Python code for it?

X1 = np.array(full_country_stats[['Self-reported health']])

LOOs = []
MSEs = []
K=30
Ks = range(1,K+1)
for k in Ks:
    knn = neighbors.KNeighborsRegressor(n_neighbors=k)
    LOOs.append(loo_risk(X1,y,knn))
    MSEs.append(emp_risk(X1,y,knn))

plt.plot(Ks,LOOs,'r',label="LOO risk")
plt.title("Risks for kNN Regression")
plt.plot(Ks,MSEs,'b',label="Emp risk")
plt.legend()
_ = plt.xlabel('k')


I am working on the file "2018_material/labs/lab1-soln.ipynb".
The first task is:
I will define the following functions to expedite the LOO risk and the Empirical risk.
Can you write Python code for it?

def loo_risk(X,y,regmod):
    """
    Construct the leave-one-out square error risk for a regression model

    Input: design matrix, X, response vector, y, a regression model, regmod
    Output: scalar LOO risk
    """
    loo = LeaveOneOut()
    loo_losses = []
    for train_index, test_index in loo.split(X):
        X_train, X_test = X[train_index], X[test_index]
        y_train, y_test = y[train_index], y[test_index]
        regmod.fit(X_train,y_train)
        y_hat = regmod.predict(X_test)
        loss = np.sum((y_hat - y_test)**2)
        loo_losses.append(loss)
    return np.mean(loo_losses)

def emp_risk(X,y,regmod):
    """
    Return the empirical risk for square error loss

    Input: design matrix, X, response vector, y, a regression model, regmod
    Output: scalar empirical risk
    """
    regmod.fit(X,y)
    y_hat = regmod.predict(X)
    return np.mean((y_hat - y)**2)

lin1 = linear_model.LinearRegression(fit_intercept=False)
print('LOO Risk: '+ str(loo_risk(X,y,lin1)))
print('Emp Risk: ' + str(emp_risk(X,y,lin1)))


I am working on the file "2019-01-10-ESRF/notebooks/01.0.ipywidgets.ipynb".
The first task is:
Interactive Jupyter widgets
Can you write Python code for it?

slider = FloatSlider(
    value=7.5,
    min=5.0,
    max=10.0,
    step=0.1,
    description='Input:',
)

slider

slider

slider.value = 9

text = FloatText(description='Value')
dlink((slider, 'value'), (text, 'value'))
text

slider


I am working on the file "2019-01-10-ESRF/notebooks/01.1.custom.ipynb".
The first task is:
An example including bidirectional communication: A Spinner Widget
Can you write Python code for it?

from traitlets import CInt

class SpinnerWidget(widgets.DOMWidget):
    _view_name = Unicode('SpinnerView').tag(sync=True)
    _view_module = Unicode('spinner').tag(sync=True)
    value = CInt().tag(sync=True)

%%javascript
requirejs.undef('spinner');

define('spinner', ["@jupyter-widgets/base"], function(widgets) {

    var SpinnerView = widgets.DOMWidgetView.extend({
        render: function() {

            var that = this;
            this.$input = $('<input />');
            this.$el.append(this.$input);
            this.$spinner = this.$input.spinner({
                change: function( event, ui ) {
                    that.handle_spin(that.$spinner.spinner('value'));
                },
                spin: function( event, ui ) {
                    //ui.value is the new value of the spinner
                    that.handle_spin(ui.value);
                }
            });

            this.value_changed();
            this.model.on('change:value', this.value_changed, this);
        },

        value_changed: function() {
            this.$spinner.spinner('value', this.model.get('value'));
        },

        handle_spin: function(value) {
            this.model.set('value', value);
            this.touch();
        },
    });

    return {
        SpinnerView: SpinnerView
    };
});


I am working on the file "2019-01-10-ESRF/notebooks/01.1.custom.ipynb".
The first task is:
Front end (JavaScript)
Models and Views
Jupyter widgets rely on Backbone.js.
Backbone.js is an MVC (model view controller) framework.
Widgets defined in the back end are automatically synchronized with generic Backbone.js models in the front end.  The traitlets are added to the front end instance automatically on first state push.  The _view_name trait that you defined earlier is used by the widget framework to create the corresponding Backbone.js view and link that view to the model.
Import  jupyter-js-widgets, define the view, implement the render method
Can you write Python code for it?

%%javascript
require.undef('hello');

define('hello', ["@jupyter-widgets/base"], function(widgets) {

    var HelloView = widgets.DOMWidgetView.extend({

        // Render the view.
        render: function() {
            this.$el.text('Hello World!');
        },
    });

    return {
        HelloView: HelloView
    };
});


I am working on the file "2019-01-10-ESRF/notebooks/01.1.custom.ipynb".
The first task is:
Making the widget stateful
Instead of displaying a static "hello world" message, we can display a string set by the back end.

First you need to add a traitlet in the back end.

(Use the name of value to stay consistent with the rest of the widget framework and to allow your widget to be used with interact.)
Can you write Python code for it?

class HelloWidget(widgets.DOMWidget):
    _view_name = Unicode('HelloView').tag(sync=True)
    _view_module = Unicode('hello').tag(sync=True)
    value = Unicode('Hello World!').tag(sync=True)

%%javascript
require.undef('hello');

define('hello', ["@jupyter-widgets/base"], function(widgets) {

    var HelloView = widgets.DOMWidgetView.extend({

        render: function() {
            this.$el.text(this.model.get('value'));
        },
    });

    return {
        HelloView : HelloView
    };
});


I am working on the file "2019-01-10-ESRF/notebooks/01.1.custom.ipynb".
The first task is:
Test
You should be able to display your widget just like any other widget now.
Can you write Python code for it?


I am working on the file "2019-01-10-ESRF/notebooks/01.1.custom.ipynb".
The first task is:
Test of the spinner widget
Can you write Python code for it?

w = SpinnerWidget(value=5)
w

w.value = 7


I am working on the file "2019-01-10-ESRF/notebooks/01.1.custom.ipynb".
The first task is:
Wiring the spinner with another widget
Can you write Python code for it?

from IPython.display import display
w1 = SpinnerWidget(value=0)
w2 = widgets.IntSlider()
display(w1,w2)

from traitlets import link
mylink = link((w1, 'value'), (w2, 'value'))


I am working on the file "2019-01-10-ESRF/notebooks/03.ipyleaflet.ipynb".
The first task is:

Repository: https://github.com/jupyter-widgets/jupyterlab-sidecar
Installation:
pip install jupyterlab_sidecar
Can you write Python code for it?

from sidecar import Sidecar
from IPython.display import display

sc = Sidecar(title='Map widget')

with sc:
    display(m)


I am working on the file "2019-01-10-ESRF/notebooks/03.ipyleaflet.ipynb".
The first task is:
Controls
Can you write Python code for it?

from ipyleaflet import Map, basemaps, basemap_to_tiles, SplitMapControl

m = Map(center=(42.6824, 365.581), zoom=5)

right_layer = basemap_to_tiles(basemaps.NASAGIBS.ModisTerraTrueColorCR, "2017-11-11")
left_layer = basemap_to_tiles(basemaps.NASAGIBS.ModisAquaBands721CR, "2017-11-11")

control = SplitMapControl(left_layer=left_layer, right_layer=right_layer)
m.add_control(control)

m


I am working on the file "2019-01-10-ESRF/notebooks/03.ipyleaflet.ipynb".
The first task is:
Heatmap layer
Can you write Python code for it?

from ipywidgets import Button, IntSlider, link
from ipyleaflet import Heatmap
from random import gauss
import time

center = (37.09, -103.66)
zoom = 5

def create_random_data(length):
    "Return a list of some random lat/lon/value triples."
    return [[gauss(center[0], 2),
             gauss(center[1], 4),
             gauss(700, 300)] for i in range(length)]

m.center = center
m.zoom = zoom

heat = Heatmap(locations=create_random_data(1000), radius=20, blur=10)
m.add_layer(heat)

def generate(_):
    heat.locations = create_random_data(1000)

button = Button(description='Generate data', button_style='success')
button.on_click(generate)
button

slider = IntSlider(min=10, max=30, value=heat.radius)
link((slider, 'value'), (heat, 'radius'))
slider


I am working on the file "2019-01-10-ESRF/notebooks/03.ipyleaflet.ipynb".
The first task is:
Plays well with other widgets libraries
Can you write Python code for it?

import numpy as np
import bqplot.pyplot as plt
from bqplot import *
from traitlets import observe
from sidecar import Sidecar
from ipywidgets import VBox, Button
from ipyleaflet import Map, Marker, Popup

axes_options = {'x': {'label': 'x'}, 'y': {'label': 'y'}}

x = np.arange(40)
y = np.cumsum(np.random.randn(2, 40), axis=1)

fig = plt.figure(animation_duration=1000)
lines = plt.plot(x=x, y=y, colors=['red', 'green'], axes_options=axes_options)

def generate(_):
    lines.y = np.cumsum(np.random.randn(2, 40), axis=1)

button = Button(description='Generate data', button_style='success')
button.on_click(generate)

box_plot = VBox([fig, button])

fig

center = (52.204793, 360.121558)

m = Map(center=center, zoom=9, close_popup_on_click=False)

marker = Marker(location=(52.1, 359.9))
m.add_layer(marker)

marker.popup = box_plot

sc = Sidecar(title='Map and bqplot')

with sc:
    display(m)


I am working on the file "2019-01-10-ESRF/notebooks/03.ipyleaflet.ipynb".
The first task is:
Velocity
Can you write Python code for it?

from ipyleaflet import Velocity
import xarray as xr

center = (0, 0)
zoom = 4

m2 = Map(center=center, zoom=zoom, interpolation='nearest', basemap=basemaps.CartoDB.DarkMatter)

sc2 = Sidecar(title='Map Velocity')

with sc2:
    display(m2)

ds = xr.open_dataset('src/wind-global.nc')
display_options = {
    'velocityType': 'Global Wind',
    'displayPosition': 'bottomleft',
    'displayEmptyString': 'No wind data'
}
wind = Velocity(data=ds,
                zonal_speed='u_wind',
                meridional_speed='v_wind',
                latitude_dimension='lat',
                longitude_dimension='lon',
                velocity_scale=0.01,
                max_velocity=20,
                display_options=display_options)
m2.add_layer(wind)


I am working on the file "2019-01-10-ESRF/notebooks/05.ipyvolume.ipynb".
The first task is:
Animations
Animations are made in the shaders
Can you write Python code for it?

stream = ipyvolume.datasets.animated_stream.fetch()

fig = ipyvolume.figure()
q = ipyvolume.quiver(*stream.data[:,0:50,:200], color="red", size=7)
ipyvolume.animation_control(q, interval=200)
ipyvolume.show()

u = np.linspace(-10, 10, 50)
x, y = np.meshgrid(u, u)
r = np.sqrt(x**2+y**2)
x = x.flatten()
y = y.flatten()
r = r.flatten()
time = np.linspace(0, np.pi*2, 50)
z = np.array([(np.cos(r + t) * np.exp(-r/5)) for t in time])
color = np.array([[np.cos(r + t), 1-np.abs(z[i]), 0.1+z[i]*0] for i, t in enumerate(time)])
size = (z+1)
color = np.transpose(color, (0, 2, 1))

ipyvolume.figure()
s = ipyvolume.scatter(x, z, y, color=color, marker="sphere")
ipyvolume.animation_control(s, interval=200)
ipyvolume.ylim(-3,3)
ipyvolume.show()

s.geo = "diamond"

s.size = 5

s.color = 1 - s.color


I am working on the file "2019-01-10-ESRF/notebooks/05.ipyvolume.ipynb".
The first task is:
Clean
Can you write Python code for it?

from ipywidgets import Widget
Widget.close_all()


I am working on the file "2019-01-10-ESRF/notebooks/06.ipywebrtc.ipynb".
The first task is:
Capture audio
Can you write Python code for it?


I am working on the file "2019-01-10-ESRF/notebooks/06.ipywebrtc.ipynb".
The first task is:
Capture image from the camera
Can you write Python code for it?


I am working on the file "2019-01-10-ESRF/notebooks/06.ipywebrtc.ipynb".
The first task is:
Capture stream from any widget
Can you write Python code for it?

import ipyvolume
from sidecar import Sidecar
import numpy as np
from ipywebrtc import WidgetStream, VideoRecorder

u = np.linspace(-10, 10, 25)
x, y = np.meshgrid(u, u)
r = np.sqrt(x**2+y**2)
x = x.flatten()
y = y.flatten()
r = r.flatten()
time = np.linspace(0, np.pi*2, 15)
z = np.array([(np.cos(r + t) * np.exp(-r/5)) for t in time])
color = np.array([[np.cos(r + t), 1-np.abs(z[i]), 0.1+z[i]*0] for i, t in enumerate(time)])
size = (z+1)
color = np.transpose(color, (0, 2, 1))

sc = Sidecar(title='ipyvolume')

with sc:
    ipyvolume.figure()
    s = ipyvolume.scatter(x, z, y, color=color, marker="sphere")
    ipyvolume.animation_control(s, interval=200)
    ipyvolume.ylim(-3,3)
    ipyvolume.show()

view_stream = WidgetStream(widget=ipyvolume.current.container)
view_stream

recorder = VideoRecorder(stream=view_stream)
recorder


I am working on the file "2019-01-10-ESRF/notebooks/06.ipywebrtc.ipynb".
The first task is:
Capture video
Can you write Python code for it?


I am working on the file "2019-01-10-ESRF/notebooks/06.ipywebrtc.ipynb".
The first task is:
Chat
Can you write Python code for it?

from ipywebrtc import chat, VideoStream, Video

room = chat(stream=VideoStream(video=Video.from_file('src/Big.Buck.Bunny.mp4')), room='QuantStack')

# chat(room='QuantStack')

image_recorder = ImageRecorder(stream=room.streams[1])
image_recorder

from ipywidgets import Image, HBox
import PIL.Image
import io
import numpy as np
from skimage.filters import sobel
from skimage.color.adapt_rgb import adapt_rgb, each_channel
from skimage import filters

out = Image()
stop_process = False

def process_image(_):
    if stop_process:
        return
    im_in = PIL.Image.open(io.BytesIO(image_recorder.image.value))
    im_array = np.array(im_in)[...,:3]
    im_array_edges = adapt_rgb(each_channel)(sobel)(im_array)
    im_array_edges = ((1-im_array_edges) * 255).astype(np.uint8)
    im_array_edges[im_array_edges < 250] = 0
    im_out = PIL.Image.fromarray(im_array_edges)
    f = io.BytesIO()
    im_out.save(f, format='png')
    out.value = f.getvalue()
    image_recorder.recording = True

image_recorder.image.observe(process_image, names=['value'])

HBox([out, image_recorder])

stop_process = True


I am working on the file "2019-01-10-ESRF/notebooks/06.ipywebrtc.ipynb".
The first task is:
Display camera video stream
Can you write Python code for it?

camera = CameraStream(constraints=
                      {'facing_mode': 'user',
                       'audio': False,
                       'video': { 'width': 640, 'height': 480 }
                       })
camera


I am working on the file "2019-03-Presentation-Micropython.ipynb".
The first task is:
Assembler

Write directly to the GPIO registers.
Can you write Python code for it?

import time, machine

blue = machine.Pin(0, machine.Pin.OUT)
loops = 200_000

@micropython.asm_thumb
def blink_me(r0):
    lsr(r0, r0, 3)
    movwt(r1, 0x60000328)
    mov(r2, 1 << 4)
    lable(loop)
    strh(r2, [r1, 0]) # High
    strh(r2, [r1, 2]) # Low
    strh(r2, [r1, 0]) # High
    strh(r2, [r1, 2]) # Low
    strh(r2, [r1, 0]) # High
    strh(r2, [r1, 2]) # Low
    strh(r2, [r1, 0]) # High
    strh(r2, [r1, 2]) # Low
    strh(r2, [r1, 0]) # High
    strh(r2, [r1, 2]) # Low
    strh(r2, [r1, 0]) # High
    strh(r2, [r1, 2]) # Low
    strh(r2, [r1, 0]) # High
    strh(r2, [r1, 2]) # Low
    strh(r2, [r1, 0]) # High
    strh(r2, [r1, 2]) # Low

# continued from above...

def time_me(func, num):
    start = time.ticks_us()
    func(num)
    end = time.ticks_us()
    diff = time.ticks_diff(end, start)
    temp = '{:5.3f} sec, {:6.3f} usec/blink, {:8.2f} kblinks/sec'
    print(temp.format(diff * 1e-6, diff / num, num / diff * 1e3))

time_me(blink_me, loops)


I am working on the file "2019-03-Presentation-Micropython.ipynb".
The first task is:
Demo time...
Simple loop with no efficiency in mind.
Can you write Python code for it?

import time, machine

blue = machine.Pin(0, machine.Pin.OUT)
loops = 200_000
start = time.ticks_us()

for i in range(loops):
    blue.on()
    blue.off()

end = time.ticks_us()
diff = time.ticks_diff(end, start)
temp = '{:5.3f} sec, {:6.3f} usec/blink, {:8.2f} kblinks/sec'
print(temp.format(diff * 1e-6, diff / loops, loops / diff * 1e3))


I am working on the file "2019-03-Presentation-Micropython.ipynb".
The first task is:
Fin
Can you write Python code for it?

help("modules")

import machine
help(machine)

def do_connect():
    import network
    sta_if = network.WLAN(network.STA_IF)
    if not sta_if.isconnected():
        print('connecting to network...')
        sta_if.active(True)
        sta_if.connect('<essid>', '<password>')
        while not sta_if.isconnected():
            pass
    print('network config:', sta_if.ifconfig())

help(sta_if)

[print(i) for i in sta_if.scan()]

help(network)

%rebootdevice


I am working on the file "2019-03-Presentation-Micropython.ipynb".
The first task is:
Getting Connected

Obtain a terminal emulater to access REPL:
Linux: mpfshell or picocom /dev/ttyUSB0 -b115200
Windows: TeraTerm
Mac: the built-in screen program

Optionally, WebREPL:
import webrepl_setup

Find your SSID

$ nmcli -f SSID,BSSID,DEVICE dev wifi
Can you write Python code for it?

import network
sta_if = network.WLAN(network.STA_IF)
sta_if.active(True)  # activate station interface
#sta_if.connect('<your ESSID>', '<your password>')
if sta_if.isconnected():
    ip_address, netmask, gateway, dns = sta_if.ifconfig()
    print('IP: {}, netmask: {}, gateway: {}, DNS: {}'.format(*sta_if.ifconfig()))
else:
    'Not Connected'

ap_if = network.WLAN(network.AP_IF)
print(ap_if.active())  # Activate access point.
ap_if.active(False)  # Disable if not using.

import socket

def http_get(url):
    _, _, host, path = url.split('/', 3)
    addr = socket.getaddrinfo(host, 80)[0][-1]
    s = socket.socket()
    s.connect(addr)
    s.send(bytes('GET /%s HTTP/1.0\r\nHost: %s\r\n\r\n' % (path, host), 'utf8'))
    while True:
        data = s.recv(100)
        if data:
            print(str(data, 'utf8'), end='')
        else:
            break
    s.close()

http_get('http://micropython.org/ks/test.html')

import machine
pins = [machine.Pin(i, machine.Pin.IN) for i in (0, 2, 4, 5, 12, 13, 14, 15)]

html = """<!DOCTYPE html>

     <title>ESP8266 Pins</title>

ESP8266 Pins

Pin	Value
 %s

"""

import socket
addr = socket.getaddrinfo('0.0.0.0', 8081)[0][-1]

s = socket.socket()
s.bind(addr)
s.listen(1)

print('listening on', addr)

while True:
    cl, addr = s.accept()
    print('client connected from', addr)
    cl_file = cl.makefile('rwb', 0)
    while True:
        line = cl_file.readline()
        if not line or line == b'\r\n':
            break
    rows = ['
%s	%d
' % (str(p), p.value()) for p in pins]
    response = html % '\n'.join(rows)
    cl.send(response)
    cl.close()


I am working on the file "2019-03-Presentation-Micropython.ipynb".
The first task is:
Micropython for Jupyter Notebook (ESP8266-specific)

https://github.com/goatchurchprime/jupyter_micropython_kernel/

If you get this error:

ModuleNotFoundError: No module named 'prompt_toolkit.formatted_text'
https://github.com/jupyter/notebook/issues/4050
Can you write Python code for it?

%serialconnect
# May need to allow permission:
#  sudo chmod 777 /dev/ttyUSB0
# Could also add your user to the dialout group
#  sudo usermod -a -G dialout your_username

%lsmagic
# Common commands:
# %rebootdevice
# %sendtofile
# %disconnect
# %sendtofile yourfilename.py

import network

wlan = network.WLAN(network.STA_IF) # create station interface
wlan.active(True)       # activate the interface
wlan.scan()             # scan for access points
wlan.isconnected()      # check if the station is connected to an AP
wlan.connect('Student', 'Improving') # connect to an AP
wlan.config('mac')      # get the interface's MAC adddress
wlan.ifconfig()         # get the interface's IP/netmask/gw/DNS addresses

ap = network.WLAN(network.AP_IF) # create access-point interface
ap.active(True)         # activate the interface
ap.config(essid='ESP-AP') # set the ESSID of the access point

# Simple demo.

import machine
from machine import Pin
import dht
from time import sleep

motion = Pin(14, Pin.IN, Pin.PULL_UP)
light = Pin(12, Pin.IN, Pin.PULL_UP)
hum = dht.DHT22(Pin(13))
blue = Pin(0, Pin.OUT)
green = Pin(4, Pin.OUT)
red = Pin(5, Pin.OUT)

def blink(led, loops=1, delay=0.25):
     for _ in range(loops):
         led.on()
         sleep(delay)
         led.off()
         sleep(delay)

def main():
     while True:
         if light.value():
             blue.on()
         else:
             blue.off()
         if motion.value():
             green.on()
         else:
             green.off()
         hum.measure()
         if hum.temperature() > 26 or hum.humidity() > 50:
             red.on()
         else:
             red.off()
         sleep(2)
main()  # To end script, interrupt kernel (or press 'Esc' twice and 'i' twice).


I am working on the file "2019-03-Presentation-Micropython.ipynb".
The first task is:
Patial loop unrolling.

Reduces the overhead of looping.
Can you write Python code for it?

import time, machine

blue = machine.Pin(0, machine.Pin.OUT)
loops = 200_000

def blink_me(num):
    num //= 8
    on = blue.on
    off = blue.off
    r = range(num)
    for i in r:
        on()
        off()
        on()
        off()
        on()
        off()
        on()
        off()
        on()
        off()
        on()
        off()
        on()
        off()
        on()
        off()

# continued from above...

def time_me(func, num):
    start = time.ticks_us()
    func(num)
    end = time.ticks_us()
    diff = time.ticks_diff(end, start)
    temp = '{:5.3f} sec, {:6.3f} usec/blink, {:8.2f} kblinks/sec'
    print(temp.format(diff * 1e-6, diff / num, num / diff * 1e3))

time_me(blink_me, loops)


I am working on the file "2019-03-Presentation-Micropython.ipynb".
The first task is:
Preload methods into local scope of function.

Further reduces lookups.
Can you write Python code for it?

import time, machine

blue = machine.Pin(0, machine.Pin.OUT)
loops = 200_000

def blink_me(num):
    for i in range(num):
        blue.on()
        blue.off()

def time_me(func, num):
    start = time.ticks_us()
    func(num)
    end = time.ticks_us()
    diff = time.ticks_diff(end, start)
    temp = '{:5.3f} sec, {:6.3f} usec/blink, {:8.2f} kblinks/sec'
    print(temp.format(diff * 1e-6, diff / num, num / diff * 1e3))

time_me(blink_me, loops)


I am working on the file "2019-03-Presentation-Micropython.ipynb".
The first task is:
Simple loop wrapped in a function.

Number of loops no longer looked up in global scope.
Can you write Python code for it?


I am working on the file "2019-03-Presentation-Micropython.ipynb".
The first task is:
Viper mode!

Writes directly to GPIO registers.
Allows direct manipulation of registers.
Can you write Python code for it?

import time, machine

blue = machine.Pin(0, machine.Pin.OUT)
loops = 200_000

@micropython.viper
def blink_me(num:int):
    num //= 8
    p = ptr32(0x60000328)
    for i in range(num)
        p[0] = 1 << 4 # High
        p[1] = 1 << 4 # Low
        p[0] = 1 << 4 # High
        p[1] = 1 << 4 # Low
        p[0] = 1 << 4 # High
        p[1] = 1 << 4 # Low
        p[0] = 1 << 4 # High
        p[1] = 1 << 4 # Low
        p[0] = 1 << 4 # High
        p[1] = 1 << 4 # Low
        p[0] = 1 << 4 # High
        p[1] = 1 << 4 # Low
        p[0] = 1 << 4 # High
        p[1] = 1 << 4 # Low
        p[0] = 1 << 4 # High
        p[1] = 1 << 4 # Low

# continued from above...

def time_me(func, num):
    start = time.ticks_us()
    func(num)
    end = time.ticks_us()
    diff = time.ticks_diff(end, start)
    temp = '{:5.3f} sec, {:6.3f} usec/blink, {:8.2f} kblinks/sec'
    print(temp.format(diff * 1e-6, diff / num, num / diff * 1e3))

time_me(blink_me, loops)


I am working on the file "2019-05-22-pydata-frankfurt/notebooks/bqplot.ipynb".
The first task is:
A simple plot with the pyplot API
Can you write Python code for it?

from bqplot import pyplot as plt

plt.figure(1)
n = 100
plt.plot(np.linspace(0.0, 10.0, n), np.cumsum(np.random.randn(n)),
         axes_options={'y': {'grid_lines': 'dashed'}})
plt.show()


I am working on the file "2019-05-22-pydata-frankfurt/notebooks/bqplot.ipynb".
The first task is:
Every component of the figure is an independent widget
Can you write Python code for it?

xs = bq.LinearScale()
ys = bq.LinearScale()
x = np.arange(100)
y = np.cumsum(np.random.randn(2, 100), axis=1) #two random walks

line = bq.Lines(x=x, y=y, scales={'x': xs, 'y': ys}, colors=['red', 'green'])
xax = bq.Axis(scale=xs, label='x', grid_lines='solid')
yax = bq.Axis(scale=ys, orientation='vertical', tick_format='0.2f', label='y', grid_lines='solid')

fig = bq.Figure(marks=[line], axes=[xax, yax], animation_duration=1000)
display(fig)

# update data of the line mark
line.y = np.cumsum(np.random.randn(2, 100), axis=1)

xs = bq.LinearScale()
ys = bq.LinearScale()
x, y = np.random.rand(2, 20)
scatt = bq.Scatter(x=x, y=y, scales={'x': xs, 'y': ys}, default_colors=['blue'])
xax = bq.Axis(scale=xs, label='x', grid_lines='solid')
yax = bq.Axis(scale=ys, orientation='vertical', tick_format='0.2f', label='y', grid_lines='solid')

fig = bq.Figure(marks=[scatt], axes=[xax, yax], animation_duration=1000)
display(fig)

#data updates
scatt.x = np.random.rand(20) * 10
scatt.y = np.random.rand(20)


I am working on the file "2019-05-22-pydata-frankfurt/notebooks/bqplot.ipynb".
The first task is:
Handdraw
Can you write Python code for it?

handdraw = bq.interacts.HandDraw(lines=line)
fig.interaction = handdraw

line.y[0]


I am working on the file "2019-05-22-pydata-frankfurt/notebooks/bqplot.ipynb".
The first task is:
Histogram
Can you write Python code for it?

plt.figure()
plt.hist(y_data, colors=['OrangeRed'])
plt.show()


I am working on the file "2019-05-22-pydata-frankfurt/notebooks/bqplot.ipynb".
The first task is:
Selections
Can you write Python code for it?

def interval_change_callback(change):
    db.value = str(change['new'])

intsel = bq.interacts.FastIntervalSelector(scale=xs, marks=[line])
intsel.observe(interval_change_callback, names=['selected'] )

db = widgets.Label()
db.value = str(intsel.selected)
display(db)

fig = bq.Figure(marks=[line], axes=[xax, yax], animation_duration=1000, interaction=intsel)
display(fig)

line.selected


I am working on the file "2019-05-22-pydata-frankfurt/notebooks/bqplot.ipynb".
The first task is:
The same holds for the attributes of scales, axes
Can you write Python code for it?

xs.min = 4

xs.min = None

xax.label = 'Some label for the x axis'


I am working on the file "2019-05-22-pydata-frankfurt/notebooks/bqplot.ipynb".
The first task is:
Use bqplot figures as input widgets
Can you write Python code for it?

xs = bq.LinearScale()
ys = bq.LinearScale()
x = np.arange(100)
y = np.cumsum(np.random.randn(2, 100), axis=1) #two random walks

line = bq.Lines(x=x, y=y, scales={'x': xs, 'y': ys}, colors=['red', 'green'])
xax = bq.Axis(scale=xs, label='x', grid_lines='solid')
yax = bq.Axis(scale=ys, orientation='vertical', tick_format='0.2f', label='y', grid_lines='solid')


I am working on the file "2019-06-04-deRSE19-widgets/notebooks/1 - ipywidgets.ipynb".
The first task is:
Clean
Can you write Python code for it?

from ipywidgets import Widget
Widget.close_all()


I am working on the file "2019-06-04-deRSE19-widgets/notebooks/1 - ipywidgets.ipynb".
The first task is:
Link two widgets
Can you write Python code for it?

from ipywidgets import IntText, IntSlider, link, HBox

text = IntText()

slider = IntSlider()

link((text, 'value'), (slider, 'value'))
HBox([text, slider])


I am working on the file "2019-06-04-deRSE19-widgets/notebooks/1 - ipywidgets.ipynb".
The first task is:
Media widgets
Image widget
Can you write Python code for it?

from ipywidgets import Image

import PIL.Image
import io
import numpy as np
from skimage.filters import sobel
from skimage.color.adapt_rgb import adapt_rgb, each_channel
from skimage import filters

image = Image.from_file("src/marie.png")
image

im_in = PIL.Image.open(io.BytesIO(image.value))
im_array = np.array(im_in)[...,:3]
im_array

im_array_edges = adapt_rgb(each_channel)(sobel)(im_array)
im_array_edges = ((1-im_array_edges) * 255).astype(np.uint8)
im_out = PIL.Image.fromarray(im_array_edges)
f = io.BytesIO()
im_out.save(f, format='png')
image.value = f.getvalue()


I am working on the file "2019-06-04-deRSE19-widgets/notebooks/1 - ipywidgets.ipynb".
The first task is:
Observe changes on the widget model
Can you write Python code for it?

from ipywidgets import ToggleButton

button = ToggleButton(description='Click me!', button_style='danger')

def update_style(change):
    button.button_style = 'info' if change['new'] else 'danger'

button.observe(update_style, 'value')
button


I am working on the file "2019-06-04-deRSE19-widgets/notebooks/1 - ipywidgets.ipynb".
The first task is:
Variety of widgets in the core library
Can you write Python code for it?

from ipywidgets import ColorPicker, DatePicker, IntProgress, Play, VBox, link

progress = IntProgress()
play = Play()
link((play, 'value'), (progress, 'value'))

VBox([ColorPicker(value='red'), DatePicker(), progress, play])


I am working on the file "2019-06-04-deRSE19-widgets/notebooks/1 - ipywidgets.ipynb".
The first task is:
Video widget
Can you write Python code for it?

from ipywidgets import Video, Image
from IPython.display import display
import numpy as np
import cv2
import base64

video = Video.from_file('src/Big.Buck.Bunny.mp4')
video

cap = cv2.VideoCapture('src/Big.Buck.Bunny.mp4')

frames = []

while(1):
    try:
        _, frame = cap.read()

        fgmask = cv2.Canny(frame, 100, 100)

        mask = fgmask > 100
        frame[mask, :] = 0

        frames.append(frame)
    except Exception:
        break

width = int(cap.get(3))
height = int(cap.get(4))

filename = 'src/output.mp4'

fourcc = cv2.VideoWriter_fourcc(*'avc1')
writer = cv2.VideoWriter(filename, fourcc, 25, (width, height))

for frame in frames:
    writer.write(frame)

cap.release()
writer.release()

with open(filename, 'rb') as f:
    video.value = f.read()


I am working on the file "2019-06-04-deRSE19-widgets/notebooks/1 - ipywidgets.ipynb".
The first task is:
Widgets protocol

Drive a boolean value
Can you write Python code for it?

from ipywidgets import Checkbox

checkbox = Checkbox(description='Check me')
checkbox

checkbox.value

checkbox.value = False


I am working on the file "2019-06-04-deRSE19-widgets/notebooks/3 - ipyleaflet.ipynb".
The first task is:
Layers
Marker
Can you write Python code for it?

from ipyleaflet import Marker, Icon

icon = Icon(icon_url='https://leafletjs.com/examples/custom-icons/leaf-red.png', icon_size=[38, 95], icon_anchor=[22,94])
mark = Marker(location=center, icon=icon, rotation_origin='22px 94px')

m.add_layer(mark)

import time

for _ in range(40):
    mark.rotation_angle += 15
    time.sleep(0.1)


I am working on the file "2019-06-04-deRSE19-widgets/notebooks/3 - ipyleaflet.ipynb".
The first task is:
Velocity
Can you write Python code for it?

from ipyleaflet import Velocity
import xarray as xr

center = (0, 0)
zoom = 4

m2 = Map(center=center, zoom=zoom, interpolation='nearest', basemap=basemaps.CartoDB.DarkMatter)
m2

ds = xr.open_dataset('src/wind-global.nc')
display_options = {
    'velocityType': 'Global Wind',
    'displayPosition': 'bottomleft',
    'displayEmptyString': 'No wind data'
}
wind = Velocity(data=ds,
                zonal_speed='u_wind',
                meridional_speed='v_wind',
                latitude_dimension='lat',
                longitude_dimension='lon',
                velocity_scale=0.01,
                max_velocity=20,
                display_options=display_options)
m2.add_layer(wind)


I am working on the file "2019-06-04-deRSE19-widgets/notebooks/5 - Custom.ipynb".
The first task is:
Dynamic updates

Adding and registering a change handler.
Can you write Python code for it?

%%javascript
require.undef('hello');

define('hello', ["@jupyter-widgets/base"], function(widgets) {

    var HelloView = widgets.DOMWidgetView.extend({

        render: function() {
            this.value_changed();
            this.model.on('change:value', this.value_changed, this);
        },

        value_changed: function() {
            this.el.innerText = this.model.get('value');
        },
    });

    return {
        HelloView : HelloView
    };
});

w = HelloWidget()
w

w.value = 'Hello!'


I am working on the file "2019-06-04-deRSE19-widgets/notebooks/5 - Custom.ipynb".
The first task is:
Front end (JavaScript)
Models and Views
Jupyter widgets rely on Backbone.js.
Backbone.js is an MVC (model view controller) framework.
Widgets defined in the back end are automatically synchronized with generic Backbone.js models in the front end.  The traitlets are added to the front end instance automatically on first state push.  The _view_name trait that you defined earlier is used by the widget framework to create the corresponding Backbone.js view and link that view to the model.
Import  jupyter-js-widgets, define the view, implement the render method
Can you write Python code for it?

%%javascript
require.undef('hello');

define('hello', ["@jupyter-widgets/base"], function(widgets) {

    var HelloView = widgets.DOMWidgetView.extend({

        // Render the view.
        render: function() {
            console.log(this)
            this.el.innerText = 'Hello World!';
        },
    });

    return {
        HelloView: HelloView
    };
});


I am working on the file "2019-06-04-deRSE19-widgets/notebooks/5 - Custom.ipynb".
The first task is:
Making the widget stateful
Instead of displaying a static "hello world" message, we can display a string set by the back end.

First you need to add a traitlet in the back end.

(Use the name of value to stay consistent with the rest of the widget framework and to allow your widget to be used with interact.)
Can you write Python code for it?

class HelloWidget(widgets.DOMWidget):
    _view_name = Unicode('HelloView').tag(sync=True)
    _view_module = Unicode('hello').tag(sync=True)
    value = Unicode('Hello World!').tag(sync=True)

%%javascript
require.undef('hello');

define('hello', ["@jupyter-widgets/base"], function(widgets) {

    var HelloView = widgets.DOMWidgetView.extend({

        render: function() {
            this.el.innerText  = this.model.get('value');
        },
    });

    return {
        HelloView : HelloView
    };
});


I am working on the file "2019-06-04-deRSE19-widgets/notebooks/5 - Custom.ipynb".
The first task is:
Test
You should be able to display your widget just like any other widget now.
Can you write Python code for it?


I am working on the file "2019-06-04-deRSE19-widgets/notebooks/5 - Custom.ipynb".
The first task is:
Wiring the spinner with another widget
Can you write Python code for it?

from IPython.display import display
w1 = SpinnerWidget(value=0)
w2 = widgets.IntSlider()
display(w1,w2)

from traitlets import link
mylink = link((w1, 'value'), (w2, 'value'))


I am working on the file "2019-06-09-Presentation-Jupyter-Notebook.ipynb".
The first task is:
Bash
Can you write Python code for it?

!dir

%%bash
echo "My shell is:" $SHELL
echo "My disk usage is:"
df -h

%%perl
@months = ("July", "August", "September");
print $months[0];

%%ruby
name = "world"
puts "Hello #{name.capitalize}!"


I am working on the file "2019-06-09-Presentation-Jupyter-Notebook.ipynb".
The first task is:
Configuration / Environment
Can you write Python code for it?

!ipython profile locate

# Otherwise run: !ipython profile create

# Sample output:

# [ProfileCreate] Generating default config file: 'C:\\Users\\randall.white\\.ipython\\profile_default\\ipython_config.py'
# [ProfileCreate] Generating default config file: 'C:\\Users\\randall.white\\.ipython\\profile_default\\ipython_kernel_config.py'


I am working on the file "2019-06-09-Presentation-Jupyter-Notebook.ipynb".
The first task is:
Embed YouTube videos.
Can you write Python code for it?

from IPython.display import YouTubeVideo
YouTubeVideo('HW29067qVWk')


I am working on the file "2019-06-09-Presentation-Jupyter-Notebook.ipynb".
The first task is:
Magic

% or %%
debug, time, timeit, run, and more...
Can you write Python code for it?

%magic

%quickref

%time some_list = [x**x for x in range(9001)]

!sudo python3.6 -m pip install matplotlib

%matplotlib notebook
import matplotlib, numpy, pandas, seaborn
from matplotlib import pyplot as plt

ts = pandas.Series(numpy.random.randn(1000), index=pandas.date_range('1/1/2000', periods=1000))
ts = ts.cumsum()
df = pandas.DataFrame(numpy.random.randn(1000, 4), index=ts.index,
                  columns=['A', 'B', 'C', 'D'])
df = df.cumsum()
df.plot(); plt.legend(loc='best')


I am working on the file "2019-06-26-GeoPython/notebooks/2.ipyleaflet.ipynb".
The first task is:
Controls
SlipMap Control
Can you write Python code for it?

from ipyleaflet import Map, basemaps, basemap_to_tiles, SplitMapControl

m = Map(center=(42.6824, 365.581), zoom=5)

right_layer = basemap_to_tiles(basemaps.NASAGIBS.ModisTerraTrueColorCR, "2017-11-11")
left_layer = basemap_to_tiles(basemaps.NASAGIBS.ModisAquaBands721CR, "2017-11-11")

control = SplitMapControl(left_layer=left_layer, right_layer=right_layer)
m.add_control(control)

m


I am working on the file "2019-06-26-GeoPython/notebooks/2.ipyleaflet.ipynb".
The first task is:
JupyterLab Themes Support
Widget Control
Can you write Python code for it?

from ipywidgets import IntSlider, ColorPicker, Button, link
from ipyleaflet import Map, basemaps, WidgetControl

m = Map(center=(46.01, 6.16), zoom=12, basemap=basemaps.Stamen.Terrain)
m

zoom_slider = IntSlider(description='Zoom level:', min=0, max=15, value=7)

link((zoom_slider, 'value'), (m, 'zoom'))
widget_control1 = WidgetControl(widget=zoom_slider, position='topright')

m.add_control(widget_control1)

color_picker = ColorPicker(description='Pick a color:')

widget_control2 = WidgetControl(widget=color_picker, position='bottomright')

m.add_control(widget_control2)

minimap = Map(
    zoom_control=False, attribution_control=False,
    zoom=5, center=m.center, basemap=basemaps.Stamen.Terrain
)

minimap.layout.width = '150px'
minimap.layout.height = '150px'

link((minimap, 'center'), (m, 'center'))
minimap_control = WidgetControl(widget=minimap, position='bottomleft')

m.add_control(minimap_control)


I am working on the file "2019-06-26-GeoPython/notebooks/2.ipyleaflet.ipynb".
The first task is:
Layers
ipyleaflet provides a set of Layers and Controls that can be dynamically created and added/removed from the Map. You can find more details in the documentation: https://ipyleaflet.readthedocs.io
Marker Layer
Can you write Python code for it?

from ipyleaflet import Marker, Icon

icon = Icon(icon_url='https://leafletjs.com/examples/custom-icons/leaf-red.png', icon_size=[38, 95], icon_anchor=[22,94])
mark = Marker(location=center, icon=icon, rotation_origin='22px 94px')

m.add_layer(mark)

import time

for _ in range(500):
    mark.rotation_angle += 1
    time.sleep(0.01)


I am working on the file "2019-06-26-GeoPython/notebooks/vaex.ipynb".
The first task is:
A Billion stars in the Jupyter notebook
Can you write Python code for it?

import vaex
#gaia = vaex.open("ws://gaia:9000/gaia-dr1")
gaia = vaex.open('/Users/maartenbreddels/datasets/gaia/gaia-dr1-minimal_f4.hdf5')
%matplotlib inline

f"{len(gaia):,}"

ra_dec_limits = [[0, 360], [-90, 90]]

gaia.set_active_fraction(0.01)

gaia.plot_widget("ra", "dec", limits=ra_dec_limits)

gaia.mean("phot_g_mean_mag", selection=True)

gaia.plot1d("phot_g_mean_mag", selection=False, n=True, limits=[10, 22])
gaia.plot1d("phot_g_mean_mag", selection=True, show=True, n=True, limits=[10, 22])


I am working on the file "2019-06-26-GeoPython/notebooks/vaex.ipynb".
The first task is:
ipyvolume

3d plotting for Python in the Jupyter notebook based on IPython widgets using WebGL
Glyphs, volume rendering, surfaces/meshes/lines/isosurfaces
Live documentation http://ipyvolume.readthedocs.io/en/stable/
Installation
$ conda install -c conda-forge ipyvolume
$ pip install ipyvolume
Can you write Python code for it?

import ipyvolume as ipv
import numpy as np
np.warnings.filterwarnings('ignore')

ipv.example_ylm();

N = 1000
x, y, z = np.random.random((3, N))

fig = ipv.figure()
scatter = ipv.scatter(x, y, z, marker='box')
ipv.show()

scatter.x = scatter.x + 0.1

scatter.color = "green"
scatter.size = 5

scatter.color = np.random.random((N,3))

scatter.size = 2

ipv.figure()
ipv.style.use('dark')
quiver = ipv.quiver(*ipv.datasets.animated_stream.fetch().data[:,::,::4], size=5)
ipv.animation_control(quiver, interval=200)
ipv.show()

ipv.style.use('light')

quiver.size = np.random.random(quiver.x.shape) * 10

quiver.color = np.random.random(quiver.x.shape + (3,))

quiver.geo = "cat"

# stereo

quiver.geo = "arrow"

N = 1000*1000
x, y, z = np.random.random((3, N)).astype('f4')

ipv.figure()
s = ipv.scatter(x, y, z, size=0.2)
ipv.show()

s.size = 0.1

#ipv.screenshot(width=2048, height=2048)

plot3d = ds.plot_widget("x", "y", "z", vx="vx", vy="vy", vz="vz",
                        backend="ipyvolume", f="log1p", shape=100, smooth_pre=0.5)

plot3d.vcount_limits = [50, 100000]

plot3d.backend.quiver.color = "red"

import ipywidgets as widgets

widgets.ColorPicker()

widgets.jslink((plot3d.backend.quiver, 'color'), (_, 'value'))

ipv.save("kapteyn-lunch-talk-2018.html")

!open kapteyn-lunch-talk-2018.html

# webrtc demo if time permits


I am working on the file "2019-07-10-CICM/notebooks/DrawControl.ipynb".
The first task is:
In addition, the DrawControl also has last_action and last_draw attributes that are created dynamicaly anytime a new drawn path arrives.
Can you write Python code for it?


I am working on the file "2019-07-10-CICM/notebooks/DrawControl.ipynb".
The first task is:
Let's draw a second map and try to import this GeoJSON data into it.
Can you write Python code for it?

m2 = Map(center=center, zoom=zoom, layout=dict(width='600px', height='400px'))
m2


I am working on the file "2019-07-10-CICM/notebooks/DrawControl.ipynb".
The first task is:
Note that the style is preserved! If you wanted to change the style, you could edit the properties.style dictionary of the GeoJSON data. Or, you could even style the original path in the DrawControl by setting the polygon dictionary of that object. See the code for details.
Now let's add a DrawControl to this second map. For fun we will disable lines and enable circles as well and change the style a bit.
Can you write Python code for it?

dc2 = DrawControl(polygon={'shapeOptions': {'color': '#0000FF'}}, polyline={},
                  circle={'shapeOptions': {'color': '#0000FF'}})
m2.add_control(dc2)


I am working on the file "2019-07-10-CICM/notebooks/DrawControl.ipynb".
The first task is:
Now create the DrawControl and add it to the Map using add_control. We also register a handler for draw events. This will fire when a drawn path is created, edited or deleted (there are the actions). The  geo_json argument is the serialized geometry of the drawn path, along with its embedded style.
Can you write Python code for it?

dc = DrawControl(marker={'shapeOptions': {'color': '#0000FF'}},
                 rectangle={'shapeOptions': {'color': '#0000FF'}},
                 circle={'shapeOptions': {'color': '#0000FF'}},
                 circlemarker={},
                 )

def handle_draw(self, action, geo_json):
    print(action)
    print(geo_json)

dc.on_draw(handle_draw)
m.add_control(dc)


I am working on the file "2019-07-10-CICM/notebooks/DrawControl.ipynb".
The first task is:
We can use link to synchronize traitlets of the two maps:
Can you write Python code for it?

map_center_link = link((m, 'center'), (m2, 'center'))
map_zoom_link = link((m, 'zoom'), (m2, 'zoom'))

new_poly = GeoJSON(data=dc.last_draw)

m2.add_layer(new_poly)


I am working on the file "2019-07-10-CICM/notebooks/bqplot.ipynb".
The first task is:
A simple plot with the pyplot API
Can you write Python code for it?

from bqplot import pyplot as plt

plt.figure(1)
n = 100
plt.plot(np.linspace(0.0, 10.0, n), np.cumsum(np.random.randn(n)),
         axes_options={'y': {'grid_lines': 'dashed'}})
plt.show()


I am working on the file "2019-07-10-CICM/notebooks/bqplot.ipynb".
The first task is:
Every component of the figure is an independent widget
Can you write Python code for it?

xs = bq.LinearScale()
ys = bq.LinearScale()
x = np.arange(100)
y = np.cumsum(np.random.randn(2, 100), axis=1) #two random walks

line = bq.Lines(x=x, y=y, scales={'x': xs, 'y': ys}, colors=['red', 'green'])
xax = bq.Axis(scale=xs, label='x', grid_lines='solid')
yax = bq.Axis(scale=ys, orientation='vertical', tick_format='0.2f', label='y', grid_lines='solid')

fig = bq.Figure(marks=[line], axes=[xax, yax], animation_duration=1000)
display(fig)

# update data of the line mark
line.y = np.cumsum(np.random.randn(2, 100), axis=1)

xs = bq.LinearScale()
ys = bq.LinearScale()
x, y = np.random.rand(2, 20)
scatt = bq.Scatter(x=x, y=y, scales={'x': xs, 'y': ys}, default_colors=['blue'])
xax = bq.Axis(scale=xs, label='x', grid_lines='solid')
yax = bq.Axis(scale=ys, orientation='vertical', tick_format='0.2f', label='y', grid_lines='solid')

fig = bq.Figure(marks=[scatt], axes=[xax, yax], animation_duration=1000)
display(fig)

#data updates
scatt.x = np.random.rand(20) * 10
scatt.y = np.random.rand(20)


I am working on the file "2019-07-10-CICM/notebooks/bqplot.ipynb".
The first task is:
Histogram
Can you write Python code for it?

plt.figure()
plt.hist(y_data, colors=['OrangeRed'])
plt.show()