I am working on the file "2018-06-18-IFPEN-Tech/notebooks/03 - Custom.ipynb".
The first task is:
An example including bidirectional communication: A Spinner Widget
Can you write Python code for it?

from traitlets import CInt

class SpinnerWidget(widgets.DOMWidget):
    _view_name = Unicode('SpinnerView').tag(sync=True)
    _view_module = Unicode('spinner').tag(sync=True)
    value = CInt().tag(sync=True)

%%javascript
requirejs.undef('spinner');

define('spinner', ["@jupyter-widgets/base"], function(widgets) {

    var SpinnerView = widgets.DOMWidgetView.extend({
        render: function() {

            var that = this;
            this.$input = $('<input />');
            this.$el.append(this.$input);
            this.$spinner = this.$input.spinner({
                change: function( event, ui ) {
                    that.handle_spin(that.$spinner.spinner('value'));
                },
                spin: function( event, ui ) {
                    //ui.value is the new value of the spinner
                    that.handle_spin(ui.value);
                }
            });

            this.value_changed();
            this.model.on('change:value', this.value_changed, this);
        },

        value_changed: function() {
            this.$spinner.spinner('value', this.model.get('value'));
        },

        handle_spin: function(value) {
            this.model.set('value', value);
            this.touch();
        },
    });

    return {
        SpinnerView: SpinnerView
    };
});


I am working on the file "2018-06-18-IFPEN-Tech/notebooks/03 - Custom.ipynb".
The first task is:
Dynamic updates

Adding and registering a change handler.
Can you write Python code for it?

%%javascript
require.undef('hello');

define('hello', ["@jupyter-widgets/base"], function(widgets) {

    var HelloView = widgets.DOMWidgetView.extend({

        render: function() {
            this.value_changed();
            this.model.on('change:value', this.value_changed, this);
        },

        value_changed: function() {
            this.$el.text(this.model.get('value'));
        },
    });

    return {
        HelloView : HelloView
    };
});

w = HelloWidget()
w

w.value = 'test'


I am working on the file "2018-06-18-IFPEN-Tech/notebooks/03 - Custom.ipynb".
The first task is:
Front end (JavaScript)
Models and Views
Jupyter widgets rely on Backbone.js.
Backbone.js is an MVC (model view controller) framework.
Widgets defined in the back end are automatically synchronized with generic Backbone.js models in the front end.  The traitlets are added to the front end instance automatically on first state push.  The _view_name trait that you defined earlier is used by the widget framework to create the corresponding Backbone.js view and link that view to the model.
Import  jupyter-js-widgets, define the view, implement the render method
Can you write Python code for it?

%%javascript
require.undef('hello');

define('hello', ["@jupyter-widgets/base"], function(widgets) {

    var HelloView = widgets.DOMWidgetView.extend({

        // Render the view.
        render: function() {
            this.$el.text('Hello World!');
        },
    });

    return {
        HelloView: HelloView
    };
});


I am working on the file "2018-06-18-IFPEN-Tech/notebooks/03 - Custom.ipynb".
The first task is:
Making the widget stateful
Instead of displaying a static "hello world" message, we can display a string set by the back end.

First you need to add a traitlet in the back end.

(Use the name of value to stay consistent with the rest of the widget framework and to allow your widget to be used with interact.)
Can you write Python code for it?

class HelloWidget(widgets.DOMWidget):
    _view_name = Unicode('HelloView').tag(sync=True)
    _view_module = Unicode('hello').tag(sync=True)
    value = Unicode('Hello World!').tag(sync=True)

%%javascript
require.undef('hello');

define('hello', ["@jupyter-widgets/base"], function(widgets) {

    var HelloView = widgets.DOMWidgetView.extend({

        render: function() {
            this.$el.text(this.model.get('value'));
        },
    });

    return {
        HelloView : HelloView
    };
});


I am working on the file "2018-06-18-IFPEN-Tech/notebooks/03 - Custom.ipynb".
The first task is:
Test
You should be able to display your widget just like any other widget now.
Can you write Python code for it?


I am working on the file "2018-06-18-IFPEN-Tech/notebooks/08 - PyThreejs.ipynb".
The first task is:
Lines
Can you write Python code for it?

# On windows, linewidth of the material has no effect
size = 4
linesgeom = Geometry(vertices=[[0, 0, 0],
                                    [size, 0, 0],
                                    [0, 0, 0],
                                    [0, size, 0],
                                    [0, 0, 0],
                                    [0, 0, size]],
                          colors = ['red', 'red', 'green', 'green', 'white', 'orange'])
lines = Line(geometry=linesgeom,
             material=LineBasicMaterial(linewidth=5, vertexColors='VertexColors'),
             type='LinePieces',
            )
scene = Scene(children=[
    lines,
    DirectionalLight(color='#ccaabb', position=[0,10,0]),
    AmbientLight(color='#cccccc'),
    ])
c = PerspectiveCamera(position=[10, 10, 10])
renderer = Renderer(camera=c, background='black', background_opacity=1, scene=scene, controls=[OrbitControls(controlling=c)],
                    width=400, height=400)
display(renderer)


I am working on the file "2018-06-18-IFPEN-Tech/notebooks/08 - PyThreejs.ipynb".
The first task is:
Simple sphere and text
Can you write Python code for it?

ball = Mesh(geometry=SphereGeometry(radius=1),
            material=MeshLambertMaterial(color='red'),
            position=[2, 1, 0])

c = PerspectiveCamera(position=[0, 5, 5], up=[0, 1, 0],
                      children=[DirectionalLight(color='white', position=[3, 5, 1], intensity=0.5)])

scene = Scene(children=[ball, c, AmbientLight(color='#777777')])

renderer = Renderer(camera=c,
                    scene=scene,
                    controls=[OrbitControls(controlling=c)])
display(renderer)

ball.geometry.radius = 0.5

import time, math
ball.material.color = '#4400dd'
for i in range(1, 150, 2):
    ball.geometry.radius = i / 100.
    ball.position = [math.cos(i / 10.), math.sin(i / 50.), i / 100.]
    time.sleep(.05)


I am working on the file "2018-06-18-IFPEN-Tech/notebooks/09 - LeafletDraw.ipynb".
The first task is:
Let's draw a second map and try to import this GeoJSON data into it.
Can you write Python code for it?


I am working on the file "2018-06-18-IFPEN-Tech/notebooks/09 - LeafletDraw.ipynb".
The first task is:
Note that the style is preserved! If you wanted to change the style, you could edit the properties.style dictionary of the GeoJSON data. Or, you could even style the original path in the DrawControl by setting the polygon dictionary of that object. See the code for details.
Now let's add a DrawControl to this second map. For fun we will disable lines and enable circles as well and change the style a bit.
Can you write Python code for it?

dc2 = DrawControl(polygon={'shapeOptions': {'color': '#0000FF'}}, polyline={},
                  circle={'shapeOptions': {'color': '#0000FF'}})
m2.add_control(dc2)


I am working on the file "2018-06-18-IFPEN-Tech/notebooks/09 - LeafletDraw.ipynb".
The first task is:
Now create the DrawControl and add it to the Map using add_control. We also register a handler for draw events. This will fire when a drawn path is created, edited or deleted (there are the actions). The  geo_json argument is the serialized geometry of the drawn path, along with its embedded style.
Can you write Python code for it?

dc = DrawControl()

def handle_draw(self, action, geo_json):
    print(action)
    print(geo_json)

dc.on_draw(handle_draw)
m.add_control(dc)


I am working on the file "2018-06-18-IFPEN-Tech/notebooks/09 - LeafletDraw.ipynb".
The first task is:
We can use link to synchronize traitlets of the two maps:
Can you write Python code for it?

map_center_link = link((m, 'center'), (m2, 'center'))
map_zoom_link = link((m, 'zoom'), (m2, 'zoom'))

new_poly = GeoJSON(data=dc.last_draw)

m2.add_layer(new_poly)


I am working on the file "2018-06-18-IFPEN-Tech/notebooks/13 - ipysheet.ipynb".
The first task is:
If flexx is installed, Python code can be transpiled to JavaScript at runtime.
Can you write Python code for it?

def renderer_negative(instance, td, row, col, prop, value, cellProperties):
    Handsontable.renderers.TextRenderer.apply(this, arguments);
    if value < 0:
        td.style.backgroundColor = 'orange'
    else:
        td.style.backgroundColor = ''
ipysheet.renderer(code=renderer_negative, name='negative_transpiled');

import random
s = ipysheet.sheet(rows=3, columns=4)
data = [[random.randint(-10, 10) for j in range(4)] for j in range(3)]
ipysheet.cell_range(data, renderer='negative_transpiled')
s


I am working on the file "2018-06-18-IFPEN-Tech/notebooks/13 - ipysheet.ipynb".
The first task is:
Renderers
ipysheet is build on Handsontable, which allows custom renderers, which we also support. Note that this means ipysheet allows arbitrary JavaScript injection (TODO: make this part optional)
Can you write Python code for it?

jscode_renderer_negative = """
function (instance, td, row, col, prop, value, cellProperties) {
    Handsontable.renderers.TextRenderer.apply(this, arguments);
    if (value < 0)
        td.style.backgroundColor = 'red'
    else
        td.style.backgroundColor = 'green'
}
"""
ipysheet.renderer(code=jscode_renderer_negative, name='negative');

import random
s = ipysheet.sheet(rows=3, columns=4)
data = [[random.randint(-10, 10) for j in range(4)] for j in range(3)]
ipysheet.cell_range(data, renderer='negative')
s


I am working on the file "2018-06-18-IFPEN-Tech/notebooks/13 - ipysheet.ipynb".
The first task is:
Using the cell function, we can create Cell widgets that are directly added to the current sheet.
Can you write Python code for it?

sheet = ipysheet.sheet(rows=3, columns=4)
cell1 = ipysheet.cell(0, 0, 'Hello')
cell2 = ipysheet.cell(2, 0, 'World')
cell_value = ipysheet.cell(2,2, 42.)
sheet


I am working on the file "2018-06-18-IFPEN-Tech/notebooks/vaex-widget.ipynb".
The first task is:
Bqplot

Interactive plotting in the Jupyter notebook
Documentation: https://bqplot.readthedocs.io
Github: https://github.com/bloomberg/bqplot
Can you write Python code for it?

import bqplot.pyplot as plt
import numpy as np

x = np.linspace(0, 2, 50)
y = x**2

fig = plt.figure()
scatter = plt.scatter(x, y)
plt.show()

fig.animation_duration = 500
scatter.y = x**0.5

scatter.selected_style = {'stroke':'red', 'fill': 'orange'}
plt.brush_selector();

scatter.selected

scatter.selected = [1,2,10,40]


I am working on the file "2018-06-18-IFPEN-Tech/notebooks/vaex-widget.ipynb".
The first task is:
Integration with vaex
Can you write Python code for it?

import vaex
import numpy as np
np.warnings.filterwarnings('ignore')
dstaxi = vaex.open('nyc_taxi2015.hdf5') # mmapped, doesn't cost extra memory

dstaxi.plot_widget("pickup_longitude", "pickup_latitude", f="log", backend="ipyleaflet", shape=600)

dstaxi.plot_widget("dropoff_longitude", "dropoff_latitude", f="log", backend="ipyleaflet",
                   z="dropoff_hour", type="slice", z_shape=24, shape=400, z_relative=True,
                   limits=[None, None, (-0.5, 23.5)])

ds = vaex.open('helmi-dezeeuw-2000-FeH.hdf5')

ds.plot_widget("x", "y", f="log", limits=[-20, 20])

ds.plot_widget("Lz", "E", f="log")


I am working on the file "2018-06-18-IFPEN-Tech/notebooks/vaex.ipynb".
The first task is:
0 dimensional
Can you write Python code for it?

ds.count()

ds.count(ds.pickup_latitude)

ds.mean(ds.pickup_latitude)


I am working on the file "2018-06-18-IFPEN-Tech/notebooks/vaex.ipynb".
The first task is:
1 dimensional
Can you write Python code for it?

ds.count(binby=ds.pickup_latitude, limits=[40.5, 41])

plt.plot(_)


I am working on the file "2018-06-18-IFPEN-Tech/notebooks/vaex.ipynb".
The first task is:
2 dimensional
Can you write Python code for it?

counts2d = ds.count(binby=[ds.pickup_longitude, ds.pickup_latitude], shape=128)
print(counts2d.shape)

plt.imshow(np.log10(counts2d+1).T, origin='lower')

limits = ds.limits([ds.pickup_longitude, ds.pickup_latitude], "98%")
limits

%%time
ds.plot(ds.pickup_longitude, ds.pickup_latitude, f="log1p",
        limits=limits, figsize=(10,8), shape=512, colormap="viridis")

# %%timeit
# counts2d = ds.count(binby=["pickup_longitude", "pickup_latitude"], shape=128, limits=limits)#, limits=[[-90, 90], [-180, 180]])


I am working on the file "2018-06-18-IFPEN-Tech/notebooks/vaex.ipynb".
The first task is:
Lazy expressions and virtual columns
Can you write Python code for it?

# do not do
#ratio = ds.data.total_amount/ds.data.trip_distance
print(len(ds.data.total_amount) * 8 / 1024**3, "GB")

ds.total_amount/ds.trip_distance

#ds.add_virtual_column("ratio", "total_amount/trip_distance")
ds['ratio'] = ds.total_amount / ds.trip_distance

ds.mean(ds.ratio), ds.ratio.mean()

def arc_distance(theta_1, phi_1, theta_2, phi_2):
    temp = (np.sin((theta_2-theta_1)/2*np.pi/180)**2
           + np.cos(theta_1*np.pi/180)*np.cos(theta_2*np.pi/180) * np.sin((phi_2-phi_1)/2*np.pi/180)**2)
    distance = 2 * np.arctan2(np.sqrt(temp), np.sqrt(1-temp))
    return distance * 6400

arc_distance(0, 0, 0, 180)

ds["arc_distance"] = arc_distance(ds.pickup_longitude,  ds.pickup_latitude,
                                ds.dropoff_longitude, ds.dropoff_latitude)

%%time
ds.arc_distance.mean()

ds['arc_distance_jit'] = ds.arc_distance.jit_numba()
# ds['arc_distance_jit'] = ds.arc_distance.jit_pythran()

%%time
ds.arc_distance_jit.mean()

ds['extra'] = (ds.trip_distance*1.6 - ds.arc_distance_jit)

ds.select(ds.pickup_longitude != ds.dropoff_longitude)

ds.extra.mean(selection=True), ds.extra.minmax(selection=True)

ds.plot1d(ds.extra, selection=True, limits=[-5, 10])

ds.plot(ds.pickup_longitude, ds.pickup_latitude, what=vaex.stat.mean(ds.extra),
       selection=True, vmin=0, vmax=3,
       shape=512, figsize=(10,8), limits=limits, colormap="Greys")


I am working on the file "2018-06-24-sklearn-KDDCUP99.ipynb".
The first task is:
Preprocessing Data
Can you write Python code for it?

X = df.values

le = preprocessing.LabelEncoder()
y = le.fit_transform(dataset_part.target)
y_dict = dict(zip(y,le.classes_))  # Saved for later lookup.

# Test options and evaluation metric
N_SPLITS = 7
SCORING = 'accuracy'

# Split-out validation dataset
test_size=0.33
SEED = 42
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=test_size, random_state=SEED)


I am working on the file "2018-06-24-sklearn-KDDCUP99.ipynb".
The first task is:
Sources

[1] - KDD Cup 99 dataset
[2] - M. Tavallaee, E. Bagheri, W. Lu, and A. Ghorbani, “A Detailed Analysis of the KDD CUP 99 Data Set,” Submitted to Second IEEE Symposium on Computational Intelligence for Security and Defense Applications (CISDA), 2009. link

Other Resources

PySpark solution to the KDDCup99

link

Logs

Public PCAP files for PCAP-based evaluation of network-based intrusion detection system (NIDS) evaluation.
The Cyber Research Center - DataSets - ITOC CDX (2009)

Labelled datasets

UNB ISCX (2012-) datasets contain a range of "sophisticated" intrusion attacks, botnets and DoS attacks.
CSIC 2010 HTTP Dataset in CSV format (for Weka Analysis) dataset is from a web penetration testing testbed for anomaly detection training.
Attack Challenge - ECML/PKDD Workshop (2007) dataset contains web penetration testing data.
NSL-KDD Data Set (2007) intended to replace the DARPA KDDCup99 dataset for IDS.
gureKddcup data base (2008) intended to replace the DARPA KDDCup99 dataset for IDS.
CTU-13 dataset - pcap files (Stratosphere IPS).

Where to go from here

Seeking more labelled datasets and determining the potential for other non-labelled datasets.
Can you write Python code for it?


I am working on the file "2018-06-24-sklearn-KDDCUP99.ipynb".
The first task is:
Train Model
Can you write Python code for it?

#  Algorithms
models = [
          #('LR', LogisticRegression()),
          ('LDA', LinearDiscriminantAnalysis()),
          #('KNN', KNeighborsClassifier()),
          #('KMN', KMeans()),
          #('CART', DecisionTreeClassifier()),
          #('NB', GaussianNB()),
         ]

# evaluate each model in turn
results = []
names = []
print('{:8}{:^8}{:^8}'.format('Model','mean','std'))
print('-' * 23)
for name, model in models:
    kfold = KFold(n_splits=N_SPLITS, random_state=SEED)

    %timeit -n1 cv_results = cross_val_score(model, X_train, y_train, cv=kfold, scoring=SCORING)
    results.append(cv_results)
    names.append(name)
    print('{:8}{:^8.2%}{:^8.2%}'.format(name, cv_results.mean(), cv_results.std()))

print(*cv_results)

previous_results = '''
LR: 98.87% (0.10%)
LDA: 99.49% (0.05%)
KNN: 99.84% (0.01%) <-- slow
CART: 99.94% (0.00%)
NB: 93.96% (0.96%)
SVM:   <-- very slow
'''

# Compare Algorithms
fig = plt.figure()
fig.suptitle('Algorithm Comparison')
ax = fig.add_subplot(111)
plt.boxplot(results)
ax.set_xticklabels(y)
plt.show()


I am working on the file "2018-06-24-sklearn-KDDCUP99.ipynb".
The first task is:
Transform Data

Dataset values are all of type "object" => convert to numeric types.
Label Encoder - replaces strings with an incrementing integer.
Can you write Python code for it?

df = pd.DataFrame(dataset_part.data)

df.head(1)

df = df.apply(pd.to_numeric, errors='ignore')

# Example from http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html
'''
le = preprocessing.LabelEncoder()
le.fit(list(names))
# le.classes_  # Shows all labels.
print(le.transform([b'icmpeco_iSF', b'icmpecr_iSF', b'icmpred_iSF']) )
print(le.inverse_transform([0, 0, 1, 2]))
'''
# https://datascience.stackexchange.com/questions/16728/could-not-convert-string-to-float-error-on-kddcup99-dataset
for column in df.columns:
    if df[column].dtype == object:
        le = preprocessing.LabelEncoder()
        df[column] = le.fit_transform(df[column])

df.head(1)  # All strings removed.


I am working on the file "2018-06-24-sklearn-KDDCUP99.ipynb".
The first task is:
Use model to make predictions:
Can you write Python code for it?


I am working on the file "2018-11-14-PyParis-widgets/notebooks/1.ipywidgets.ipynb".
The first task is:
Link two widgets
Can you write Python code for it?

from ipywidgets import IntText, IntSlider, link, HBox

text = IntText()

slider = IntSlider()

link((text, 'value'), (slider, 'value'))
HBox([text, slider])


I am working on the file "2018-11-14-PyParis-widgets/notebooks/1.ipywidgets.ipynb".
The first task is:
Media widgets
Image widget
Can you write Python code for it?

from ipywidgets import Image

import PIL.Image
import io
import numpy as np
from skimage.filters import sobel
from skimage.color.adapt_rgb import adapt_rgb, each_channel
from skimage import filters

image = Image.from_file("src/marie.png")
image

im_in = PIL.Image.open(io.BytesIO(image.value))
im_array = np.array(im_in)[...,:3]
im_array

im_array_edges = adapt_rgb(each_channel)(sobel)(im_array)
im_array_edges = ((1-im_array_edges) * 255).astype(np.uint8)
im_out = PIL.Image.fromarray(im_array_edges)
f = io.BytesIO()
im_out.save(f, format='png')
image.value = f.getvalue()


I am working on the file "2018-11-14-PyParis-widgets/notebooks/1.ipywidgets.ipynb".
The first task is:
Observe changes on the widget model
Can you write Python code for it?

from ipywidgets import ToggleButton

button = ToggleButton(description='Click me!', button_style='danger')

def update_style(change):
    button.button_style = 'info' if change['new'] else 'danger'

button.observe(update_style, 'value')
button


I am working on the file "2018-11-14-PyParis-widgets/notebooks/1.ipywidgets.ipynb".
The first task is:
Variety of widgets in the core library
Can you write Python code for it?

from ipywidgets import ColorPicker, DatePicker, IntProgress, Play, VBox, link

progress = IntProgress()
play = Play()
link((play, 'value'), (progress, 'value'))

VBox([ColorPicker(value='red'), DatePicker(), progress, play])


I am working on the file "2018-11-14-PyParis-widgets/notebooks/1.ipywidgets.ipynb".
The first task is:
Widgets protocol

Drive a boolean value
Can you write Python code for it?

from ipywidgets import Checkbox

checkbox = Checkbox(description='Check me')
checkbox

checkbox.value

checkbox.value = False


I am working on the file "2018-11-14-PyParis-widgets/notebooks/3.ipyleaflet.ipynb".
The first task is:

Repository: https://github.com/jupyter-widgets/jupyterlab-sidecar
Installation:
pip install jupyterlab_sidecar
Can you write Python code for it?

from sidecar import Sidecar
from IPython.display import display

sc = Sidecar(title='Map widget')

with sc:
    display(m)


I am working on the file "2018-11-14-PyParis-widgets/notebooks/3.ipyleaflet.ipynb".
The first task is:
Clean
Can you write Python code for it?

from ipywidgets import Widget
Widget.close_all()


I am working on the file "2018-11-14-PyParis-widgets/notebooks/3.ipyleaflet.ipynb".
The first task is:
Controls
Can you write Python code for it?

from ipyleaflet import Map, basemaps, basemap_to_tiles, SplitMapControl

m = Map(center=(42.6824, 365.581), zoom=5)

right_layer = basemap_to_tiles(basemaps.NASAGIBS.ModisTerraTrueColorCR, "2017-11-11")
left_layer = basemap_to_tiles(basemaps.NASAGIBS.ModisAquaBands721CR, "2017-11-11")

control = SplitMapControl(left_layer=left_layer, right_layer=right_layer)
m.add_control(control)

m


I am working on the file "2018-11-14-PyParis-widgets/notebooks/3.ipyleaflet.ipynb".
The first task is:
Plays well with other widgets libraries
Can you write Python code for it?

import numpy as np
import bqplot.pyplot as plt
from bqplot import *
from traitlets import observe
from sidecar import Sidecar
from ipywidgets import VBox, Button
from ipyleaflet import Map, Marker, Popup

axes_options = {'x': {'label': 'x'}, 'y': {'label': 'y'}}

x = np.arange(40)
y = np.cumsum(np.random.randn(2, 40), axis=1)

fig = plt.figure(animation_duration=1000)
lines = plt.plot(x=x, y=y, colors=['red', 'green'], axes_options=axes_options)

def generate(_):
    lines.y = np.cumsum(np.random.randn(2, 40), axis=1)

button = Button(description='Generate data', button_style='success')
button.on_click(generate)

box_plot = VBox([fig, button])

fig

center = (52.204793, 360.121558)

m = Map(center=center, zoom=9, close_popup_on_click=False)

marker = Marker(location=(52.1, 359.9))
m.add_layer(marker)

marker.popup = box_plot

sc = Sidecar(title='Map and bqplot')

with sc:
    display(m)


I am working on the file "2018-11-14-PyParis-widgets/notebooks/3.ipyleaflet.ipynb".
The first task is:
Velocity
Can you write Python code for it?

from ipyleaflet import Velocity
import xarray as xr

center = (0, 0)
zoom = 4

m2 = Map(center=center, zoom=zoom, interpolation='nearest', basemap=basemaps.CartoDB.DarkMatter)

sc2 = Sidecar(title='Map Velocity')

with sc2:
    display(m2)

ds = xr.open_dataset('src/wind-global.nc')
display_options = {
    'velocityType': 'Global Wind',
    'displayPosition': 'bottomleft',
    'displayEmptyString': 'No wind data'
}
wind = Velocity(data=ds,
                zonal_speed='u_wind',
                meridional_speed='v_wind',
                latitude_dimension='lat',
                longitude_dimension='lon',
                velocity_scale=0.01,
                max_velocity=20,
                display_options=display_options)
m2.add_layer(wind)


I am working on the file "2018-11-14-PyParis-widgets/notebooks/5.ipyvolume.ipynb".
The first task is:
Animations
Animations are made in the shaders
Can you write Python code for it?

stream = ipyvolume.datasets.animated_stream.fetch()

fig = ipyvolume.figure()
q = ipyvolume.quiver(*stream.data[:,0:50,:200], color="red", size=7)
ipyvolume.animation_control(q, interval=200)
ipyvolume.show()

u = np.linspace(-10, 10, 50)
x, y = np.meshgrid(u, u)
r = np.sqrt(x**2+y**2)
x = x.flatten()
y = y.flatten()
r = r.flatten()
time = np.linspace(0, np.pi*2, 50)
z = np.array([(np.cos(r + t) * np.exp(-r/5)) for t in time])
color = np.array([[np.cos(r + t), 1-np.abs(z[i]), 0.1+z[i]*0] for i, t in enumerate(time)])
size = (z+1)
color = np.transpose(color, (0, 2, 1))

ipyvolume.figure()
s = ipyvolume.scatter(x, z, y, color=color, marker="sphere")
ipyvolume.animation_control(s, interval=200)
ipyvolume.ylim(-3,3)
ipyvolume.show()

s.geo = "diamond"

s.size = 5

s.color = 1 - s.color


I am working on the file "2018-11-14-PyParis-widgets/notebooks/5.ipyvolume.ipynb".
The first task is:
Clean
Can you write Python code for it?


I am working on the file "2018/Class2.ipynb".
The first task is:
Case 1: Quotes to Scrape
Can you write Python code for it?

#robots.txt checked
url = "http://quotes.toscrape.com/"

#get html
response = requests.get(url)
response.status_code #200 then good, 404 then bad

page=response.content #html to scrape
page = BeautifulSoup(page,"html.parser") #change type

#find all hashtags, extract text from the first
hashtags_html = page.find_all("a",class_ = "tag") #tag finder
hashtags_html[0].get_text() #get text from the whole tag

#extract text from all hashtags and save it
#Appraoch 1: for loop
hashtags = []
for i in hashtags_html:
    hashtags.append(i.get_text())
print(hashtags)

#Appraoch 2: get texts using list comprehension
hashtags = [i.get_text() for i in hashtags_html]
print(hashtags)

#from now on, use list comprehensions for simplicity
#get links
links = [i.get("href") for i in hashtags_html]
pprint(links)

#get hashtags from links, treating links as string
hashtags_from_links = [i.split("/")[2] for i in links]

authors_html = page.find_all("small",class_ = "author")
authors = [i.get_text() for i in authors_html]
print(authors)


I am working on the file "2018/Class2.ipynb".
The first task is:
Case 3: Unicorn companies, CBInsights
Can you write Python code for it?

url = "https://www.cbinsights.com/research-unicorn-companies"
#as the data is provided inside an HTML table, we can use pandas to read it
data = pd.read_html(url)
#result is a list of one lement, which turns out to be a daraframe
type(data[0])

startups = data[0]
startups.head()


I am working on the file "2018/Class2.ipynb".
The first task is:
Case 4: Careercenter
Can you write Python code for it?

url_career = "https://careercenter.am/ccidxann.php"
tables_career = pd.read_html(url_career)
len(tables_career)

#it turned out there were 5 tables on this website, we ar einterested in the first one
tables_career[0].head()

tables_career[0].to_csv("jobs.csv")


I am working on the file "2018/Class2.ipynb".
The first task is:
Now we will try to get book titles. The difficulty of hte task is that its link does not have an identifier. We will try 3 solutions, first will not work, the other 2 will.
Can you write Python code for it?

#trial 1 - parent search (did not work)
article_tag = page.find_all("article",class_="product_pod")
titles_wrong = [i.get_text() for i in article_tag]

#trial 2 - for loop
#find all h3 tags then first (only) a tags inside
h3s = page.find_all("h3")
titles = [i.find("a").get_text() for i in h3s]

#trial 3 - content of <a>
#find all a tags, then choose only those that
#have the keyword title inside and get their text
a_tags = page.find_all("a")
titles = []
for i in a_tags:
    if str(i).find("title=")>-1:
        text = i.get_text()
        titles.append(text)


I am working on the file "2018/CodingInterview/Lecture_note/lecture_1.ipynb".
The first task is:
Lecture 1.2

문자열 뒤집기 알고리즘

Method

파이썬의 reverse() 함수이용
리스트 인덱싱 사용
Can you write Python code for it?

input_list = [1,2,3,4]
def reverse_str_1(input_list):
    input_list.reverse()
    return input_list

def reverse_str_2(input_list):
    return input_list[::-1]


I am working on the file "2018/CodingInterview/Lecture_note/lecture_1.ipynb".
The first task is:
Lecture 1.3

두개의 문자열이 순열인지 파악하는 알고리즘 (대, 소문자는 구분하지 않는다, 중복 고려x)
["I" "A" "E"] , ["A" "E" "I] -> return True
["A" "A"] , ["B", "A"] -> return False

Process

문자열 소문자 처리.
문자열 Sort
Strip 을 이용한 공백 제거
비교 연산자

Tip
파이썬의 sort() 는 공백을 우선적으로 처리한다.
Can you write Python code for it?

def permutation(str_1,str_2):
    if ''.join(sorted(list(str_1))).lower().strip() == ''.join(sorted(list(str_2))).lower().strip():
        return True
    else:
        return False

permutation("abed","abde")


I am working on the file "2018/CodingInterview/Lecture_note/lecture_1.ipynb".
The first task is:
Lecture 1.4

문자열에 공백이 있다면 "%20" 의 문자로 대체하기

Method

파이썬 replace()
Can you write Python code for it?

def change_str(input_str):
    result = input_str.replace(" ","%20")
    return result

change_str("열공 하세요")


I am working on the file "2018/Crawling.ipynb".
The first task is:
NLP - NLTK
Can you write Python code for it?

#get top 5 most popular words
freq = nltk.FreqDist(first_quote.words)
freq.plot(5, cumulative=False)

#stemming
stemmer = PorterStemmer()
word_roots = [stemmer.stem(i) for i in first_quote.words]
print(word_roots)

#stopwords and their removal
sw = stopwords.words('english')
first_quote_clean = [i for i in first_quote.words if i not in sw]
print(first_quote.words)
print("\n")
print(first_quote_clean)

#synonyms, definitions, and examples
syn = wordnet.synsets("honey")

print(syn[0].definition())
print(syn[0].examples())


I am working on the file "2018/Crawling.ipynb".
The first task is:
NLP - Textblob
Can you write Python code for it?

quotes_list = get_quotes(base)
quotes_blob = [TextBlob(i) for i in quotes_list]

#sentence sentiment analysis (opinion polarity and subjectivity)
for i in quotes_blob:
    print(i)
    print(i.sentiment)
    print("\n") #to print a separator line in between quotes

#tokenization
first_quote = quotes_blob[0]
print(first_quote,"\n")
print(first_quote.sentences,"\n")
print(first_quote.words,"\n")

#Inflection
word1 = Word("bought")
word2 = Word("animal")
word3 = Word("birds")

print(word2, "->", word2.pluralize())
print(word3, "->", word3.singularize())

#lemmatization / v-verb, n-noun, a-adjective, r-adverb
print(word1, "->", word1.lemmatize("v"))
print(word2, "->", word2.lemmatize("n"))
print(word3, "->", word3.lemmatize("n"))

#tagging / NN - noun, JJ - adjective, IN - preposition, VB_ - verb
print(first_quote.tags)

#definitions
for i in word1,word2,word3:
    print(i,"->",i.definitions,"\n")

#spellcheck and correction
print(Word("bougt").spellcheck())
print("\n")
print(Word("bougt").correct())

# langauge detection and translation
word4 = Word("մարդ")
print(word4.detect_language())
print(word4.translate(from_lang="hy",to='en'))

#comparison
word1>word2

#3-grams
first_quote.ngrams(n=3)[0:3]

#synonyms
word3.synsets[0]


I am working on the file "2018/Text_mining_-_Clustering.ipynb".
The first task is:
Important terms according to K-Means
Can you write Python code for it?

important_terms = k5.cluster_centers_.argsort()[:, ::-1]
key_list = list(tfidf_vectorizer.vocabulary_.keys())
val_list = list(tfidf_vectorizer.vocabulary_.values())
key_list[val_list.index(74)]

for i in range(k):
    for j in important_terms[i, :5]:
        print("Cluster: ", i, key_list[val_list.index(j)])


I am working on the file "2018/Text_mining_-_Clustering.ipynb".
The first task is:
K-Means clustering
Can you write Python code for it?

k=5
k5 = KMeans(n_clusters=k)
k5.fit(tfidf_matrix)
clusters = k5.labels_.tolist()
my_dict = {'quotes': quotes, 'cluster': clusters}
df = pd.DataFrame(my_dict)
print(df)
df.cluster.value_counts()


I am working on the file "2018/data_wrangling/Create Boulders Final.ipynb".
The first task is:
<br>
Add column Adjectives
Can you write Python code for it?

def add_column_adjectives(column_name="Adjectives"):
    def set_adjectives(row):
        boulder_name = row.name
        adjectives1_s = spb2018_df.loc[(spb2018_df["Boulderin nimi"] == boulder_name), "Kuvaile boulderia kolmella (3) adjektiivilla"]
        adjectives2_s = spb2018_df.loc[(spb2018_df["Boulderin nimi.1"] == boulder_name), "Kuvaile boulderia kolmella (3) adjektiivilla.1"]
        adjectives3_s = spb2018_df.loc[(spb2018_df["Boulderin nimi.2"] == boulder_name), "Kuvaile boulderia kolmella (3) adjektiivilla.2"]
        adjectives_s = adjectives1_s.append(adjectives2_s).append(adjectives3_s)
        adjectives = ",".join(adjectives_s)
        # Clean adjectives
        adjectives = ",".join(sorted(list(set([adjective.strip().lower() for adjective in adjectives.split(",")]))))
        return adjectives

    boulder_details_df[column_name] = boulder_details_df.apply(set_adjectives, axis=1)

add_column_adjectives()
boulder_details_df.head()


I am working on the file "2018/data_wrangling/Create Boulders Final.ipynb".
The first task is:
<br>
Add column GradeNumeric.
Can you write Python code for it?

def add_column_gradenumeric(column_name="GradeNumeric"):
    # Grade mappings from Font to numeric
    grade_dict = {
        "?":    0,
        "1":    1,
        "2":    2,
        "3":    3,
        "4":    4,
        "4+":   5,
        "5":    6,
        "5+":   7,
        "6A":   8,
        "6A+":  9,
        "6B":  10,
        "6B+": 11,
        "6C":  12,
        "6C+": 13,
        "7A":  14,
        "7A+": 15,
        "7B":  16,
        "7B+": 17,
        "7C":  18,
        "7C+": 19,
        "8A":  20,
        "8A+": 21,
        "8B":  22,
        "8B+": 23,
        "8C":  24,
        "8C+": 25,
        "9A":  26
    }

    boulder_details_df[column_name] = boulder_details_df.apply(lambda row: str(grade_dict[row["Grade"]]) if pd.notnull(row["Grade"]) else np.nan, axis=1)
    boulder_details_df[column_name] = boulder_details_df[column_name].astype(int)

add_column_gradenumeric()
boulder_details_df.head()


I am working on the file "2018/data_wrangling/Create Boulders Final.ipynb".
The first task is:
<br>
Add column MainHoldTypes
Can you write Python code for it?

def add_column_main_hold_types(column_name="MainHoldTypes"):
    def set_main_hold_types(row):
        boulder_name = row.name
        main_hold_types1_s = spb2018_df.loc[(spb2018_df["Boulderin nimi"] == boulder_name), "Boulderin pääotetyypit"]
        main_hold_types2_s = spb2018_df.loc[(spb2018_df["Boulderin nimi.1"] == boulder_name), "Boulderin pääotetyypit.1"]
        main_hold_types3_s = spb2018_df.loc[(spb2018_df["Boulderin nimi.2"] == boulder_name), "Boulderin pääotetyypit.2"]
        main_hold_types_s = main_hold_types1_s.append(main_hold_types2_s).append(main_hold_types3_s)
        main_hold_types = ",".join(main_hold_types_s)
        # Clean main_hold_types
        main_hold_types = ",".join(sorted(list(set([main_hold_type.strip().lower() for main_hold_type in main_hold_types.split(",")]))))
        return main_hold_types

    boulder_details_df[column_name] = boulder_details_df.apply(set_main_hold_types, axis=1)

add_column_main_hold_types()
boulder_details_df.head()


I am working on the file "2018/data_wrangling/Create Boulders Final.ipynb".
The first task is:
<br>
Add column VotedBy
Can you write Python code for it?

"""
# Simpler but slower (appr. four times) implementation
# 533 ms ± 95.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
def add_column_votedby(column_name="VotedBy"):
    # Gender mappings from Finnish to English
    gender_dict = {
        "Mies": "Male",
        "Nainen": "Female"
    }

    # Iterate over boulders
    for index, row in boulder_details_df.iterrows():
        boulder_name = index
        gender_s = spb2018_df.loc[(spb2018_df["Boulderin nimi"] == boulder_name) | (spb2018_df["Boulderin nimi.1"] == boulder_name) | (spb2018_df["Boulderin nimi.2"] == boulder_name), "Sukupuoli"]
        boulder_details_df.loc[boulder_name, column_name] = gender_dict[gender_s.iloc[0]] if gender_s.nunique() == 1 else "Both"
"""
"""
# More complex but faster (appr. four times) implementation
# 136 ms ± 5.42 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
def add_column_votedby(column_name="VotedBy"):
    # Initialize the new column
    boulder_details_df[column_name] = ""

    # Gender mappings from Finnish to English
    gender_dict = {
        "Mies": "Male",
        "Nainen": "Female"
    }

    def update_genders(gender, boulder_names):
        for boulder_name in boulder_names:
            previous_gender = boulder_details_df.loc[boulder_name, column_name]
            if previous_gender == "" or previous_gender == gender:
                boulder_details_df.loc[boulder_name, column_name] = gender
            else:
                boulder_details_df.loc[boulder_name, column_name] = "Both"

    # Iterate over form responses
    for index, row in spb2018_df.iterrows():
        gender = gender_dict[row["Sukupuoli"]]
        boulder_names = [row["Boulderin nimi"], row["Boulderin nimi.1"], row["Boulderin nimi.2"]]
        boulder_names = [boulder_name for boulder_name in boulder_names if pd.notnull(boulder_name)]
        update_genders(gender, boulder_names)
"""
# Typical implementation
# 430 ms ± 78.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
def...


I am working on the file "2018/data_wrangling/Create Boulders Final.ipynb".
The first task is:
<br>
Add column Votes.
Can you write Python code for it?

def add_column_votes(column_name="Votes"):
    boulder_name_columns = [spb2018_df["Boulderin nimi"], spb2018_df["Boulderin nimi.1"], spb2018_df["Boulderin nimi.2"]]
    all_voted_boulders_s = pd.concat(boulder_name_columns, ignore_index=True).dropna()
    boulder_votes_s = all_voted_boulders_s.value_counts()
    boulder_details_df[column_name] = boulder_votes_s

add_column_votes()
boulder_details_df.sort_values(by=["Votes"], ascending=[False]).loc[boulder_details_df["Votes"] >= 3]


I am working on the file "2018/data_wrangling/Create Boulders Final.ipynb".
The first task is:
<br>
Add columns AreaLevel1, AreaLevel2, and AreaLevel3
Can you write Python code for it?

def add_columns_arealevel1_arealevel2_and_arealevel3(column_names=["AreaLevel1", "AreaLevel2", "AreaLevel3"]):
    boulder_details_df.drop(columns=[column_names[0], column_names[1], column_names[2]], inplace=True, errors="ignore")
    geolocator = GoogleV3(api_key=GOOGLE_MAPS_JAVASCRIPT_API_KEY)

    def extract_administrative_area_levels(location_results, approximateLocation, area_levels_dict):
        # List of location result types that we are interested in
        location_result_types = ["administrative_area_level_1", "administrative_area_level_2", "administrative_area_level_3"]

        # Iterate over location results
        for location_result in location_results:
            location_result_json = location_result.raw
            # Extract data only from those location results that we are interested in
            if any(location_result_type in location_result_json["types"] for location_result_type in location_result_types):
                # Extract location result type
                location_result_type = location_result_json["types"][0]
                # Iterate over address components
                for address_component in location_result_json["address_components"]:
                    # Extract data only from the matched location result type
                    if location_result_type in address_component["types"]:
                        # Extract the name of the administrative area level 1
                        if location_result_type == location_result_types[0]:
                            area_levels_dict["AreaLevel1"] = address_component["long_name"]
                        # Extract the name of the administrative area level 2
                        if location_result_type == location_result_types[1] and approximateLocation == "No":
                            area_levels_dict["AreaLevel2"] = address_component["long_name"]
                        # Extract the name of the administrative area level 3
                        if location_result_type == location_result_types[2] and...


I am working on the file "2018/data_wrangling/Create Boulders Final.ipynb".
The first task is:
<br>
Load the datafile boulders_-_prefilled.csv, which contains manually added details of each voted boulder.
Can you write Python code for it?

boulder_details_df = pd.read_csv("data/boulders_-_prefilled.csv", index_col="Name")
boulder_details_df.head()


I am working on the file "2018/data_wrangling/Create Boulders Final.ipynb".
The first task is:
<br>
Load the datafile spb2018_-_cleaned.csv, which contains the form responses to the Suomen Parhaat Boulderit 2018 survey.
Can you write Python code for it?

# Load cleaned dataset
spb2018_df = pd.read_csv("data/survey_-_cleaned.csv")

# Drop duplicates (exclude the Timestamp column from comparisons)
spb2018_df = spb2018_df.drop_duplicates(subset=spb2018_df.columns.values.tolist()[1:])
spb2018_df.head()


I am working on the file "2018/ferran/day12/subterranean_sustainability.ipynb".
The first task is:
Part 2
A pattern is a configuration of plants modulo shifting. Every pattern has the following signature: the sequence of numbers occupied by plants, with respect to the leftmost position which we take as 0.
Can you write Python code for it?

forever = 50000000000

def pattern_repetition(state, rules, time):
    hashes = {}
    period = None
    for i, l in enumerate(evolve(state, rules, time)):
        sig = hash(tuple([c - l[0] for c in l]))
        if sig in hashes:
            period = i - hashes[sig]
            break
        hashes[hash(sig)] = i
    return i + 1, period, l

generation, period, l = pattern_repetition(initial, r, forever)
generation, period

len(l) * (forever - generation) + sum(l)


I am working on the file "2018/ferran/day12/subterranean_sustainability.ipynb".
The first task is:
Solution
Can you write Python code for it?


I am working on the file "2018/ferran/day12/subterranean_sustainability.ipynb".
The first task is:
Test
Can you write Python code for it?

initial_test = '#..#.#..##......###...###'
rules_test = ! cat input_test.txt | tr '\n' ';'
rules_test = dict(list(map(lambda x: tuple(x.split(' => ')), rules_test[0].split(';')[:-1])))

sumofpots(initial_test, rules_test, 20)


I am working on the file "2018/json-xml.ipynb".
The first task is:
JSON
Can you write Python code for it?

#example json file
my_json = [{"name":"Hrant","surname":"Davtyan"},
           {"name":"Davit","surname":"Abgaryan"}]

url = "http://api.open-notify.org/astros.json"

response = requests.get(url)
response.content

type(response.content)

# if website (API) provides not HTML but directly JSON,
# you can easily get that with requests
my_json = response.json()

pprint(my_json)

#get names from the JSON above
[my_json["people"][i]["name"] for i in range(len(my_json["people"]))]

#shorter code for getting names
[i["name"] for i in my_json["people"]]

#pandas can read JSON as well, however it loses the structure,
#if the JSON document is nested as show below
df = pd.read_json(url)

df

#saving JSON into file
with open("my_json.json","w") as f:
    json.dump(my_json,f,indent=4)

#reading JSON from a file
with open('my_json.json',"r") as f:
    data = json.load(f)

pprint(data)

#creating a string of dictionaires
json_str =   '''
{"message": "success",
"number": 6,
"people": [{"craft": "ISS", "name": "Oleg Artemyev"},
            {"craft": "ISS", "name": "Oleg Artemyev"}]}'''

str(my_json)

print(json_str)

#loading JSON from string
new_json = json.loads(json_str)

pprint(new_json)


I am working on the file "2018/materials/boulder/day2-digital-trace-data/BoulderSICSS-NoAuth.ipynb".
The first task is:

2. Pulling ego tweets

Can you write Python code for it?

# Read saved ego tweets
with open('egotweet.json', 'r') as file:
    ego = json.load(file)

# Looking at a json object
ego[0]

# Accessing an element of ego tweets
ego[0]["id_str"]

# Storing one of ego's tweet id
egoid = ego[0]["id_str"]

# Storing and printing ego tweet ids and retweet counts
tweetids = []
retweets = []

if len(ego) != 0:
    for egotweet in ego:
        tweetids.append(egotweet["id_str"])
        retweets.append(egotweet["retweet_count"])
        print(egotweet["id_str"],egotweet["retweet_count"])


I am working on the file "2018/materials/boulder/day2-digital-trace-data/BoulderSICSS-NoAuth.ipynb".
The first task is:

4. Visualizing the network of retweeters

Can you write Python code for it?

# Assigning edge weight to be number of tweets retweeted
weight = Counter()
for (i, j) in allretweeters:
    weight[(i, j)] +=1

weight

# Defining weighted edges
weighted_edges = list(weight.items())

weighted_edges

# Defining the network object
G = networkx.Graph()
G.add_edges_from([x[0] for x in weighted_edges])

# Visualizing the network
networkx.draw(G, width=[x[1] for x in weighted_edges])


I am working on the file "2018/materials/boulder/day2-digital-trace-data/BoulderSICSS-NoAuth.ipynb".
The first task is:

6. Pulling retweeters of retweeters

Can you write Python code for it?

# Reading saved alter retweeters
with open('altcheck.json', 'r') as file:
    altcheck = json.load(file)

with open('altself.json', 'r') as file:
    altself = json.load(file)

with open('altretweeters.json', 'r') as file:
    altretweeters = json.load(file)

with open('allalt.json', 'r') as file:
    allalt = json.load(file)

# Printing alter user ids, tweet ids, retweet counts,
# retweeters obtained, and whether a self tweet is included
for a, b, c, d, e in zip(altids,alttweetids,altretweets,altcheck,altself):
    print(a, b, c, d, e)

len(allalt)

allalt


I am working on the file "2018/materials/boulder/day2-digital-trace-data/BoulderSICSS.ipynb".
The first task is:

3. Pulling retweeters

Can you write Python code for it?

# Collecting Retweets
egort = api.retweets(ego[0]["id_str"])
callsRT += 1

len(egort)

callsRT

# Non-existing tweet
egort = api.retweets("garblegarble")
callsRT += 1

# Note: callsRT did not increase in the last command
callsRT

callsRT += 1

# Sleep for 10 seconds
time.sleep(10)

# Collecting retweeters of ego tweets
allretweeters = []
self = []
check = []

for egotweet in ego:
    retweeters = []
    try:
        selftweet = 0
        if callsRT >= 75:
            time.sleep(900)
        egort_raw = api.retweets(egotweet["id_str"])
        egort = [egoretweet._json for egoretweet in egort_raw]
        for retweet in egort:
            if retweet["user"]["id_str"]!=egoid:
                allretweeters.append((egoid,retweet["user"]["id_str"]))
                retweeters.append(retweet["user"]["id_str"])
            else:
                selftweet = 1
        check.append(len(retweeters))
        self.append(selftweet)
    except tweepy.TweepError:
        check.append(0)
        self.append(0)

    callsRT += 1

# Writing results to files
with open('check.json', 'w') as file:
    json.dump(check, file)

with open('self.json', 'w') as file:
    json.dump(self, file)

with open('allretweeters.json', 'w') as file:
    json.dump(allretweeters, file)

# Printing tweet ids, retweet counts,
# retweeters obtained, and whether a self tweet is included
for a, b, c, d in zip(tweetids,retweets,check,self):
    print(a, b, c, d)

len(allretweeters)

allretweeters


I am working on the file "2018/materials/boulder/day2-digital-trace-data/BoulderSICSS.ipynb".
The first task is:

5. Pulling retweeter tweets

Can you write Python code for it?

# Defining the set of unique retweeters
unique = [x[0][1] for x in weighted_edges]

len(unique)

unique

callsUT

# Collecting and storing the tweets of retweeters
alter = []
alters = []

for retweeter in unique:
    try:
        if callsUT >= 900:
            time.sleep(900)
        alter_raw = api.user_timeline(retweeter, count = A, include_rts = False, exclude_replies = True)
        alter = [altertweet._json for altertweet in alter_raw]
        alters.append(alter)
    except tweepy.TweepError:
        print("fail!")
    callsUT += 1

with open('alters.json', 'w') as file:
    json.dump(alters, file)

callsUT

len(alters)

# Printing the number of tweets pulled for each retweeter
for alt in alters:
    print(len(alt))

# Storing and printing alter ids, tweet ids, and retweet counts
altids = []
alttweetids = []
altretweets = []

for alt in alters:
    for alttweet in alt:
        altids.append(alttweet["user"]["id_str"])
        alttweetids.append(alttweet["id_str"])
        altretweets.append(alttweet["retweet_count"])
        print(alttweet["user"]["id_str"],alttweet["id_str"],alttweet["retweet_count"])


I am working on the file "2018/materials/boulder/day2-digital-trace-data/BoulderSICSS.ipynb".
The first task is:

6. Pulling retweeters of retweeters

Can you write Python code for it?

# Collecting retweeters of alter tweets
allalt = []
altself = []
altcheck = []

for alt in alters:
    for alttweet in alt:
        altid = alttweet["user"]["id_str"]
        altretweeters = []
        try:
            selftweet = 0
            if callsRT >= 75:
                time.sleep(900)
            altrt_raw = api.retweets(alttweet["id_str"])
            altrt = [altretweet._json for altretweet in altrt_raw]
            for retweet in altrt:
                if retweet["user"]["id_str"]!=altid:
                    allalt.append((altid,retweet["user"]["id_str"]))
                    altretweeters.append(retweet["user"]["id_str"])
                else:
                    selftweet = 1
            altcheck.append(len(altretweeters))
            altself.append(selftweet)
        except tweepy.TweepError:
            altcheck.append(0)
            altself.append(0)

        callsRT += 1

# Writing results to files
with open('altcheck.json', 'w') as file:
    json.dump(altcheck, file)

with open('altself.json', 'w') as file:
    json.dump(altself, file)

with open('altretweeters.json', 'w') as file:
    json.dump(altretweeters, file)

with open('allalt.json', 'w') as file:
    json.dump(allalt, file)

# Printing alter user ids, tweet ids, retweet counts,
# retweeters obtained, and whether a self tweet is included
for a, b, c, d, e in zip(altids,alttweetids,altretweets,altcheck,altself):
    print(a, b, c, d, e)

len(allalt)

allalt


I am working on the file "2018/materials/boulder/day2-digital-trace-data/Day 2 - Case Study - Web Scraping.ipynb".
The first task is:
Acknowledgements: The code below is very much inspired by Chris Bail's "Screen-Scraping in R". Thanks Chris!
Collecting Digital Trace Data: Web Scraping
Web scraping (also sometimes called "screen-scraping") is a method for extracting data from the web. There are many techniques which can be used for web scraping — ranging from requiring human involvement (“human copy-paste”) to fully automated systems. For research questions where you need to visit many webpages, and collect essentially very similar information from each, web scraping can be a great tool.
The typical web scraping program:

 Loads the address of a webpage to be scraped from your list of webpages

 Downloads the HTML or XML of that website

 Extracts any desired information

 Saves that information in a convenient format (e.g. CSV, JSON, etc.)

From Chris Bail's "Screen-Scraping in R": https://cbail.github.io/SICSS_Screenscraping_in_R.html
Legality & Politeness
When the internet was young, web scraping was a common and legally acceptable practice for collecting data on the web. But with the rise of online platforms, some of which rely heavily on user-created content (e.g. Craigslist), the data made available on these sites has become recognized by their companies as highly valuable. Furthermore, from a website developer's perspective, web crawlers are able request many pages from your site in rapid succession, increasing server loads, and generally being a nuisance.
Thus many websites, especially large sites (e.g. Yelp, AllRecipes, Instagram, The New York Times, etc.), have now forbidden "crawlers" / "robots" / "spiders" from harvesting their data in their "Terms of Service" (TOS). From Yelp's <a...
Can you write Python code for it?

pet_pages = ["https://www.boulderhumane.org/animals/adoption/dogs",
             "https://www.boulderhumane.org/animals/adoption/cats",
             "https://www.boulderhumane.org/animals/adoption/adopt_other"]

r = requests.get(pet_pages[0])
html = r.text
print(html[:500]) # Print the first 500 characters of the HTML. Notice how it's the same as the screenshot above.


I am working on the file "2018/materials/boulder/day2-digital-trace-data/Day 2 - Case Study - Web Scraping.ipynb".
The first task is:
Furthermore, we can select the name, breeds, age, and gender of the pet by find-ing the div tags which contain this information. Notice how the div tag has the attribute (attrs) class with value "views-field views-field-field-pp-animalname".
Can you write Python code for it?

name = pet[0].find('div', attrs = {'class': 'views-field views-field-field-pp-animalname'})
primary_breed = pet[0].find('div', attrs = {'class': 'views-field views-field-field-pp-primarybreed'})
secondary_breed = pet[0].find('div', attrs = {'class': 'views-field views-field-field-pp-secondarybreed'})
age = pet[0].find('div', attrs = {'class': 'views-field views-field-field-pp-age'})

# We can call `get_text()` on those objects to print them nicely.
print({
    "name": name.get_text(strip = True),
    "primary_breed": primary_breed.get_text(strip = True),
    "secondary_breed": secondary_breed.get_text(strip = True),
    "age": age.get_text(strip=True)
})


I am working on the file "2018/materials/boulder/day2-digital-trace-data/Day 2 - Case Study - Web Scraping.ipynb".
The first task is:
Now to get at the HTML object for each pet, we could find the CSS selector for each. Or, we can exploit the fact that every pet lives in a similar  HTML structure for each pet. That is, we can find all of the div tags with the class attribute which contain the string views-row. We'll print out their attributes like we just did.
Can you write Python code for it?

all_pets = soup.find_all('div', {'class': 'views-row'})

for pet in all_pets:
    name = pet.find('div', {'class': 'views-field views-field-field-pp-animalname'}).get_text(strip=True)
    primary_breed = pet.find('div', {'class': 'views-field views-field-field-pp-primarybreed'}).get_text(strip=True)
    secondary_breed = pet.find('div', {'class': 'views-field views-field-field-pp-secondarybreed'}).get_text(strip=True)
    age = pet.find('div', {'class': 'views-field views-field-field-pp-age'}).get_text(strip=True)
    print([name, primary_breed, secondary_breed, age])


I am working on the file "2018/materials/boulder/day2-digital-trace-data/Day 2 - Case Study - Web Scraping.ipynb".
The first task is:
Requesting a Webpage with Selenium
Sometimes our interactions with webpages involve rendering Javascript. For example, think of visiting a webpage with a search box, typing in a query, pressing search, and viewing the result. Or visiting a webpage that requires a login, or clicking through pages in a list. To handle pages like these we'll use a package in Python called Selenium.
Installing Selenium can be a little tricky. You'll want to follow the directions as best you can here. Requirements (one of the below):
- Firefox + geckodriver (https://github.com/mozilla/geckodriver/releases)
- Chrome + chromedriver (https://sites.google.com/a/chromium.org/chromedriver/)
First a fairly simple example: let's visit xkcd and click through the comics.
Can you write Python code for it?

driver = selenium.webdriver.Safari() # This command opens a window in Safari
# driver = selenium.webdriver.Chrome(executable_path = "<path to chromedriver>") # This command opens a window in Chrome
# driver = selenium.webdriver.Firefox(executable_path = "<path to geckodriver>") # This command opens a window in Firefox

# Get the xkcd website
driver.get("https://xkcd.com/")

# Let's find the 'random' buttom
element = driver.find_element_by_xpath('//*[@id="middleContainer"]/ul[1]/li[3]/a')
element.click()

# Find an attribute of this page - the title of the comic.
element = driver.find_element_by_xpath('//*[@id="comic"]/img')
element.get_attribute("title")

# Continue clicking throught the comics
driver.find_element_by_xpath('//*[@id="middleContainer"]/ul[1]/li[3]/a').click()

driver.quit() # Always remember to close your browser!


I am working on the file "2018/materials/boulder/day2-digital-trace-data/Day 2 - Case Study - Web Scraping.ipynb".
The first task is:
This may seem like a fairly silly example of webscraping, but one could imagine several research questions using this data. For example, if we collected this data over time (e.g. using Wayback Machine), could we identify what features of pets -- names, breeds, ages -- make them more likely to be adopted? Are there certain names that are more common for certain breeds? Or maybe your research question is something even wackier.
Aside: Read Tables from Webpages
Pandas has really neat functionality in read_html where you can download an HTML table directly from a webpage, and load it into a dataframe.
Can you write Python code for it?

table = pd.read_html("https://en.wikipedia.org/wiki/List_of_sandwiches", header=0)[0]
#table.to_csv("filenamehere.csv") # Write table to CSV

table.head(20)


I am working on the file "2018/materials/boulder/day3-networks/Day 3 - Case Study - Networkx.ipynb".
The first task is:
Another network feature that you might like to know about your network, is how assortative or modular is it. Another way of asking this, is how likely is it for similar nodes to be connected to each other? This similarity can be measured along any number of network attributes. Here we ask, how much more likely are nodes from the same group within the karate club connected to each, than we would expect at random?
Can you write Python code for it?

assort = nx.attribute_assortativity_coefficient(karate_club, 'group')
print("Assortativity coefficient: {0}".format(assort))


I am working on the file "2018/materials/boulder/day3-networks/Day 3 - Case Study - Networkx.ipynb".
The first task is:
From both measures, we can see that nodes 1 and 34 have the highest degree. (These happen to be the two leaders from the two groups within the club.)
On large networks, you might want to look at the degree distribution of your network ...
Can you write Python code for it?

# Enron email data set: http://snap.stanford.edu/data/email-Enron.html.
# (You can search "Email network (Enron corpus)" in ICON.)
with open('email_enron.txt', 'rb') as file:
    enron = nx.read_edgelist(file, comments='#') # Read in the edges

print("Enron network contains {0} nodes, and {1} edges.".format(len(enron.nodes()), len(enron.edges())))

degree_sequence = list(dict(enron.degree()).values())
print("Average degree: {0}, Maximum degree: {1}".format(np.mean(degree_sequence), max(degree_sequence)))

plt.hist(degree_sequence, bins=30) # Plots histogram of degree sequence
plt.show()


I am working on the file "2018/materials/boulder/day3-networks/Day 3 - Case Study - Networkx.ipynb".
The first task is:
Lastly, one might want to create a function on top of these networks. For example, to measure the average degree of a node's neighbors:
Can you write Python code for it?

# Example borrowed from: https://www.cl.cam.ac.uk/teaching/1314/L109/tutorial.pdf
def avg_neigh_degree(g):
    data = {}
    for n in g.nodes():
        if g.degree(n):
            data[n] = float(sum(g.degree(i) for i in g[n]))/g.degree(n)
    return data

avg_neigh_degree(g) # Can you confirm that this is returning the correct results?


I am working on the file "2018/materials/boulder/day3-networks/Day 3 - Case Study - Networkx.ipynb".
The first task is:
You can also add edge attributes, either all at once using set_edge_attributes (like we did above for set_node_attributes), or on an edge by edge basis as shown below. The shortest path between two nodes using that weight can then be calculated.
Can you write Python code for it?

# Example borrowed from: https://www.cl.cam.ac.uk/teaching/1314/L109/tutorial.pdf
g = nx.Graph()
g.add_edge('a', 'b', weight=0.1)
g.add_edge('b', 'c', weight=1.5)
g.add_edge('a', 'c', weight=1.0)
g.add_edge('c', 'd', weight=2.2)

print(nx.shortest_path(g, 'b', 'd'))
print(nx.shortest_path(g, 'b', 'd', weight='weight'))


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 1 - Text processing.ipynb".
The first task is:
A string is basically a list of these mapping numbers. We can use some other functions and methods to analyze a string like we would with other iterables (like a list).
The len of a string returns the number of characters in it.
Can you write Python code for it?


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 1 - Text processing.ipynb".
The first task is:
Appendix 2: Retrieving Wikipedia content from a list
Wikipedia maintains a (superficially) up-to-date List of S&P 500 companies, but not a category of the constituent members. Like the presidents, we want to retrieve a list of all their Wikipedia articles, parse their content, and perform some NLP tasks.
First, get the content of the article so we can parse out the list.
Can you write Python code for it?

title = 'List of S&P 500 companies'
lang = 'en'
redirects = 1

params = {'action':'parse',
          'format':'json',
          'page':title,
          'redirects':1,
          'prop':'text',
          'disableeditsection':1,
          'disabletoc':1
         }

url = 'https://en.wikipedia.org/w/api.php'
req = requests.get(url,params=params)

json_string = json.loads(req.text)

if 'parse' in json_string.keys():
    page_html = json_string['parse']['text']['*']

    # Parse the HTML into Beautiful Soup
    soup = BeautifulSoup(page_html,'lxml')


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 1 - Text processing.ipynb".
The first task is:
But there are others like wordpunct_tokenize tha makes different assumptions about the language.
Can you write Python code for it?

example_wpt_tokens = nltk.wordpunct_tokenize(example)
print("There are {0:,} words when splitting on white spaces with wordpunct_tokenize.".format(len(example_wpt_tokens)))
example_wpt_tokens[:25]


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 1 - Text processing.ipynb".
The first task is:
Case study: Preprocess the S&P500 articles and compute statistics
Step 1: Load the "sp500_wiki_articles.json", use the text_preprocessor function (or some other sequence of functions) from above to clean these articles up, and save the cleaned content to "sp500_wiki_articles_cleaned.json".
Step 2: Compute some descriptive statistics about the company articles with the most words, most unique words, greatest lexical diversity, most used words across articles, and number of unique words across all articles.
Appendix 1: Retrieving Wikipedia content by category
Functions and operations to scrape the most recent (10 August 2018) Wikipedia content from every member of "Category:Presidents of the United States".
The get_page_content function will get the content of the article as HTML and parse the HTML to return something close to a clean string of text. The get_category_subcategories and get_category_members will get all the members of a category in Wikipedia.
Can you write Python code for it?

def get_page_content(title,lang='en',redirects=1):
    """Takes a page title and returns a (large) string of the HTML content
    of the revision.

    title - a string for the title of the Wikipedia article
    lang - a string (typically two letter ISO 639-1 code) for the language
        edition, defaults to "en"
    redirects - 1 or 0 for whether to follow page redirects, defaults to 1
    parse - 1 or 0 for whether to return the raw HTML or paragraph text

    Returns:
    str - a (large) string of the content of the revision
    """

    bad_titles = ['Special:','Wikipedia:','Help:','Template:','Category:','International Standard','Portal:','s:','File:','Digital object identifier','(page does not exist)']

    # Get the response from the API for a query
    params = {'action':'parse',
          'format':'json',
          'page':title,
          'redirects':redirects,
          'prop':'text',
          'disableeditsection':1,
          'disabletoc':1
         }

    url = 'https://{0}.wikipedia.org/w/api.php'.format(lang)
    req = requests.get(url,params=params)

    json_string = json.loads(req.text)

    new_title = json_string['parse']['title']

    if 'parse' in json_string.keys():
        page_html = json_string['parse']['text']['*']

        # Parse the HTML into Beautiful Soup
        soup = BeautifulSoup(page_html,'lxml')

        # Remove sections at end
        bad_sections = ['See_also','Notes','References','Bibliography','External_links']
        sections = soup.find_all('h2')
        for section in sections:
            if section.span['id'] in bad_sections:

                # Clean out the divs
                div_siblings = section.find_next_siblings('div')
                for sibling in div_siblings:
                    sibling.clear()

                # Clean out the ULs
                ul_siblings = section.find_next_siblings('ul')
                for sibling in ul_siblings:
                    sibling.clear()
       ...


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 1 - Text processing.ipynb".
The first task is:
Comparative descriptive statistics
Now that we have cleaned biographies for each president, we can perform some basic analyses of the text. Which presidents have the longest biographies?
Can you write Python code for it?

potus_total_words = {}

for bio_name,bio_text in cleaned_bios.items():
    potus_total_words[bio_name] = len(bio_text)

pd.Series(potus_total_words).sort_values(ascending=False)


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 1 - Text processing.ipynb".
The first task is:
Confirm there are 44 presidents (shaking fist at Grover Cleveland, the 22nd and 24th POTUS) in the dictionary.
Can you write Python code for it?

print("There are {0} biographies of presidents.".format(len(bios)))


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 1 - Text processing.ipynb".
The first task is:
How many unique words?
Can you write Python code for it?

potus_unique_words = {}

for bio_name,bio_text in cleaned_bios.items():
    potus_unique_words[bio_name] = len(set(bio_text))

pd.Series(potus_unique_words).sort_values(ascending=False)


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 1 - Text processing.ipynb".
The first task is:
If we try to discover where these characters live and then map them back, we run into problems.
Can you write Python code for it?


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 1 - Text processing.ipynb".
The first task is:
Lemmatization
Lemmatization (and the related concept of stemming) are methods for dealing with conjugated words. Word like "ate" or "eats" are counted as distinct from "eat", although semantically they are similar and should likely be grouped together. Where stemming just removes commons suffixes and prefixes, sometimes resulting in mangled words, lemmatization attempts returns the root word. However, lemmatization can be extremely expensive computationally, which does not make it a good candidate for large corpora.
The get_wordnet_pos and lemmatizer functions below work eith each other to lemmatize a word to its root. This involves attempting to discover the part-of-speech (POS) for each word and passing this POS to NLTK's lemmatize function, ultimately returning the root word (if it exists in the "wordnet" corpus).
Can you write Python code for it?

from nltk.corpus import wordnet

from nltk.stem import WordNetLemmatizer
wnl = WordNetLemmatizer()

def get_wordnet_pos(treebank_tag):
    if treebank_tag.startswith('J'):
        return wordnet.ADJ
    elif treebank_tag.startswith('V'):
        return wordnet.VERB
    elif treebank_tag.startswith('N'):
        return wordnet.NOUN
    elif treebank_tag.startswith('R'):
        return wordnet.ADV
    else:
        return wordnet.NOUN

def lemmatizer(token):
    token,tb_pos = nltk.pos_tag([token])[0]
    pos = get_wordnet_pos(tb_pos)
    lemma = wnl.lemmatize(token,pos)
    return lemma


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 1 - Text processing.ipynb".
The first task is:
Let's combine them so get a list of all_stopwords that we can ignore.
Can you write Python code for it?

all_stopwords = english_stopwords + list(string.punctuation) + ['–']


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 1 - Text processing.ipynb".
The first task is:
Loop through all the tokens in wpt_lowered_no_stopwords, applying the lemmatizer function to each. Then inspect 25 examples of words where the lemmatizer changed the word length.
Can you write Python code for it?

wpt_lemmatized = [lemmatizer(t) for t in wpt_lowered_no_stopwords]
[(i,j) for (i,j) in list(zip(wpt_lowered_no_stopwords,wpt_lemmatized)) if len(i) != len(j)][:25]


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 1 - Text processing.ipynb".
The first task is:
Loop through the fourth through rest of the presidents list and get each president's biography using get_page_content. Store the results in the presidents_wiki_bios dictionary.
Can you write Python code for it?

presidents_wiki_bios = {}
for potus in presidents[3:]:
    presidents_wiki_bios.update(get_page_content(potus))


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 1 - Text processing.ipynb".
The first task is:
Now we can use get_page_content to get the content of each company's page and add it to the sp500_articles dictionary.
Can you write Python code for it?

sp500_articles = {}

for company in set(company_links):
    sp500_articles.update(get_page_content(company))


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 1 - Text processing.ipynb".
The first task is:
Save the data to a JSON file.
Can you write Python code for it?

with open('sp500_wiki_articles.json','w') as f:
    json.dump(sp500_articles,f)


with open('potus_wiki_bios.json','w') as f:
    json.dump(presidents_wiki_bios,f)


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 1 - Text processing.ipynb".
The first task is:
Stop words
English, like many languages, repeats many words in typical language that don't always convey a lot of information by themselves. When we do text processing, we should make sure to remove these "stop words".
Can you write Python code for it?


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 1 - Text processing.ipynb".
The first task is:
The distribution of word frequencies, even after stripping out stopwords, follows a remarkable strong pattern. Most terms are used infrequently (upper-left) but a handful of terms are used repeatedly! Zipf's law states:

"the frequency of any word is inversely proportional to its rank in the frequency table. Thus the most frequent word will occur approximately twice as often as the second most frequent word, three times as often as the third most frequent word, etc."
Can you write Python code for it?

freq_counter = Counter(fdist_wpt_lowered_no_stopwords.values())

f,ax = plt.subplots(1,1)

ax.scatter(x=list(freq_counter.keys()),y=list(freq_counter.values()))
ax.set_xscale('log')
ax.set_yscale('log')
ax.set_xlabel('Term frequency')
ax.set_ylabel('Number of terms')


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 1 - Text processing.ipynb".
The first task is:
The easy eay is to use use pandas's read_html function to parse the table into a DataFrame and access the "Security" (second) column.
Can you write Python code for it?

company_df = pd.read_html(str(component_stock_table),header=0)[0]
company_df.head()

company_names = company_df['Security'].tolist()


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 1 - Text processing.ipynb".
The first task is:
The hard way to get a list of the company names out is parsing the HTML table. We:

Find all the tables in the soup
Get the first table out
Find all the rows in the table
Loop through each row
Find the links in each row
Get the second link's title in each row
Add the title to company_names
Can you write Python code for it?

company_names = []

# Get the first table
component_stock_table = soup.find_all('table')[0]

# Get all the rows after the first (header) row
rows = component_stock_table.find_all('tr')[1:]

# Loop through each row and extract the title
for row in rows:
    # Get all the links in a row
    links = row.find_all('a')
    # Get the title in the 2nd cell from the left
    title = links[1]['title']
    # Add it to the company_links
    company_names.append(title)

print("There are {0:,} titles in the list".format(len(set(company_names))))


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 1 - Text processing.ipynb".
The first task is:
The length of this b string somehow got a new character in it compared to the original s string.
Can you write Python code for it?


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 1 - Text processing.ipynb".
The first task is:
The lexical diversity is the ratio of unique words to total words. Values closer to 0 indicate the presence of repeated words (low diversity) and values closer to 1 indicate words used only once (high diversity).
Can you write Python code for it?

def lexical_diversity(token_list):
    unique_tokens = len(set(token_list))
    total_tokens = len(token_list)
    if total_tokens > 0:
        return unique_tokens/total_tokens
    else:
        return 0

potus_lexical_diversity = {}

for bio_name,bio_text in cleaned_bios.items():
    potus_lexical_diversity[bio_name] = lexical_diversity(bio_text)

pd.Series(potus_lexical_diversity).sort_values(ascending=False)


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 1 - Text processing.ipynb".
The first task is:
There are a variety of strategies for splitting a text document up into its constituent words, each making different assumptions about word boundaries, which results in different counts of the resulting tokens.
Can you write Python code for it?

for name,tokenlist in zip(['space_split','re_tokenizer','word_tokenizer','wordpunct_tokenizer','toktok_tokenizer'],[example_ws_tokens,example_re_tokens,example_wt_tokens,example_wpt_tokens,example_ttt_tokens]):
    print("{0:>20}: {1:,} total tokens, {2:,} unique tokens".format(name,len(tokenlist),len(set(tokenlist))))


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 1 - Text processing.ipynb".
The first task is:
There are many, many kinds of character encodings for representing non-ASCII text.
 This cartoon pretty much explains why there are so many standards rather than a single standard:

Latin-1: the basis for many encodings
CP-1252: a common default encoding in Microsoft products similar to Latin-1
UTF-8: one of the most widely adopted and compatible - use it wherever possible
CP-437: used by the original IBM PC (predates latin1) but this old zombie is still lurking
GB-2312: implemented to support Chinese & Japanese characters, Greek & Cyrillic alphabets
UTF-16: treats everyone equally poorly, here there also be emojis

Other resources on why Unicode is what it is by Ned Batchelder, this tutorial by Esther Nam and Travis Fischer, or this Unicode tutorial in the docs.
Can you write Python code for it?

for codec in ['latin1','utf8','cp437','gb2312','utf16']:
    print(codec.rjust(10),s.encode(codec), sep=' = ')


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 1 - Text processing.ipynb".
The first task is:
This last character é also exists at a specific location.
Can you write Python code for it?


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 1 - Text processing.ipynb".
The first task is:
Two (or more) strings can be combined by adding them together.
Can you write Python code for it?


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 1 - Text processing.ipynb".
The first task is:
Using a different decoding standard like CP1252 returns something much more grotesque without throwing any errors.
Can you write Python code for it?


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 1 - Text processing.ipynb".
The first task is:
We can also use string module's "punctuation" attribute as well.
Can you write Python code for it?


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 1 - Text processing.ipynb".
The first task is:
We can apply this function to every presidential biography (this may take a minute or so) and write the resulting list of cleaned tokens to the "potus_wiki_bios_cleans.json" file. We'll use this file in the next lecture as well.
Can you write Python code for it?

# Clean each bio
cleaned_bios = {}

for bio_name,bio_text in bios.items():
    cleaned_bios[bio_name] = text_preprocessor(bio_text)

# Save to disk
with open('potus_wiki_bios_cleaned.json','w') as f:
    json.dump(cleaned_bios,f)


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 1 - Text processing.ipynb".
The first task is:
We can convert from this byte-level representation back into Unicode with the .decode method.
Can you write Python code for it?

b.decode('utf8')

ord(b'\xc3\xa9'.decode('utf8')), chr(233)


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 1 - Text processing.ipynb".
The first task is:
We can count how often a word occurs in each biography.
Can you write Python code for it?

# Import the Counter function
from collections import Counter

# Get counts of each token from the cleaned_bios for Grover Cleveland
cleveland_counts = Counter(cleaned_bios['Grover Cleveland'])

# Convert to a pandas Series and sort
pd.Series(cleveland_counts).sort_values(ascending=False).head(25)

potus_word_counts = {}

for bio_name,bio_text in cleaned_bios.items():
    potus_word_counts[bio_name] = Counter(bio_text)

potus_word_counts_df = pd.DataFrame(potus_word_counts).T

potus_word_counts_df.to_csv('potus_word_counts.csv',encoding='utf8')

print("There are {0:,} unique words across the {1} presidents.".format(potus_word_counts_df.shape[1],potus_word_counts_df.shape[0]))


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 1 - Text processing.ipynb".
The first task is:
We can use a list comprehension to exclude the words in this stopword list from analysis while also gives each word similar cases. This is not perfect, but an improvement over what we had before.
Can you write Python code for it?

wpt_lowered_no_stopwords = []

for word in example_wpt_tokens:
    if word.lower() not in all_stopwords:
        wpt_lowered_no_stopwords.append(word.lower())

fdist_wpt_lowered_no_stopwords = nltk.FreqDist(wpt_lowered_no_stopwords)
fdist_wpt_lowered_no_stopwords.most_common(25)


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 1 - Text processing.ipynb".
The first task is:
We could use regular expressions to split on repeated whitespaces.
Can you write Python code for it?

example_re_tokens = re.split(r'\s+',example)
print("There are {0:,} words when splitting on white spaces with regular expressions.".format(len(example_re_tokens)))
example_re_tokens[0:25]


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 1 - Text processing.ipynb".
The first task is:
When you're doing comparisons, you're basically comparing these numbers to each other.
Can you write Python code for it?


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 1 - Text processing.ipynb".
The first task is:
Which words occur the most across presidential biographies?
Can you write Python code for it?


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 1 - Text processing.ipynb".
The first task is:
Word cases
Remember that strings of different cases (capitalizations) are treated as different words: "young" and "Young" are not the same. An important part of text processing is to remove un-needed variation, and mixed cases are variation we generally don't care about.
Can you write Python code for it?

example_wpt_lowered = [token.lower() for token in example_wpt_tokens]
unique_wpt = len(set(example_wpt_tokens))
unique_lowered_wpt = len(set(example_wpt_lowered))
difference = unique_wpt - unique_lowered_wpt

print("There are {0:,} unique words in example before lowering and {1:,} after lowering,\na difference of {2} unique tokens.".format(unique_wpt,unique_lowered_wpt,difference))


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 1 - Text processing.ipynb".
The first task is:
You will almost certainly encounter string encoding problems whenever you work with text data. Let's look at how quickly things can go wrong trying to decode a string when we don't know the standard.
Some standards map the \xe9 byte-level representation to the é character we intended, while other standards have nothing at that byte location, and still others map that byte location to a different character.
Can you write Python code for it?

montreal_s = b'Montr\xe9al'

for codec in ['cp437','cp1252','latin1','gb2312','iso8859_7','koi8_r','utf8','utf16']:
    print(codec.rjust(10),montreal_s.decode(codec,errors='replace'),sep=' = ')


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 1 - Text processing.ipynb".
The first task is:
You'll notice that this ASCII character mapping doesn't include characters that have accents.
Can you write Python code for it?


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 2 - Sentiment analysis.ipynb".
The first task is:
Apply the clean_tweets functions to the hydrated tweets for each candidate. Write the cleaned data to disk.
Can you write Python code for it?

obama_cleaned_tweets = clean_tweets(obama_hydrated_tweets)
clinton_cleaned_tweets = clean_tweets(clinton_hydrated_tweets)
romney_cleaned_tweets = clean_tweets(romney_hydrated_tweets)
trump_cleaned_tweets = clean_tweets(trump_hydrated_tweets)

for filename,cleaned_tweets in zip(['cleaned_tweets_obama.json','cleaned_tweets_clinton.json','cleaned_tweets_romney.json','cleaned_tweets_trump.json'],[obama_cleaned_tweets,clinton_cleaned_tweets,romney_cleaned_tweets,trump_cleaned_tweets]):
    with open(filename,'w') as f:
        json.dump(cleaned_tweets,f)


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 2 - Sentiment analysis.ipynb".
The first task is:
Case study: Exploring other presidential candidates' Twitter histories
Step 1: Convert Obama (obama_tweets), Clinton (clinton_tweets), or Romney's (romney_tweets) tweets to a pandas DataFrame and use the new_tweet_features function to add some new variables to the DataFrame.
Step 2: Use seaborn's catplot to visualize some features of these candidates' Twitter histories (by year, by day of week, by hour of day, by device, etc.). How do these candidates' activity compare to Trump's Twitter activity?
Step 3: Plot the moving average of other candidates' sentiment over time using a group-aggregation, reindexing, and rolling average like we did with Trump above. What are some key changepoints?
Appendix 1: Get complete user timelines and hydrate their tweets
Here is some code called GetOldTweets with some complex dependencies that flagrantly violates Twitter's Terms of Service in order to get all of the tweets from a user's timeline (rather than the 3,200 most recent). Twitter will eventually block your IP address if you run this code too much.
I am comfortable violating Twitter's ToS in order to retrieve an archive of presidential candidates' tweets that is not otherwise available.
Can you write Python code for it?

from GetOldTweets import got3
from datetime import datetime, timedelta

def tweet_converter(tweet):
    """
    Takes a tweet object generated by the GetOldTweets getTweets() function
    and converts it to a dictionary
    """
    d = {'author_id': tweet.author_id,
         'favorites': tweet.favorites,
         'formatted_date': tweet.formatted_date,
         'location': tweet.geo,
         'hashtags': tweet.hashtags,
         'id': tweet.id,
         'mentions': tweet.mentions,
         'permalink': tweet.permalink,
         'retweets': tweet.retweets,
         'text': tweet.text,
         'urls': tweet.urls,
         'username': tweet.username}
    return d

def tweet_grabber(screen_name,year,write_json=False):
    """
    Takes a screen_name as a string and a year and pulls all the tweets and replies from that account in that year
    Writes the tweets as two separate JSON files in the format 'tweets_<screenname>_<year>.json'
    and 'replies_<screenname>_<year>.json'

    Returns two dictionaries, containing the tweets and replies
    """
    year = int(year)

    # Get the tweets for a single year and convert the tweet objects to a dictionary
    tweet_criteria = got3.manager.TweetCriteria().setUsername(screen_name).setSince("{0}-01-01".format(year)).setUntil("{0}-01-01".format(year+1)).setMaxTweets(9999)
    tweets = got3.manager.TweetManager.getTweets(tweet_criteria)
    converted_tweets = [tweet_converter(t) for t in tweets]

    # Get the replies for a single year and convert the tweet objects to a dictionary
    replies_criteria = got3.manager.TweetCriteria().setUsername(screen_name).setSince("{0}-01-01".format(year)).setUntil("{0}-01-01".format(year+1)).setQuerySearch('filter:replies').setMaxTweets(9999)
    replies = got3.manager.TweetManager.getTweets(replies_criteria)
    converted_replies = [tweet_converter(t) for t in replies]

    # Only write to disk at this step if write_json is True
    if write_json:
        # Save to disk
        with...


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 2 - Sentiment analysis.ipynb".
The first task is:
Compared to traditional "bag of words" approaches for sentiment analysis, VADER also preserves word order; includes emoticons (:-)), slang (meh), and initialisms (lol); and punctuation cues.
We initialized the analyzer model from VADER's SentimentIntensityAnalyzer class we imported and pass a sentence to the polarity_scores method. VADER takes care of a lot of the tokenizing, casing, and stemming, so we can give it uncleaned sentences.
The polarity_scores method returns a dictionary containing the metrics for the proportion of the text that are negative, neutral, and positive. The most important value for our purposes is the "compound" value with is the normalized sum of these valences: this is the sentiment score of a sentence where -1 is extremely negative and 1 is extremely positive.
If we apply this to the two sentences, we can see VADER is much better at identifying the proper sentiment.
Can you write Python code for it?

analyzer.polarity_scores("The book was good.")

analyzer.polarity_scores("The book was not good.")


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 2 - Sentiment analysis.ipynb".
The first task is:
Comparing daily approval with daily sentiment
How has President Trump's Twitter sentiment score varied with his approval number among voters? We grab the daily averaged approval data from FiveThirtyEight.
Can you write Python code for it?

fte_approvals = pd.read_csv('https://projects.fivethirtyeight.com/trump-approval-data/approval_topline.csv',parse_dates=['modeldate'])
fte_voters = fte_approvals[fte_approvals['subgroup'] == 'Voters']
fte_approvals.head()


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 2 - Sentiment analysis.ipynb".
The first task is:
Convert this (flat) JSON data to a pandas DataFrame so it's easier to manipulate and visualize. We'll also use this new_tweet_features function to generate some new features (date, year, hour, weekday, word count) that will be useful.
Can you write Python code for it?

def new_tweet_features(df):
    df['timestamp'] = pd.to_datetime(df['timestamp'])
    df['date'] = pd.to_datetime(df['date'])
    df['weekday'] = df['date'].apply(lambda x:x.weekday())
    df['year'] = df['date'].apply(lambda x:x.year)
    df['hour'] = df['timestamp'].apply(lambda x:x.hour)
    df['wordcount'] = df['text'].apply(lambda x:len(wordpunct_tokenize(x)))
    df = df.sort_values('timestamp')
    return df

trump_tweets_df = pd.DataFrame(trump_tweets)
trump_tweets_df = new_tweet_features(trump_tweets_df)
trump_tweets_df.head()


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 2 - Sentiment analysis.ipynb".
The first task is:
Get the tweets for each candidate. This code is "commented" out to prevent it from being inadvertently run because it takes on the order of 2 hours to complete.
Using the python-twitter library and API developer keys, we can "hydrate" the tweets from the official Twitter API to get the full payload of information.
Can you write Python code for it?

import twitter
from bs4 import BeautifulSoup

api = twitter.Api(consumer_key = 'get your own',
                  consumer_secret = 'get your own',
                  access_token_key = 'get your own',
                  access_token_secret = 'get your own',
                  tweet_mode = 'extended',
                  sleep_on_rate_limit = True
                 )

def chunks(l, n):
    # https://stackoverflow.com/questions/312443/how-do-you-split-a-list-into-evenly-sized-chunks
    """Yield successive n-sized chunks from l."""
    for i in range(0, len(l), n):
        yield l[i:i + n]

def hydrate_tweets(df,column='id'):
    tweets = []

    for chunk in chunks(df[column].tolist(),100):
        _statuses = api.GetStatuses(chunk)
        _statuses = [t.AsDict() for t in _statuses]
        tweets += _statuses

    return tweets


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 2 - Sentiment analysis.ipynb".
The first task is:
Hydrate the tweets for each candidate. This code is "commented" out to prevent it from being inadvertently run because it takes on the order of 5 minutes to complete.
Write the hydrated data to disk.
Can you write Python code for it?

for filename,hyrdrated_tweets in zip(['raw_tweets_obama.json','raw_tweets_clinton.json','raw_tweets_romney.json','raw_tweets_trump.json'],[obama_hydrated_tweets,clinton_hydrated_tweets,romney_hydrated_tweets,trump_hydrated_tweets]):
    with open(filename,'w') as f:
        json.dump(hyrdrated_tweets,f)


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 2 - Sentiment analysis.ipynb".
The first task is:
I've made an approval_sentiment_df DataFrame by combining FiveThirtyEight and sentiment scores we computed. We can plot the relationship between these values using seaborn's lmplot with LOWESS and print the correlation.
Can you write Python code for it?

fte_approval = fte_voters.set_index('modeldate')['approve_estimate']
trump_sentiment = daily_trump_sentiment.loc[fte_approval.index]
approval_sentiment_df = pd.DataFrame({'Approval':trump_approval,'Sentiment':trump_sentiment})

corr = approval_sentiment_df.corr().loc['Approval','Sentiment']
print('Correlation is: {0:.4f}'.format(corr))

sb.lmplot(x = 'Approval',
          y = 'Sentiment',
          data = approval_sentiment_df,
          lowess = True,
          line_kws = {'lw':3,'color':'red'},
          aspect = 2
         )

f,ax = plt.subplots(1,1,figsize=(12,4))

approval_sentiment_df.rolling(7).mean().plot(secondary_y='Sentiment',ax=ax)


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 2 - Sentiment analysis.ipynb".
The first task is:
Once we have a sentiment score for each presidential sentence, we can compute the average across all sentences within a biography, and sort the resulting average presidential sentiment.
Can you write Python code for it?

mean_potus_sentiment = pd.Series({potus:np.mean(scores) for potus,scores in potus_sentiments.items()})
mean_potus_sentiment.sort_values(ascending=False)


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 2 - Sentiment analysis.ipynb".
The first task is:
Sentiment analysis of Trump tweets
The Trump Twitter Archive maintains an up-to-date archive of all the tweets from @realDonaldTrump. I broke Twitter's TOS and screen-scaped all of this historical tweets (but not retweets) generated by this account as well as for Hillary Clinton, Mitt Romney, and Barack Obama.
Can you write Python code for it?

with open('cleaned_tweets_trump.json','r') as f:
    trump_tweets = json.load(f)

with open('cleaned_tweets_clinton.json','r') as f:
    clinton_tweets = json.load(f)

with open('cleaned_tweets_romney.json','r') as f:
    romney_tweets = json.load(f)

with open('cleaned_tweets_obama.json','r') as f:
    obama_tweets = json.load(f)


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 2 - Sentiment analysis.ipynb".
The first task is:
Sentiment analysis of presidential biographies
Let's return to Wikipedia's presidential biographies and do some sentiment analysis.
Can you write Python code for it?

with open('potus_wiki_bios.json','r') as f:
    bios = json.load(f)

print("There are {0} biographies of presidents.".format(len(bios)))


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 2 - Sentiment analysis.ipynb".
The first task is:
Traditional sentiment analysis tools like LIWC use a "bag of words" approach that is indifferent to word order, negations, etc. Let's use the word-level sentiment scores from VADER's lexicon, but use them in a super-naive "bag of words" approach.
In the case of these two sentences, we tokenize the sentences into words, look up the words' sentiment scores in the lexicon_dict, print out each word's score, and then add up the scores for all the words in sentiment_score_words1. We see that this super-naive bag of words approach gives both of these sentences the same sentiment score.
Can you write Python code for it?

words1 = wordpunct_tokenize("The book was good.")
words2 = wordpunct_tokenize("The book was not good.")

sentiment_score_words1 = 0
for w in words1:
    lexicon_score = lexicon_dict.get(w.lower(),0)
    print(w,lexicon_score)
    sentiment_score_words1 += lexicon_score

print('\n')

sentiment_score_words2 = 0
for w in words2:
    lexicon_score = lexicon_dict.get(w.lower(),0)
    print(w,lexicon_score)
    sentiment_score_words2 += lexicon_score

print('\n')
print("The book was good. ==>",sentiment_score_words1)
print("The book was not good. ==>",sentiment_score_words2)


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 2 - Sentiment analysis.ipynb".
The first task is:
Trump, like other Twitter users, can tweet multiple times in a day. We will perform a groupby-aggregation operation to compute the average sentiment of all tweets per day.
I've plotted the 90-day rolling average of sentiment and indicated five major events:

The 2011 White House Correspondent's Dinner where Trump was roasted by Obama and Seth Meyers
Trump's 2016 campaign launched in June 2015
Trump unexpectedly won the presidency in November 16
Trump was inaugurated in January 2017
The Department of Justice appointed a Special Investigator in May 2017
Can you write Python code for it?

daily_trump_sentiment = trump_tweets_df.groupby('date').agg({'sentiment':np.mean})['sentiment']
daily_trump_sentiment = daily_trump_sentiment.reindex(pd.date_range(daily_trump_sentiment.index.min(),daily_trump_sentiment.index.max()),fill_value=0)

f,ax = plt.subplots(1,1,figsize=(12,4))

daily_trump_sentiment.rolling(90).mean().plot(ax=ax,lw=3)

ax.set_ylim((-.1,.4))
ax.axhline(y=0,c='k',ls='--')
ax.axvline(x=pd.Timestamp('2011-05-01'),c='k') # WH Correspondent's Dinner
ax.axvline(x=pd.Timestamp('2015-06-16'),c='k') # Campaign launched
ax.axvline(x=pd.Timestamp('2016-11-08'),c='k') # Election
ax.axvline(x=pd.Timestamp('2017-01-20'),c='k') # Inauguration
ax.axvline(x=pd.Timestamp('2017-05-17'),c='k') # Special Investigator appointed
ax.set_ylabel('Mean sentiment')


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 2 - Sentiment analysis.ipynb".
The first task is:
We need to tokenize each sentence in the biography. Let's do Grover Cleveland as an example.
Can you write Python code for it?

sentences = sent_tokenize(bios['Grover Cleveland'])
sentences


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 3 - Text networks and word embeddings.ipynb".
The first task is:
A really basic exploratory scatterplot for the number of words in each President's biography compared to their POTUS index.
Can you write Python code for it?


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 3 - Text networks and word embeddings.ipynb".
The first task is:
Case study: S&P500 company clusters
Step 1: Using the sp500_tfidf_dense array/DataFrame, experiment with different dimensionality reduction tools we covered above. Visualize and inspect the distribution of S&P500 companies for interesting dimensions (do X and Y dimensions in this reduced data capture anything meaningful?) or clusters (do companies clusters together as we'd expect?).
Visualizing word embeddings
Using the bio_counts, we can find the top-N most frequent words and save them as top_words.
Can you write Python code for it?

top_words = pd.DataFrame(bio_counts.todense().sum(0).T,
                              index=count_vect.get_feature_names())[0]

top_words = top_words.sort_values(0,ascending=False).head(1000).index.tolist()


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 3 - Text networks and word embeddings.ipynb".
The first task is:
Case study: Text similarity network of the S&P 500 companies
Step 1: Load and preprocess the content of the articles.
Can you write Python code for it?

# Load the data
with open('sp500_wiki_articles.json','r') as f:
    sp500_articles = json.load(f)

# Bring in the text_preprocessor we wrote from Day 4, Lecture 1
def text_preprocessor(text):
    """Takes a large string (document) and returns a list of cleaned tokens"""
    tokens = nltk.wordpunct_tokenize(text)
    clean_tokens = []
    for t in tokens:
        if t.lower() not in all_stopwords and len(t) > 2:
            clean_tokens.append(lemmatizer(t.lower()))
    return clean_tokens

# Clean each article
cleaned_sp500 = {}

for name,text in sp500_articles.items():
    cleaned_sp500[name] = text_preprocessor(text)

# Save to disk
with open('sp500_wiki_articles_cleaned.json','w') as f:
    json.dump(cleaned_sp500,f)


I am working on the file "2018/materials/boulder/day4-text-analysis/Day 4, Lecture 3 - Text networks and word embeddings.ipynb".
The first task is:
Each word in the vocabulary exists as a N-dimensional vector, where N is the "size" hyper-parameter set in the model. The "congress" token in located at this position in the 100-dimensional space we trained in bios_model.
Can you write Python code for it?

bios_model.wv['congress']

bios_model.wv.most_similar('congress')

bios_model.wv.most_similar('court')

bios_model.wv.most_similar('war')

bios_model.wv.most_similar('election')