I am working on the file "Fall_2015/LB03/Python_lunchbytes_RSMAS.ipynb".
The first task is:
Compute the field time mean value using CDO and return it as a numpy array without writing nc file to disk.
Can you write Python code for it?

%%timeit
cdo.timmean(input=ifile,returnCdf=True).variables['sst'][:]

tavg_cdo = cdo.timmean(input=ifile,returnCdf=True).variables['sst'][:]

figure(figsize=(13, 4))
m = Basemap(projection='cyl', resolution='c', llcrnrlon=0, llcrnrlat=-90,
            urcrnrlon=360, urcrnrlat=90)
x, y = m(*np.meshgrid(lons, lats))
clevs = np.linspace(-1, 30, 64)
cs = m.pcolormesh(x,y,tavg_cdo.squeeze(), shading='flat', cmap=plt.cm.jet)
m.drawcoastlines()
m.drawparallels((-90, -60, -30, 0, 30, 60, 90), labels=[1,0,0,0], fontsize=14)
m.drawmeridians((0, 60, 120, 180, 240, 300, 360), labels=[0,0,0,1], fontsize=14)
m.fillcontinents()
colorbar(cs, label='degrees $^o$C', extend='both')
plt.title('Time Mean ERSST V4 1854-2015 from CDO ', fontsize=16)

diff = tavg_cdo.squeeze() - temp
diff.max()

ifile = '20101221-JPL-L4UHfnd-GLOB-v01-fv04-MUR.nc'
f = Dataset(ifile)
temp = f.variables['analysed_sst']
temp

temp = f.variables['analysed_sst'][:].squeeze()
lats = f.variables['lat'][:]
lons = f.variables['lon'][:]
f.close()

from matplotlib.patches import Polygon
from mpl_toolkits.axes_grid1.inset_locator import inset_axes

def make_map(llcrnrlon=-100, urcrnrlon=-50, llcrnrlat=15, urcrnrlat=50,
             projection='cyl', resolution='l', figsize=(10, 3.5), inset=True):

    m = Basemap(llcrnrlon=llcrnrlon, urcrnrlon=urcrnrlon,
                llcrnrlat=llcrnrlat, urcrnrlat=urcrnrlat,
                projection=projection, resolution=resolution)

    fig, ax = plt.subplots(figsize=figsize)
    m.drawcoastlines()
    m.bluemarble()
    meridians = np.arange(llcrnrlon, urcrnrlon + 2, 10)
    parallels = np.arange(llcrnrlat, urcrnrlat + 1, 10)
    m.drawparallels(parallels, linewidth=0, labels=[1, 0, 0, 0], fontsize=20)
    m.drawmeridians(meridians, linewidth=0, labels=[0, 0, 0, 1], fontsize=20)
    m.llcrnrlon = llcrnrlon
    m.urcrnrlon = urcrnrlon
    m.llcrnrlat = llcrnrlat
    m.urcrnrlat = urcrnrlat
    m.ax = ax

    if inset:
        axin = inset_axes(m.ax, width="45%", height="45%", loc=4)
        # Global inset...


I am working on the file "Fall_2015/LB03/Python_lunchbytes_RSMAS.ipynb".
The first task is:
Even got python documentation for old_sin
Can you write Python code for it?


I am working on the file "Fall_2015/LB03/Python_lunchbytes_RSMAS.ipynb".
The first task is:
Good old Fortran!
Can you write Python code for it?

from IPython.display import Image

Image("http://s18.postimg.org/xoghfy291/Olde_Fortran_Malt_Liquor.jpg")


I am working on the file "Fall_2015/LB03/Python_lunchbytes_RSMAS.ipynb".
The first task is:
In order to see what is going on there it is useful to look at the box plots for every month:
Can you write Python code for it?

ax = aonao.boxplot(column=['AO'], by='mon')

ax = aonao.boxplot(column=['NAO'], by='mon')


I am working on the file "Fall_2015/LB03/Python_lunchbytes_RSMAS.ipynb".
The first task is:
Install CDO Python bindings to use embeded CDO commands with "pip install cdo"
Can you write Python code for it?

from cdo import *

cdo = Cdo()

cdo.sinfon(input=ifile)


I am working on the file "Fall_2015/LB03/Python_lunchbytes_RSMAS.ipynb".
The first task is:
Lets use Numpy version
Can you write Python code for it?

import numpy as np
%timeit np.sin(30.* np.pi / 180.)

single(np.sin(30.* np.pi / 180.))


I am working on the file "Fall_2015/LB03/Python_lunchbytes_RSMAS.ipynb".
The first task is:
Time Series analysis with Pandas (statistical package)
Can you write Python code for it?

import pandas as pd
import numpy as np

pd.set_option('max_rows',15)

!wget http://www.cpc.ncep.noaa.gov/products/precip/CWlink/daily_ao_index/monthly.ao.index.b50.current.ascii
!wget http://www.cpc.ncep.noaa.gov/products/precip/CWlink/pna/norm.nao.monthly.b5001.current.ascii

ao = np.loadtxt('monthly.ao.index.b50.current.ascii')
ao

dates = pd.date_range('1950-01', '2015-10', freq='M')
AO = pd.Series(ao[:,2], index=dates)
AO

!head norm.nao.monthly.b5001.current.ascii

NAO = pd.read_table('norm.nao.monthly.b5001.current.ascii', sep='\s*', \
              parse_dates={'dates':[0, 1]}, header=None, index_col=0, squeeze=True, engine='python' )


I am working on the file "Fall_2015/LB03/Python_lunchbytes_RSMAS.ipynb".
The first task is:
first argument is obviously the file name
'\s*' - regular expression, that describe separator.
parse_dates - combine columns 0 and 1, convert resulting column to dates and give it the name "dates"
header - don't use 0 row as header
index_col - make column 0 (this will be already result of the parse_dates parsing)
squeeze - create Series instead of DataFrame.
Can you write Python code for it?

NAO

aonao = pd.DataFrame({'AO':AO, 'NAO':NAO})

aonao.head(10)

aonao = pd.DataFrame({'AO':AO.to_period(freq='M'),
                      'NAO':NAO.to_period(freq='M')} )

aonao.head(10)

aonao['2013-12':'2014-05']

NAO[NAO > 0]


I am working on the file "Fall_2018/LB14/py_intro_lb_nb_plotting.ipynb".
The first task is:
Xarray examples (http://xarray.pydata.org/en/stable/plotting.html)
Can you write Python code for it?

import xarray as xr
airtemps = xr.tutorial.load_dataset('air_temperature')
airtemps

# Convert to celsius
air = airtemps.air - 273.15
# copy attributes to get nice figure labels and change Kelvin to Celsius
air.attrs = airtemps.air.attrs
air.attrs['units'] = 'deg C'

air.sel(lat=50, lon=225).plot()

fig, axes = plt.subplots(ncols=2)
air.sel(lat=50, lon=225).plot(ax=axes[0])
air.sel(lat=50, lon=225).plot.hist(ax=axes[1])
plt.tight_layout()
plt.show()

air.sel(time='2013-09-03T00:00:00').plot()

# Faceting
# Plot evey 250th point
air.isel(time=slice(0, 365 * 4, 250)).plot(x='lon', y='lat', col='time', col_wrap=3)

# Overlay data on cartopy map
ax = plt.axes(projection=ccrs.Orthographic(-80, 35))
air.isel(time=0).plot.contourf(ax=ax, transform=ccrs.PlateCarree());
ax.set_global(); ax.coastlines();


I am working on the file "Fall_2018/LB22/NeuralNetDemo.ipynb".
The first task is:
Before feeding a neural net, we need to set up our data. First, we note that these data are sorted by class (the last column, 0s and 1s), as is often the case with pre-classified data. It is good practice to shuffle the data to help ensure that the distribution of classes is roughly the same in both the training and testing subsets.
Can you write Python code for it?


I am working on the file "Fall_2018/LB22/NeuralNetDemo.ipynb".
The first task is:
Finally, replace targets 1 and 0 with 0.8 and 0.2. (Why?)
Can you write Python code for it?

#train_targets[np.where(train_targets==1)] = 0.8
#train_targets[np.where(train_targets==0)] = 0.2

#test_targets[np.where(test_targets==1)] = 0.8
#test_targets[np.where(test_targets==0)] = 0.2


I am working on the file "Fall_2018/LB22/NeuralNetDemo.ipynb".
The first task is:
Let's test this on a data set from the UC Irving Machine Learning Respository (https://archive.ics.uci.edu/ml/index.php). This handy respository contains hundreds of pre-classified data sets that are ideal for testing machine learning algorithms.
Consider the banknote authentication data set. The documentation states that it contains 1372 examples of banknote-like specimens that are classified as either authentic (class 1) or not authentic (class 0) based on four attributes: variance, skewness, curtosis, and entropy of the image.
Can you write Python code for it?

data = pd.read_csv("~/Documents/Classes/RSMAS/MachLearn/Project1/banknote.csv",
                   names=['var', 'skew', 'curt', 'ent', 'class-yes'])

data.head(3)

data.tail(3)


I am working on the file "Fall_2018/LB22/NeuralNetDemo.ipynb".
The first task is:
Next we need to normalize our data such that all columns are between 0 and 1:
Can you write Python code for it?

data_norm = (data - data.min(axis=0)) / (data.max(axis=0) - data.min(axis=0))


I am working on the file "Fall_2018/LB22/NeuralNetDemo.ipynb".
The first task is:
Now train the neural net!
Can you write Python code for it?

I2H, H2O, bHID, bOUT = nnet(examples=train_examples, targets=train_targets,
                            hidden=5, learn=0.05)


I am working on the file "Fall_2018/LB22/NeuralNetDemo.ipynb".
The first task is:
Now, split the data into training and testing subsets. Here we choose to use 80% of the examples for training and 20% for testing. Finally, separate into examples, which will contain the attributes, and the targets, which will contain the class flag.
Can you write Python code for it?

train, test = model_selection.train_test_split(data_norm, test_size = 0.2)

train_examples = np.array(train.iloc[:,0:4], dtype=np.float64)
train_targets = np.array(train[['class-yes', 'class-no']], dtype=np.float64)

test_examples = np.array(test.iloc[:,0:4], dtype=np.float64)
test_targets = np.array(test[['class-yes', 'class-no']], dtype=np.float64)


I am working on the file "Fall_2018/LB22/NeuralNetDemo.ipynb".
The first task is:
Remember that python uses zero-based indexing. Thus, a 0 here means the first column contains the highest number, meaning the example is authentic. This is admittedly a little confusing. How does this compare to the actual testing data? The neural net is correct if the predicted class agrees with the actual class:
Can you write Python code for it?

sum(target_classes == predicted_classes)/len(test_targets) * 100


I am working on the file "Fall_2018/LB22/NeuralNetDemo.ipynb".
The first task is:
We can think of these data as having two classes: authentic and not authentic. Our data set already contains a column that contains 1s whenever the example is in the authentic class. Now let's add a column that has a 1 when the example is in the not authentic class. Then print the first few lines to make sure we have what we think we have.
Can you write Python code for it?

data['class-no'] = np.where(data['class-yes']==0, 1, 0)

data.head()


I am working on the file "Fall_2018/LB22/NeuralNetDemo.ipynb".
The first task is:
What do these numbers mean? Remember that our target matrix consists of 0.8s and 0.2s (originally 1s and 0s) and correspond to whether the example was in the class authentic or not authentic. Our neural network had 2 output neurons, one for each class. The columns of 'out' therefore represent these two classes, in the same order as in the 'data'.
Because this is a simple classification problem, we can choose for each example the class (i.e., column) with the higher number:
Can you write Python code for it?

# Predicted classes
predicted_classes = np.array(pd.DataFrame(out).idxmax(axis=1))
print(predicted_classes)

# Target classes
target_classes = np.array(pd.DataFrame(test_targets).idxmax(axis=1))
print(target_classes)


I am working on the file "Famous_Models.ipynb".
The first task is:
An Example
Can you write Python code for it?

np.random.seed(100)
pop_size = 10
no_time_steps = 100

# initial opinion of agents at time t=0
initial_opinions = abs(np.random.randn(pop_size,))

# Build the trust matrix (weighted adjacency matrix)
trust_matrix = abs(np.random.randn(pop_size,pop_size))
trust_matrix = make_stochastic(trust_matrix)

# Initialize history of evolution
r_evolution_of_opinions = np.zeros((pop_size, no_time_steps))
r_evolution_of_opinions[:,0] = initial_opinions

# Do the game
for time_step in xrange(1,no_time_steps):
    r_evolution_of_opinions[:,time_step] = np.dot(trust_matrix, r_evolution_of_opinions[:,time_step-1])

fig = plt.figure()
plt.plot(r_evolution_of_opinions[:,:20].T)
plt.xlabel('Time')
plt.ylabel('Opinionds')
plt.title('Evolution of Opinions')
plt.show()


I am working on the file "Famous_Models.ipynb".
The first task is:
DeGroot Model
Convert to row stochastic Matrix
This code uses the method of the Sinkhorn paper:
Sinkhorn, R. (1964). A relationship between arbitrary positive matrices and doubly stochastic matrices. The Annals of Mathematical Statistics, 35(2):876–879.)
Can you write Python code for it?

def converged_test_stochastic(matrix, threshold=0.0000001):
    localMatrix = np.copy(matrix).astype(float);
    e1 = sum(abs(np.sum(localMatrix , axis = 0) - 1));
    e2 = sum(abs(np.sum(localMatrix , axis = 1) - 1));
    return (e2) > threshold

def one_step_stochastic(matrix):
    """ Here we will do one step towards
        Making a given matrix a bio-stochastic one
        It does what OneStep does
    """
    # copy the input so that the original input is not changed.
    localMatrix = np.copy(matrix).astype(float);

    # Divide each row by sum of the entries in the given row.
    localMatrix = np.dot(np.diag(1/np.sum(localMatrix, axis=1)), localMatrix);
    return localMatrix

def make_stochastic(matrix):
    localMatrix = np.copy(matrix).astype(float);
    while (converged_test_stochastic(localMatrix)):
        localMatrix = one_step_stochastic(localMatrix);
    return localMatrix


I am working on the file "Fantasy (first steps).ipynb".
The first task is:
Add the fumble recoveries to the overall defensive team's stats
The overall stats are in rows for each team with the player name as TOTAL:
Can you write Python code for it?

stats[stats.Player == 'TOTAL'][[
        'Player', 'Team',
        'interceptionsINT', 'interceptionsTD',
        'pointreturnsTD', 'kickreturnsTD', 'fumblesRecoveredTD',
        'defenseSCK', 'defenseF']]


I am working on the file "Fantasy (first steps).ipynb".
The first task is:
Clean up, convert to 2015
The stats are for the previous season. Use the roster data ('data/nfl_rosters2015.csv') we have to map the player to the correct team, and add the bye week.

Rename the player 'TOTAL' to the player 'Defense' because that's what it is.
Read in 'data/nfl_rosters2015.csv' and merge it with stats to correctly map players to their 2015 team.

After we merge, we will have pairs of columns that need to be reconciled:
- 'Team' and 'Team2014' -- the player 'Defense' doesn't actually exist so after the merge the 'Team' will be null for 'Defense'...make sure the 'Team' column is populated with the 2014 value
- 'Pos' (for 2015) and 'pos' (for 2014). <br/>
  Take the 2015 value if it exists, otherwise the 2014 value
And then add the Bye Week for each player
- Use the game schedule in 'nfl_season2015.csv' to add a column Bye to indicate the player's bye week
- Read in 'data/nfl_top_100s.csv' and add a column indicating the player's rank in the 2015 list
  (it is the NFL one the players vote in. ESPN also has one.)
Can you write Python code for it?

# Rename the player 'TOTAL' to the player 'Defense' because that's what it is.
stats.ix[stats.Player == 'TOTAL', 'Player'] = 'Defense'

# Read in 'data/nfl_rosters2015.csv' and merge it with stats
stats.columns = ['Team2014' if c == 'Team' else c for c in stats.columns]
rosters2015 = pd.read_csv(os.path.join('data', 'nfl_rosters2015.csv'))
rosters2015.head()
full_stats = stats.merge(rosters2015, on='Player', how='outer')

print(full_stats.shape)
print('Total players with fantasy numbers:', full_stats.FantasyPtsTotal.count())
print('Total players on 2015 rosters with fantasy numbers:',
      ((full_stats.Team.notnull()) & (full_stats.FantasyPtsTotal.notnull())).sum())

# Reconcile 'Team' and 'Team2014' for the player 'Defense'
full_stats.ix[full_stats.Player == 'Defense', 'Team'
             ] = full_stats[full_stats.Player == 'Defense'].Team2014

# Reconcile 'Pos' (for 2015) and 'pos' (for 2014).
full_stats.ix[full_stats.Pos.isnull(), 'Pos'] = full_stats[full_stats.Pos.isnull()].pos
del full_stats['pos']

# Read in 'nfl_top_100s.csv' and add a column indicating the player's rank
# in the 2015 list that the NFL players vote on.
nfl100 = pd.read_csv(os.path.join('data', 'nfl_top_100s.csv'))
nfl100 = nfl100[nfl100.year == 2015]
nfl100.columns = nfl100.columns.str.capitalize()
nfl100.columns = ['Top100Rank' if c == 'Rank' else c for c in nfl100.columns]
full_stats = full_stats.merge(
    nfl100[['Player', 'Team', 'Top100Rank']],
    on=['Player', 'Team'],
    how='outer')

# Peek at the data
full_stats[full_stats.FantasyPtsTotal.notnull()][
    ['Team', 'Pos', 'Player', 'FantasyPtsTotal', 'Top100Rank']
].sort('Top100Rank').head()

##
# Show teams by bye week
season2015 = pd.read_csv("data/nfl_season2015.csv")
all_teams = set(season2015.homeTeam.unique())
byes = dict(ByeWeek=[], Team=[])
for wk, dataset in season2015.groupby('week'):
    bye_teams = all_teams.difference(dataset.homeTeam).difference(dataset.awayTeam)
    byes['ByeWeek'].extend([wk] * len(bye_teams))
    byes['Team'].extend(bye_teams)

byes...


I am working on the file "Fantasy (first steps).ipynb".
The first task is:
Column definitions

Team - the full team name, exactly the same as in the other datasets
player - the player name, plus team totals (TOTAL) and opponent totals (OPPONENT TOTAL)
pos - abbreviation for positions, such as QB=quarterback, K=kicker
starts - number of starts in the 2014 season
fumblesLost - fumbles lost to the opposing team in 2014
fumblesRecoveredTD - fumbles recovered for a touchdown in 2014
twoPt - two point conversion
Passing
passingATT - attempted passes in 2014
passingCOMP - completed passes in 2014
passingINT - intercepted passes in 2014
passingTD - passing touchdowns in 2014
passingYDS - passing yards in 2014

Receiving
receivingREC - receptions in 2014
receivingTD - touchdowns made off of a reception in 2014
receivingYDS - receiving yards in 2014

Rushing
rushingATT - rushing attempts in 2014
rushingTD - rushing touchdowns in 2014
rushingYDS - rushing yards in 2014

Kicking
kicking_extraPt - extra points made in 2014
kicking_FGge50A - field golds $\ge$ 50 yards attempted
kicking_FGge50M - field golds $\ge$ 50 yards made
kicking_FGlt50A - field golds $\lt$ 50 yards attempted
kicking_FGlt50M - field golds $\lt$ yards made

Defense
defenseF - fumbles forced in 2014
defenseSCK - sacks in 2014
defenseTOTAL - tackles in 2014
defenseFumblesRecovered - fumbles recovered in 2014
pointreturnsFC - fair catches on point returns in 2014
pointreturnsRETURNS - returns made on point returns in 2014
pointreturnsTD - point returns for a touchdown in 2014
interceptionsINT - interceptions in 2014
interceptionsTD - interceptions for a touchdown in 2014
interceptionsYDS - yards gained on interceptions in 2014
kickreturnsRETURNS - kick returns in 2014
kickreturnsTD - kick returns returned for a touchdown in 2014
kickreturnsYDS - yards gained during kick returns in 2014

Aggregate the fumble recovery data for the defense
In Fantasy Football, you choose a defensive team, not individual players.
The dataset we have has rows in the column player as 'TOTAL' and 'OPPONENT TOTAL' for total defensive stats, except...
Can you write Python code for it?

byteam_fumbleRecoveries = stats.groupby('Team').fumblesRecoveredTD.agg({'totalFumblesRecoveredTD': 'sum'}).reset_index()
byteam_fumbleRecoveries.describe()


I am working on the file "Fantasy (first steps).ipynb".
The first task is:
First we need to load the data
It's the total 2014 stats for people who started in NFL games last year. Thanks to TeamRankings.com. We're missing the count of safeties but hopefully they're so infrequent they don't affect rankings.
Can you write Python code for it?

##
# Player data and the season schedule (for byes)
import os

player_stats_file = os.path.join('data', 'nfl_player_stats_season2014.csv')
season2015_file = os.path.join('data', 'nfl_season2015.csv')
prior_seasons_file = os.path.join('data', 'nfl_season2008to2014.csv')

stats = pd.read_csv(player_stats_file)
season2015 = pd.read_csv(season2015_file)
prior_seasons = pd.read_csv(prior_seasons_file)  # Don't delete this, we need it later

stats.columns


I am working on the file "Fantasy (first steps).ipynb".
The first task is:
Picking strategy
The obvious advice is don't pick a backup player that's got the same bye week as your primary one.
Next, picking order could be
- by straight points (the highest rated player in one of your open slots)
- or by point differential (the biggest drop between this player
The dataset we have now is the total Fantasy points (according to the standard rules) that the player earned in the regular 2014 season.
Can you write Python code for it?

# Add the difference in total points between each player and one below,
# and each player and 10 below (worst case draft pick)
full_stats = full_stats[full_stats.Team.notnull()]
full_stats = full_stats.sort(['Pos', 'FantasyPtsTotal'])  # Don't forget to sort
full_stats['FantasyPtsDelta'] = full_stats.groupby('Pos').FantasyPtsTotal.diff(1)
full_stats['FantasyPtsDelta10'] = full_stats.groupby('Pos').FantasyPtsTotal.diff(10)
full_stats['FantasyPtsBelowBest'] = full_stats.groupby('Pos').FantasyPtsTotal.apply(
        lambda x: max(x) - x)

full_stats = full_stats.sort(['Pos', 'FantasyPtsTotal'], ascending=['True', 'False'])
full_stats.to_csv(os.path.join('excel_files', 'fantasy_points.xls'), index=False)


I am working on the file "Fantasy (first steps).ipynb".
The first task is:
Ranking the players
...finally!
Fantasy scoring
The scoring shown in the table below uses the NFL fantasy standard rules. The fixed width denotes the corresponding column name in the dataset.

Offense	Kicking	Defense

Passing

Yards: +1 / 25 yds
 `passingYDS`

Touchdowns: +4
 `passingTD`

Interceptions: -2
 `passingINT`

Rushing

Yards: +1 / 10 yds
 `rushingYDS`

Touchdowns: +6
 `rushingTD`

Receiving

Yards: +1 / 10 yds
 `receivingYDS`

Touchdowns: +6
 `receivingTD`

Fumbles recovered
for Touchdown: +6 `fumblesRecoveredTD`

2-Point Conversions: +2
 `twoPt`

Fumbles Lost: -2
 `fumblesLost`

Point Attempts Made

+1  `kicking_extraPt`

Field Goals Made

0-49 yds: +3
 `kicking_FGlt50M`

50+ yds: +5
 `kicking_FGge50M`

Sacks: +1
 `defenseSCK`

Interceptions: +2
 `interceptionsINT`

Fumbles Recovered: +2
 `defenseF`

Safeties: +2
 **no data**

Defensive Touchdowns: +6
  `interceptionsTD` and the `fumblesRecoveredTD` attributed to defensive players

Kick / Punt Return Touchdowns: +6
 `kickreturnsTD` and `pointreturnsTD`

Points Allowed

 (get by aggregating the 2014 data from `nfl_season2008to2014.csv`)

(0): +10

(1-6): +7

(7-13): +4

(14-20): +1

(21-27): +0

(28-34): -1

(35+): -4

Calculate the fantasy points for 'Points Allowed'
We have the game-by-game data from the 2014 season in the dataset...
Can you write Python code for it?

# 0. Function that will be used in (1) to assign points to each game in a season
def fantasy_points(points_allowed):
    """Calculate the fantasy score for the season, given the points allowed.

    Points:     (0)  (1-6) (7-13) (14-20) (21-27) (28-34) (35+)
                +10    +7     +4      +1      +0      -1    -4
    """
    points = 10 * (points_allowed == 0).sum()
    points += 7 * points_allowed.between(1,6).sum()
    points += 4 * points_allowed.between(7,13).sum()
    points += 1 * points_allowed.between(14,20).sum()
    points += 0 * points_allowed.between(21,27).sum()
    points += -1 * points_allowed.between(28,34).sum()
    points += -4 * (points_allowed >= 35).sum()
    return points

# 1. Assign each game from 2014 in the `prior_seasons` data frame its respective Fantasy points
defense_fantasy_points = prior_seasons[(prior_seasons.Season == 2014) & (prior_seasons.Week <= 17)
             ].groupby('Team').PointsAllowed.agg({'fantasyPA' :fantasy_points})

# reset the index (the groupby() made the 'Team' column into the index...get it out)
defense_fantasy_points.reset_index(inplace=True)

# 2 Merge it into the main 'stats' data frame
#   2a -- Add the 'player' column with value 'TOTAL'
defense_fantasy_points['Player'] = 'TOTAL'

#   2b -- Merge the 'defense_fantasy_points' into the 'stats' data frame.
#         This will add the column 'fantasyPA' to 'stats'
if 'fantasyPA' in stats.columns:
    del stats['fantasyPA']  # in case people run this cell more than once

stats = stats.merge(defense_fantasy_points, on=['Team', 'Player'], how='left')

stats[stats.fantasyPA.notnull()][['Team', 'fantasyPA']].head()


I am working on the file "Fantasy (first steps).ipynb".
The first task is:
Select only the players we want
For standard fantasy, you get 9 players on your starting roster and 6 players on the bench.
The defensive players are rolled into one (we just did that for the last relevant type of scoring play) and we don't care about the other positions.
The parenthesized abbreviations are the corresponding value in the pos column in this dataset:

Quarterback (QB):1
Running Back (RB):2
Wide Receiver (WR):2
Tight End (TE):1
Wide Receiver / Running Back:1
Kicker (K):1
Defensive Team:1
Bench:6

First, let's add a value 'DEFENSE' as the position when the player is named 'TOTAL'. Then we'll delete all of the players who aren't the kinds we can use.
Can you write Python code for it?

# These are the positions
stats.pos.unique()

# 1. Add a value 'DEFENSE' as the position when the player is named 'TOTAL'
stats.ix[stats.Player == 'TOTAL', 'pos'] = 'DEFENSE'

# 2. Delete players that aren't the types used in Fantasy
stats = stats[stats.pos.isin(('QB', 'RB', 'WR', 'TE', 'K', 'DEFENSE'))]

print(stats.shape)

stats.head(6)


I am working on the file "Fast KMeans Notebook.ipynb".
The first task is:
Let's plot the results. We keep the first 2 features for 2D plotting
Can you write Python code for it?

cmap = { 0:'k',1:'b',2:'y',3:'g',4:'r' }

fig = plt.figure(figsize=(5,5))
plt.xlim(0,1)
plt.ylim(0,1)

for i in range(nb_cluster):
    X_extract = X[labels == i]
    X1 = np.transpose(X_extract[:,0]).tolist()
    X2 = np.transpose(X_extract[:,1]).tolist()
    plt.plot(X1, X2, cmap[i%5]+'.', alpha=0.5)

mu1 = np.transpose(mu[:,0]).tolist()
mu2 = np.transpose(mu[:,1]).tolist()
plt.plot(mu1, mu2, 'ro')

plt.show()


I am working on the file "Fast KMeans Notebook.ipynb".
The first task is:
Randomly draw some samples and initial centroids. It is important to convert the lists to numpy arrays with 'double' type so the C function recognizes them.
Can you write Python code for it?

X = [[random.random() for i in range(nb_features)] for j in range(nb_samples)]
mu = random.sample(X, nb_cluster)
X = np.array(X, dtype="double")
mu = np.array(mu, dtype="double")
mu


I am working on the file "Fast KMeans Notebook.ipynb".
The first task is:
Set the initial weights for each features at 1 (all features have the same importance).
Also we need create the labels list, which will be passed to Cfast_km function to get the cluster number for each sample
Can you write Python code for it?

weights = np.array([1 for i in range(nb_features)], dtype="double")
labels = np.zeros(len(X), dtype="int")


I am working on the file "Fast KMeans Notebook.ipynb".
The first task is:
Then, set the number of target clusters, number of features and number of samples:
Can you write Python code for it?

nb_cluster = 5
nb_features = 2
nb_samples = 500
nb_cluster, nb_features,nb_samples


I am working on the file "FastRoot.ipynb".
The first task is:
A Tale of two Variables

Let's forget about the magic constant 0x5f3759df for a moment.

There are actually two magical constants in rsqrt, the other constant is the $-\frac{1}{2}$.

Hey, $\frac{1}{\sqrt{x}}$ is just $x^{-\frac{1}{2}}$, maybe they are related!

Let's look at what happens when we change these two constants.

Since we don't really know what it's doing, let's just name it as if we don't know what we're doing:
$$
\textsf{foobar}_{M,C}(x) = \textsf{l2f}\left(M + C \cdot \textsf{f2l}(x)\right)
$$
Foo, meet Bar
$$
\textsf{foobar}_{M,C}(x) = \textsf{l2f}\left(M + C \cdot \textsf{f2l}(x)\right)
$$
Can you write Python code for it?

def foobar(M, C):
    return np.vectorize(lambda x: l2f(M + C * f2l(x)))

# rsqrt(x) is instantiated with M = 0x5f3759df and C = -1/2
rsqrt = foobar(0x5f3759df, -1.0/2.0)


I am working on the file "FastRoot.ipynb".
The first task is:
An error of around $10\%$? That's like nothing!
Conclusion

The idea behind Fast Inverse Sqrt isn't hard.

In fact, you can write a "Fast Exponentiation" method for any arbitrary exponent!

We're all wizards.

Next Steps

We didn't really talk about how people went from 0x5f400000 to 0x5f3759df. There's a wealth of resources for this, but as a shameless plug, check out my Fast Inverse Sqrt the Easy(ish) Way.

We can generate arbitrary fast power approximations, but it would be cool to generate something along the style of the original inverse-sqrt program.
Can you write Python code for it?

%%html

[Double Click for Code]

// For x^(-0.5)
float qpow(float x) {
  long i  = * ( long * ) &x;
  i  = 0x5f400000 + -0.5 * i;
  return * ( float * ) &i;
}

<input type="text" id="pown" val="-0.5"/>


I am working on the file "FastRoot.ipynb".
The first task is:
Close Enough!
So what's up with the 0x5f3759df
Where did this thing come from?

Is this some alien technology long lost to our generation?

Is it super complicated and does it require advanced mathematical machinery?

Is it just plain magic that only wizards understand?

Should we open up Pandora's Box and peer inside? We are but mere mortals after all.

As it turns out, yes we can
In fact, I believe that the idea behind this technique is so easy that an $8^{th}$ grader can figure it all out.
Seriously? When I was an $8^{th}$ grader, I could barely use a calculator...
Hey, $8^{th}$ graders are pretty smart nowadays, plus, this isn't really as much of a mindfuck as you think it is.
So easy, even a caveman can do it!

Notation
Recall that the fast inverse square root method looks like
c
long i  = * ( long * ) &x;
i  = 0x5f3759df - ( i >> 1 );
return * ( float * ) &i;
Let's look at those two casts:
* *(long*)& takes a float and outputs a long
* *(float*)& takes a long and outputs a float
Let's name these two operations the float-to-long ($\textsf{f2l}: \textsf{float} \to \textsf{long}$) and the long-to-float ($\textsf{l2f}: \textsf{long} \to \textsf{float}$) operators.
$$
\textsf{rsqrt}(x) = \textsf{l2f}\left(\textsf{0x5f3759df} - \frac{1}{2} \textsf{f2l}\left(x\right)\right)
$$

Do Snakes Dream of IEEE 754?
What would this...
Can you write Python code for it?

from struct import pack, unpack

to_long    = lambda hole: unpack('i', hole)[0] # y = (long*) x
to_float   = lambda hole: unpack('f', hole)[0] # y = (float*) x

from_long  = lambda hole: pack('i', int(hole) % 0x80000000) # long*  y = &x
from_float = lambda hole: pack('f', float(hole))            # float* y = &x

hide_code_in_slideshow()

@np.vectorize
def f2l(x):
    return to_long(from_float(x))

@np.vectorize
def l2f(y):
    return to_float(from_long(y))


I am working on the file "FastRoot.ipynb".
The first task is:
Graphing Calculator Woes

This clearly suggests that all of the lines have the same slopes since they are all parallel.

We only varied the $C$ with time, which changes the slopes of the lines.

We can also definitely see the lines shift vertically over time as well, with the exception of $x^C$

Also, the distances between the solid lines do not change even though their slopes do.

Throwback to the $8^{th}$ Grade

If you think back to Algebra 1, you should remember that the equation for a line is just

$$
w = a + b \cdot z
$$

Here, we can substitute $w = \log(y)$, $z = \log(x)$, and $b = C$ to get it log-log equation:

$$
\log(y) = a + C \cdot \log(x)
$$

In the case of $y = 1 \cdot x^C$, the corresponding log-log equation is

$$
\begin{align}
\log(y) &= \log\left(1 \cdot x^C\right) \
&= \log(1) + C \cdot \log(x).
\end{align}
$$

Since all of the other lines are parallel to the $x^C$ line, we also know, for some unknown constant $\alpha$, that
$$
\log\left(\textsf{foobar}_{M,C}(x)\right) = \alpha + C \cdot \log(x)\
$$
This confirms our earlier suspicion that the slope $C$ is related to the exponent.

In fact, since the distances between the lines do not change as we vary the slope $C$ in the video, it's clear that the unknown constant $\alpha$ will only depend on the value of our magical constant $M$:
$$
\alpha = \alpha_M
$$

Eureka!

It means that for every exponent $C$, there exists some constant $M$ such that $\textsf{foobar}_{M,C}(x) \approx x^C$.

This is all we need to find meaningful "magic" constants!

In fact, you can find a magic constant for every exponent $x^C$.

Fast Exponentiation [1]
We know that for each $C$, there's some magic $M^$ such that
$$
\textsf{foobar}_{M^,C}(x) = \textsf{l2f}\left(M^* + C \cdot \textsf{f2l}(x)\right) \approx x^C
$$
How do we find $M^$? We know that for inverse square-roots, $M^$ is around 0x5f3759df.

Answer: More $8^{th}$ grade algebra.

Fast Exponentiation [2]
Question: What happens if you send $x = 1$ into $x^C$?
  * Answer: $1^C = 1$ no matter what...
Can you write Python code for it?

# What is 1#?
display(Latex(r'Just $\textsf{f2l}(1) = \textsf{%s}$.' % hex(f2l(1))))

# What about inverse square-root?
display(Latex(r'For the inverse square-root, its magical constant should be \
  $$\left(1 - \frac{-1}{2}\right)\textsf{f2l}(1) = \textsf{%s}$$'
              % hex(3 * f2l(1) // 2)))

hide_code_in_slideshow()


I am working on the file "FastRoot.ipynb".
The first task is:
Here, $$\begin{align}
\textsf{f2l}(x) &= (\textsf{long}) \textsf{&} x \
\textsf{l2f}(y) &= (\textrm{float}) \textsf{&} y
\end{align}
$$
Don't worry too much about the implementation details.
I can't believe it's not Casting!
What do we know about $\textsf{f2l}$ and $\textsf{l2f}$?

Inverse Laws:

$$
\textsf{f2l}(\textsf{l2f}(x)) = x = \textsf{l2f}(\textsf{f2l}(x))
$$
e.g. If you convert a float to a long and then convert that long back to a float, you get the same float back.

Anihilator Laws:

$$
\textsf{f2l}(0) = 0 = \textsf{l2f}(0)
$$
e.g. If you convert 0 from a float to a long, it's still 0.
There are a few other algebraic properties of f2l and l2f, but unfortunately, they're pretty unstructured. In particular, $x + y \ne \textsf{l2f}(\textsf{f2l}(x) + \textsf{f2l}(y))$:
Can you write Python code for it?


I am working on the file "FastRoot.ipynb".
The first task is:
Hey, that actually looks pretty good! But what about the errors?
Can you write Python code for it?

from matplotlib.ticker import FuncFormatter

def to_percent(y, position):
    # Ignore the passed in position. This has the effect of scaling the default
    # tick locations.
    s = str(int(100 * y))

    # The percent symbol needs escaping in latex
    if matplotlib.rcParams['text.usetex'] is True:
        return s + r'$\%$'
    else:
        return s + '%'

# Create the formatter using the function to_percent. This multiplies all the
# default labels by 100, making them all percentages
formatter = FuncFormatter(to_percent)

# ax.yaxis.set_major_formatter(formatter)
hide_code_in_slideshow()

x = np.linspace(1, 1000, 5000)
properties = {
    (0, 0): {'C': -1},
    (1, 0): {'C': 2},
    (0, 1): {'C': 0.3},
    (1, 1): {'C': -0.6},
}
fig, axarr = plt.subplots(2, 2, figsize=(14,8));
for key, property in properties.items():
    axarr[key].set_ylim(0, 0.5)
    axarr[key].yaxis.set_major_formatter(formatter)
    C = property['C']
    baz = qexp(C)
    handle, = axarr[key].plot(x, np.abs(x ** C - baz(x))/(x ** C));
    axarr[key].set_title(r'Relative error for $x^{%s}$' % C)
    axarr[key].legend(
        [handle],
        [r'Relative error for $x^{%s}$' % C])

hide_code_in_slideshow()


I am working on the file "FastRoot.ipynb".
The first task is:
Notice how as $C$ gets closer to $-\frac{1}{2}$, the 0x5f3759df line also gets closer to $x^C$.
What do you see?

All the slopes are parallel to each other.

...

Honestly, it's hard to grok anything off of a series of pictures, so let's invoke the ancient wisdom of Python and look at a video of this instead.
Can you write Python code for it?

from IPython.display import HTML
from matplotlib import animation
animation.Animation._repr_html_ = lambda anim: anim.to_html5_video()

x = np.linspace(1, 1000, 5000)
allM = (1 << 26, 1 << 28, 0x5f3759df)
fig = plt.figure(figsize=(14,8))
ax = plt.axes(ylim=(1e-39, 1e41))

def plotSomeMagic(C, fig, ax, handles=None):
    if not handles:
        handle, = ax.loglog(x, x ** C, linestyle='dotted');
        handles = [handle]
        for M in allM:
            baz = foobar(M, C)
            kwargs = {'ls' : 'dashed'} if M == 0x5f3759df else {}
            handle, = ax.loglog(x, np.abs(baz(x)), **kwargs)
            handles.append(handle)
    else:
        handles[0].set_data(x, x ** C)
        baz = foobar(allM[0], C)
        handles[1].set_data(x, np.abs(baz(x)))
        baz = foobar(allM[1], C)
        handles[2].set_data(x, np.abs(baz(x)))
        baz = foobar(allM[2], C)
        handles[3].set_data(x, np.abs(baz(x)))
    ax.set_title(r'For slope C = $%s$, ${\rm foobar}_{M,%s}(x)$' % (C, C))
    ax.legend(
        handles,
        [
            r'$x^{%s}$' % C,
            r'$M = 2^{26}$',
            r'$M = 2^{28}$',
            r'$M = {\rm 0x5f3759df}$'
        ], loc=4)
    return tuple(handles)

handles = plotSomeMagic(0, fig, ax)

# initialization function: plot the background of each frame
def init():
    return plotSomeMagic(1, fig, ax, handles)

# animation function.  This is called sequentially
def animate(i):
    return plotSomeMagic(i, fig, ax, handles)

hide_code_in_slideshow()

video = animation.FuncAnimation(fig, animate, init_func=init, frames=np.arange(-2,8,0.10), interval=100, blit=True)

plt.close();

video


I am working on the file "FastRoot.ipynb".
The first task is:
Recall that the log-log plot for $y = M \cdot x^C$ is linear because
$$\begin{align}
\log(y) &= \log\left(M x^C\right) \
&= \log(M) + C \cdot \log(x)
\end{align}$$
where $\log(y)$ varies linearly with $\log(x)$ with slope $C$.
What does foobar looks like under the log-log scale?
Can you write Python code for it?

import matplotlib
matplotlib.rcParams['text.usetex'] = False
matplotlib.rcParams['text.latex.unicode'] = False

x = np.linspace(1, 1000, 5000)
allM = (1 << 26, 1 << 28, 0x5f3759df)
properties = {
    (0, 0): {'M': allM, 'C': -2},
    (1, 0): {'M': allM, 'C': 8},
    (0, 1): {'M': allM, 'C': 0.3},
    (1, 1): {'M': allM, 'C': -0.6},
}
fig, axarr = plt.subplots(2, 2, figsize=(14,8));
for key, property in properties.items():
    C = property['C']
    axarr[key].set_ylim(1e-39, 1e41)
    handle, = axarr[key].loglog(x, x ** C, linestyle='dotted');
    handles = [handle]
    for M in property['M']:
        baz = foobar(M, C)
        kwargs = {'ls' : 'dashed'} if M == 0x5f3759df else {}
        handle, = axarr[key].loglog(x, np.abs(baz(x)), **kwargs)
        handles.append(handle)
    axarr[key].set_title(r'For slope C = $%s$, ${\rm foobar}_{M,%s}(x)$' % (C, C))
    axarr[key].legend(
        handles,
        [
            r'$x^{%s}$' % C,
            r'$M = 2^{26}$',
            r'$M = 2^{28}$',
            r'$M = {\rm 0x5f3759df}$'
        ], loc=4)

hide_code_in_slideshow()


I am working on the file "FastRoot.ipynb".
The first task is:
Reinventing Fast Inverse Roots Using  $8^{th}$ Grade Math.
Algebra, Straight Lines, and Logarithms Oh My!
By: Lee
Notebook View
Press the Right key to navigate these slides.
Can you write Python code for it?

setup_html = r'''
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/github-fork-ribbon-css/0.2.0/gh-fork-ribbon.min.css" />

'''
# IPython.display.display_html(setup_html, raw=True)
hide_code_in_slideshow()


I am working on the file "FastRoot.ipynb".
The first task is:
What's the Fast Inverse Root Method you ask?
This is actual code used in Quake and other early 3D games to help us calculate $\frac{1}{\sqrt{x}}$, seriously
c
float rsqrt(float x) {
  long i  = * ( long * ) &x;                  // floating point bit hack
  i  = 0x5f3759df - ( i >> 1 );               // what the fuck?
  return * ( float * ) &i;
}
Supposedly,
$$
\textsf{rsqrt}(x) \approx \frac{1}{\sqrt{x}}
$$
and the secret is unlocked behind this impenetrable magical constant 0x5f3759df.

Does this actually work?
Can you write Python code for it?

%matplotlib inline

from struct import pack, unpack
import numpy as np
import matplotlib.pyplot as plt

@np.vectorize
def sharp(x):
    return unpack('I', pack('f', x))[0]

@np.vectorize
def flat(y):
    return unpack('f', pack('I', int(y) & 0xffffffff))[0]

star_long_star_amp = sharp;
star_float_star_amp = flat;

hide_code_in_slideshow();

@np.vectorize
def rsqrt(x):                      # float rsqrt(float x) {
    i = star_long_star_amp(x);     #   long i  = * ( long * ) &x;
    i = 0x5f3759df - ( i >> 1 );   #   i = 0x5f3759df - ( i >> 1 );
    return star_float_star_amp(i); #   return * ( float * ) &i;
                                   # }

# Construct a plot
fig = plt.figure(figsize=(16,8));
ax = plt.axes();

# Plot the approximation and the actual inverse sqrt function
x = np.linspace(1, 50, 5000);
approximation, = ax.plot(x, rsqrt(x))
actual, = ax.plot(x, 1/np.sqrt(x))

fig.legend(handles=[approximation, actual], labels=[r'qsqrt(x)', r'$\frac{1}{\sqrt{x}}$'], fontsize=20);
fig.suptitle(r"$\frac{1}{\sqrt{x}}$ versus qsqrt(x)", fontsize=26);

hide_code_in_slideshow()


I am working on the file "Fasta_Input_File_to_Graph-read one file-submit.ipynb".
The first task is:
Experiment
Can you write Python code for it?

#Read fasta files and separate sequences from the head of the sequence
fgraph = FastaToGraph()
path = "/home/alshaikm/EDeN_examples/fasta/testfasta"
seqs = fgraph._sequeceWrapper(path)

#for seq in seqs:
#    h, s = seq
#    print ("h", h)
#    print ("s", s)


I am working on the file "Fastest_dimension_of_array.ipynb".
The first task is:
3D arrays
In a nutshell:
C order: first axis is fastest, last axis is slowest; factor of two between others.
Fortran order: last axis is fastest, first axis is slowest; factor of two between others.
Can you write Python code for it?

dim = 3

v = makend(dim, 600)
for n in range(dim):
    %timeit get_slice_3d(v, 201, n)
del v


I am working on the file "Fastest_dimension_of_array.ipynb".
The first task is:
6 dimensions and beyond
In general, first n/2 dimensions are fast, then gets slower until last dimension is several (5-ish) times slower than the first.
Can you write Python code for it?

def get_slice_6d(volume, x, axis, n=None):
    """
    Naive function... but only works on 6 dimensions.
    """
    if n is None and not np.sum(np.diff(volume.shape)):
        n = np.min(volume.shape)
    if axis == 0:
        data = volume[x, :n, :n, :n, :n, :n]
    if axis == 1:
        data = volume[:n, x, :n, :n, :n, :n]
    if axis == 2:
        data = volume[:n, :n, x, :n, :n, :n]
    if axis == 3:
        data = volume[:n, :n, :n, x, :n, :n]
    if axis == 4:
        data = volume[:n, :n, :n, :n, x, :n]
    if axis == 5:
        data = volume[:n, :n, :n, :n, :n, x]
    return data + 1

dim = 6

v = makend(dim, 23)
for n in range(dim):
    %timeit get_slice_6d(v, 12, n)
del v


I am working on the file "Fastest_dimension_of_array.ipynb".
The first task is:
A seismic volume
Can you write Python code for it?

volume = np.load('data/F3_volume_3x3_16bit.npy')

volume.shape


I am working on the file "Fastest_dimension_of_array.ipynb".
The first task is:
A tiny example
Can you write Python code for it?

a = np.arange(9).reshape(3, 3)
a

' '.join(str(i) for i in a.ravel(order='C'))

' '.join(str(i) for i in a.ravel(order='F'))


I am working on the file "Fastest_dimension_of_array.ipynb".
The first task is:
Accessing the seismic data
Let's make all the dimensions the same, to avoid having to slice later. I'll make a copy, otherwise we'll have a view of the original array.
Alternatively, change the shape here to see effect of small dimensions, eg try volume = volume[:10, :290, :290] with C ordering.
Can you write Python code for it?

volume = volume[:190, :190, :190].copy()

def get_slice_3d(volume, x, axis, n=None):
    """
    Naive function... but only works on 3 dimensions.
    NB Using ellipses slows down last axis.
    """
    # Force cube shape
    if n is None and not np.sum(np.diff(volume.shape)):
        n = np.min(volume.shape)
    if axis == 0:
        data = volume[x, :n, :n]
    if axis == 1:
        data = volume[:n, x, :n]
    if axis == 2:
        data = volume[:n, :n, x]
    return data + 1

%timeit get_slice_3d(volume, 150, axis=0)
%timeit get_slice_3d(volume, 150, axis=1)
%timeit get_slice_3d(volume, 150, axis=2)


I am working on the file "Fastest_dimension_of_array.ipynb".
The first task is:
Aside: figure for blog post
Can you write Python code for it?

fig, axs = plt.subplots(ncols=2, figsize=(10,3), facecolor='w')

for i, ax in enumerate(axs):
    data = idx[i, :3, :3, 0].ravel(order='C')
    ax.plot(data, 'o-', c='gray')
    ax.text(0, 1.8, f'axis {i}', color='gray', size=15, ha='left')
plt.savefig('/home/matt/Pictures/2d-array-corder.png')


I am working on the file "Fastest_dimension_of_array.ipynb".
The first task is:
At the risk of making it more confusing, it might help to look at the plots together. Shown here is the C ordering:
Can you write Python code for it?

plt.figure(figsize=(15,3))
plt.plot(idx[0, :5, :5, :5].ravel(), zorder=10)
plt.plot(idx[1, :5, :5, :5].ravel(), zorder=9)
plt.plot(idx[2, :5, :5, :5].ravel(), zorder=8)


I am working on the file "Fastest_dimension_of_array.ipynb".
The first task is:
Axes 0 and 1 are > 10 times faster than axis 2.
What about if we do something like take a Fourier transform over the first axis?
Can you write Python code for it?

from scipy.signal import welch

%timeit s = [welch(tr, fs=500) for tr in volume[:, 10]]

%timeit s = [welch(tr, fs=500) for tr in volumef[:, 10]]


I am working on the file "Fastest_dimension_of_array.ipynb".
The first task is:
Fake data in n dimensions
Let's make a function to generate random data in any number of dimensions.
Be careful: these volumes get big really quickly!
Can you write Python code for it?

def makend(n, s, equal=True, rev=False, fortran=False):
    """
    Make an n-dimensional hypercube of randoms.
    """
    if equal:
        incr = np.zeros(n, dtype=int)
    elif rev:
        incr = list(reversed(np.arange(n)))
    else:
        incr = np.arange(n)
    shape = incr + np.ones(n, dtype=int)*s
    a = np.random.random(shape)
    m = f"Shape: {tuple(shape)}    "
    m += f"Memory: {a.nbytes/1e6:.0f}MB    "
    m += f"Order: {'F' if fortran else 'C'}"
    print (m)
    if fortran:
        return np.asfortranarray(a)
    else:
        return a


I am working on the file "Fastest_dimension_of_array.ipynb".
The first task is:
For C ordering, the last dimension is more than 20x slower than the other two.
4 dimensions
Axes 0 and 1 are fast (for C ordering), axis 2 is half speed, axis 3 is ca. 15 times slower than fast axis.
Can you write Python code for it?

def get_slice_4d(volume, x, axis, n=None):
    """
    Naive function... but only works on 4 dimensions.
    """
    if n is None and not np.sum(np.diff(volume.shape)):
        n = np.min(volume.shape)
    if axis == 0:
        data = volume[x, :n, :n, :n]
    if axis == 1:
        data = volume[:n, x, :n, :n]
    if axis == 2:
        data = volume[:n, :n, x, :n]
    if axis == 3:
        data = volume[:n, :n, :n, x]
    return data + 1

dim = 4

v = makend(dim, 100, equal=True)
for n in range(dim):
    %timeit get_slice_4d(v, 51, n)
del v

dim = 4

v = makend(dim, 100, equal=True, fortran=True)
for n in range(dim):
    %timeit get_slice_4d(v, 51, n)
del v


I am working on the file "Fastest_dimension_of_array.ipynb".
The first task is:
I tried implementing this as a context manager, so you wouldn't have to delete the volume each time after using it. I tried the @contextmanager decorator, and I tried making a class with __enter__() and __exit__() methods. Each time, I tried putting the del command as part of the exit routine. They both worked fine... except they did not delete the volume from memory.
2D data
Can you write Python code for it?

def get_slice_2d(volume, x, axis, n=None):
    """
    Naive function... but only works on 2 dimensions.
    """
    if n is None and not np.sum(np.diff(volume.shape)):
        n = np.min(volume.shape)
    if axis == 0:
        data = volume[x, :n]
    if axis == 1:
        data = volume[:n, x]
    return data + 1

dim = 2

v = makend(dim, 6000, fortran=False)
for n in range(dim):
    %timeit get_slice_2d(v, 3001, n)
del v

dim = 2

v = makend(dim, 6000, fortran=True)
for n in range(dim):
    %timeit get_slice_2d(v, 3001, n)
del v


I am working on the file "Fastest_dimension_of_array.ipynb".
The first task is:
Let's check that changing the memory layout to Fortran ordering makes the last dimension fastest:
Can you write Python code for it?

volumef = np.asfortranarray(volume)

%timeit get_slice_3d(volumef, 150, axis=0)
%timeit get_slice_3d(volumef, 150, axis=1)
%timeit get_slice_3d(volumef, 150, axis=2)


I am working on the file "Fastest_dimension_of_array.ipynb".
The first task is:
Let's look at how the indices vary:
Can you write Python code for it?

idx = np.indices(volume.shape)

idx.shape


I am working on the file "Fastest_dimension_of_array.ipynb".
The first task is:
Let's see how this looks on our data:
Can you write Python code for it?

dim = 2

v = makend(dim, 6000, fortran=False)
%timeit np.mean(v, axis=0)
%timeit np.mean(v, axis=1)
del v

dim = 2

v = makend(dim, 6000, fortran=True)
%timeit np.mean(v, axis=0)
%timeit np.mean(v, axis=1)
del v


I am working on the file "Fastest_dimension_of_array.ipynb".
The first task is:
No practical difference. Hm.
I'm guessing this is because the DFT takes way longer than the data access.
Can you write Python code for it?


I am working on the file "Fastest_dimension_of_array.ipynb".
The first task is:
Speed is double on fast axis, i.e. second axis on default C order.
np.mean() across axes
Let's try taking averages across different axes. In C order it should be faster to get the mean on axis=1 because that involves getting the rows:
Can you write Python code for it?


I am working on the file "Fastest_dimension_of_array.ipynb".
The first task is:
This has been between 3.3 and 12 times faster.
1D convolution on an array
Can you write Python code for it?

def convolve(data, kernel=np.arange(10), axis=0):
    func = lambda tr: np.convolve(tr, kernel, mode='same')
    return np.apply_along_axis(func, axis=axis, arr=data)

dim = 2

v = makend(dim, 6000, fortran=False)
%timeit convolve(v, axis=0)
%timeit convolve(v, axis=1)
del v

dim = 2

v = makend(dim, 6000, fortran=True)
%timeit convolve(v, axis=0)
%timeit convolve(v, axis=1)
del v


I am working on the file "Fastest_dimension_of_array.ipynb".
The first task is:
What about when we're doing something like getting the mean on an array?
Can you write Python code for it?

dim = 5

v = makend(dim, 40, fortran=True)
for n in range(dim):
    %timeit np.mean(v, axis=n)
del v


I am working on the file "Feature Co-Occurrence.ipynb".
The first task is:
Here is the impact on the value for 'arthropod' in one document, using the two transformations above.
Can you write Python code for it?

print 'Before: '.ljust(15), corpus.features['abstract'].features['WOS:000324532900018'].value('arthropod')
print 'TF*IDF: '.ljust(15), corpus.features['abstracts_tfidf'].features['WOS:000324532900018'].value('arthropod')
print 'mytransformer: '.ljust(15), corpus.features['abstracts_transformed'].features['WOS:000324532900018'].value('arthropod')


I am working on the file "Feature Co-Occurrence.ipynb".
The first task is:
I can specify some other transformation by first defining a transformer function, and then passing it as an argument to transform. A transformer function should accept the following parameters, and return a single numerical value (int or float).
| Parameter | Description                                                       |
| --------- | ----------------------------------------------------------------- |
| f     | Representation of the feature (e.g. string).                      |
| v     | Value of the feature in the document (e.g. frequency).            |
| C     | Value of the feature in the Corpus (e.g. global frequency).   |
| DC    | Number of documents in which the feature occcurs.                 |
For example:
Can you write Python code for it?

def mytransformer(s, c, C, DC):
    """
    Doubles the feature value and divides by the overall value in the Corpus.
    """
    return c*2./(C)


I am working on the file "Feature Co-Occurrence.ipynb".
The first task is:
Networks of features based on co-occurrence
The features module in the tethne.networks subpackage contains a few functions for generating networks of features based on co-occurrence.
Can you write Python code for it?


I am working on the file "Feature Co-Occurrence.ipynb".
The first task is:
The mutual_information function in the features module generates a network based on the pointwise mutual information of each pair of features in a featureset.
The first argument is a list of Papers, just like most other network-building functions. The second argument is the featureset that we wish to use.
Can you write Python code for it?

MI_graph = features.mutual_information(corpus, 'abstracts_filtered', min_weight=0.7)


I am working on the file "Feature Engineering/KMeans Impute Missing Data.ipynb".
The first task is:
Example with fake data
Can you write Python code for it?

import matplotlib
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
matplotlib.style.use('ggplot')

from sklearn.datasets import make_blobs
from matplotlib import pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

def make_fake_data(fraction_missing, n_clusters=5, n_samples=5000,
                   n_features=3, seed=0):
    # complete data
    gen = np.random.RandomState(seed)
    X, true_labels = make_blobs(n_samples, n_features, n_clusters,
                                random_state=gen)
    # with missing values
    missing = gen.rand(*X.shape) < fraction_missing
    Xm = np.where(missing, np.nan, X)
    return X, true_labels, Xm

X, true_labels, Xm = make_fake_data(fraction_missing=0.1, n_clusters=6, seed=10)

labels, centroids, X_hat, X_gm = kmeans_missing(Xm, n_clusters=6, max_iter=40)

fig = plt.figure(figsize=(18,12))
ax = fig.add_subplot(221, projection='3d')
ax.scatter(X[:, 0], X[:, 1], X[:, 2], c=true_labels, cmap='gist_rainbow')
ax.set_title('Original data')
ax = fig.add_subplot(222, projection='3d')
ax.scatter(Xm[:, 0], Xm[:, 1], Xm[:, 2], c=true_labels, cmap='gist_rainbow')
ax.set_title('Missing data(30% missing values)')
ax = fig.add_subplot(223, projection='3d')
ax.scatter(X_gm[:, 0], X_gm[:, 1], X_gm[:, 2], c=true_labels, cmap='gist_rainbow')
ax.set_title('Imputed data using global mean values')
ax = fig.add_subplot(224, projection='3d')
ax.scatter(X_hat[:, 0], X_hat[:, 1], X_hat[:, 2], c=true_labels, cmap='gist_rainbow')
ax.set_title('Imputed data using cluster algorithm')


I am working on the file "Feature Engineering/KMeans Impute Missing Data.ipynb".
The first task is:
可以看出，采用全局平均值填充缺失值后，数据分布差别较大；相反采用聚类算法填充缺失值的效果较好。
Performance Benchmark
The Mutual Information (互信息量) can be useful to measure the agreement of two independent label assignments strategies on the same dataset when the real ground truth is not known.
Can you write Python code for it?

from sklearn.metrics import adjusted_mutual_info_score

cls = KMeans(n_clusters=5, n_jobs=-1)

true_labels = cls.fit_predict(X)
global_labels = cls.fit_predict(X_gm)
kmeans_labels = cls.fit_predict(X_hat)

print '全局平均值填充后的实际labels 与 聚类后的 labels 的互信息量：', adjusted_mutual_info_score(true_labels, global_labels)
print 'KMeans填充后的实际labels 与 聚类后的 labels 的互信息量：', adjusted_mutual_info_score(true_labels, kmeans_labels)


I am working on the file "Feature Selection.ipynb".
The first task is:
Beware of BUGS!!!
When Katie was working on the Enron POI identifier, she engineered a feature that identified when a given person was on the same email as a POI. So for example, if Ken Lay and Katie Malone are both recipients of the same email message, then Katie Malone should have her "shared receipt" feature incremented. If she shares lots of emails with POIs, maybe she's a POI herself.
Here's the problem: there was a subtle bug, that Ken Lay's "shared receipt" counter would also be incremented when this happens. And of course, then Ken Lay always shares receipt with a POI, because he is a POI. So the "shared receipt" feature became extremely powerful in finding POIs, because it effectively was encoding the label for each person as a feature.
We found this first by being suspicious of a classifier that was always returning 100% accuracy. Then we removed features one at a time, and found that this feature was driving all the performance. Then, digging back through the feature code, we found the bug outlined above. We changed the code so that a person's "shared receipt" feature was only incremented if there was a different POI who received the email, reran the code, and tried again. The accuracy dropped to a more reasonable level.
We take a couple of lessons from this:
- Anyone can make mistakes--be skeptical of your results!
- 100% accuracy should generally make you suspicious. Extraordinary claims require extraordinary proof.
- If there's a feature that tracks your labels a little too closely, it's very likely a bug!
- If you're sure it's not a bug, you probably don't need machine learning--you can just use that feature alone to assign labels.
Feature Selection Mini Project
Can you write Python code for it?

sys.path.append(dataPath+'text_learning/')

words_file = "your_word_data.pkl"
authors_file = "your_email_authors.pkl"
word_data = pickle.load( open(words_file, "r"))
authors = pickle.load( open(authors_file, "r") )

### test_size is the percentage of events assigned to the test set (the
### remainder go into training)
### feature matrices changed to dense representations for compatibility with
### classifier functions in versions 0.15.2 and earlier

from sklearn import cross_validation
features_train, features_test, labels_train, labels_test = cross_validation.train_test_split(word_data,
                                                                            authors, test_size=0.1, random_state=42)

from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer = TfidfVectorizer(sublinear_tf=True, max_df=0.5, stop_words='english')
features_train = vectorizer.fit_transform(features_train)
features_test  = vectorizer.transform(features_test).toarray()

### a classic way to overfit is to use a small number
### of data points and a large number of features;
### train on only 150 events to put ourselves in this regime

features_train = features_train[:150].toarray()
labels_train   = labels_train[:150]


I am working on the file "Feature Selection.ipynb".
The first task is:
get email of author
compare to list of known persons of interest
return boolean if author is person of interest
aggregate count over all emails to person
Can you write Python code for it?

from __future__ import division

data_point = data_dict['METTS MARK']
frac = data_point["from_poi_to_this_person"] / data_point["to_messages"]
print frac

def computeFraction( poi_messages, all_messages ):
    """ given a number messages to/from POI (numerator)
        and number of all messages to/from a person (denominator),
        return the fraction of messages to/from that person
        that are from/to a POI

   """
    ### you fill in this code, so that it returns either
    ###     the fraction of all messages to this person that come from POIs
    ###     or
    ###     the fraction of all messages from this person that are sent to POIs
    ### the same code can be used to compute either quantity

    ### beware of "NaN" when there is no known email address (and so
    ### no filled email features), and integer division!
    ### in case of poi_messages or all_messages having "NaN" value, return 0.

    fraction = 0

    if poi_messages != 'NaN':
        fraction = float(poi_messages) / float(all_messages)

    return fraction

submit_dict = {}
for name in data_dict:

    data_point = data_dict[name]

    from_poi_to_this_person = data_point["from_poi_to_this_person"]
    to_messages = data_point["to_messages"]
    fraction_from_poi = computeFraction( from_poi_to_this_person, to_messages )
    print'{:5}{:35}{:.2f}'.format('FROM ', name, fraction_from_poi)
    data_point["fraction_from_poi"] = fraction_from_poi

    from_this_person_to_poi = data_point["from_this_person_to_poi"]
    from_messages = data_point["from_messages"]
    fraction_to_poi = computeFraction( from_this_person_to_poi, from_messages )
    #print fraction_to_poi
    print'{:5}{:35}{:.2f}'.format('TO: ', name, fraction_to_poi)
    submit_dict[name]={"from_poi_to_this_person":fraction_from_poi,
                       "from_this_person_to_poi":fraction_to_poi}
    data_point["fraction_to_poi"] = fraction_to_poi

#####################

def submitDict():
    return submit_dict


I am working on the file "Feature Selector Usage.ipynb".
The first task is:
1. Missing Values
The first feature selection method is straightforward: find any columns with a missing fraction greater than a specified threshold. For this example we will use a threhold of 0.6 which corresponds to finding features with more than 60% missing values. (This method does not one-hot encode the features first).
Can you write Python code for it?


I am working on the file "Feature Selector Usage.ipynb".
The first task is:
3. Collinear (highly correlated) Features
This method finds pairs of collinear features based on the Pearson correlation coefficient. For each pair above the specified threshold (in terms of absolute value), it identifies one of the variables to be removed. We need to pass in a correlation_threshold.
This method is based on code found at https://chrisalbon.com/machine_learning/feature_selection/drop_highly_correlated_features/
For each pair, the feature that will be removed is the one that comes last in terms of the column ordering in the dataframe. (This method does not one-hot encode the data beforehand unless one_hot=True. Therefore correlations are only calculated between numeric columns)
Can you write Python code for it?

fs.identify_collinear(correlation_threshold=0.975)

correlated_features = fs.ops['collinear']
correlated_features[:5]


I am working on the file "Feature Selector Usage.ipynb".
The first task is:
5. Low Importance Features
This method builds off the feature importances from the gradient boosting machine (identify_zero_importance must be run first) by finding the lowest importance features not needed to reach a specified cumulative total feature importance. For example, if we pass in 0.99, this will find the lowest important features that are not needed to reach 99% of the total feature importance.
When using this method, we must have already run identify_zero_importance and need to pass in a cumulative_importance that accounts for that fraction of total feature importance.
Note of caution: this method builds on the gradient boosting model features importances and again is non-deterministic. I advise running these two methods several times with varying parameters and testing each resulting set of features rather than picking one number and sticking to it.
Can you write Python code for it?


I am working on the file "Feature Selector Usage.ipynb".
The first task is:
Example Dataset
This dataset was used as part of the Home Credit Default Risk competition on Kaggle. It is intended for a supervised machine learning classification task where the objective is to predict if a client will default on a loan. The entire dataset can be downloaded [here] and we will work with a small sample of 10,000 rows.
The feature selector was designed to be used for machine learning tasks, but can be applied to any dataset.\ The feature importance based methods do require a supervised machine learning problem.
Can you write Python code for it?

train = pd.read_csv('data/credit_example.csv')
train_labels = train['TARGET']
train.head()


I am working on the file "Feature Selector Usage.ipynb".
The first task is:
For detailed information on the missing fractions, we can access the missing_stats attribute, a dataframe of the missing fractions for all features.
Can you write Python code for it?


I am working on the file "Feature Selector Usage.ipynb".
The first task is:
Handling One-Hot Features
If we look at the dataframe that is returned, we may notice several new columns that were not in the original data. These are created when the data is one-hot encoded for machine learning. To remove all the one-hot features, we can pass in keep_one_hot = False to the remove method.
Can you write Python code for it?

train_removed_all = fs.remove(methods = 'all', keep_one_hot=False)

print('Original Number of Features', train.shape[1])
print('Final Number of Features: ', train_removed_all.shape[1])


I am working on the file "Feature Selector Usage.ipynb".
The first task is:
Implementation
The FeatureSelector has five functions for identifying columns to remove:

identify_missing
identify_single_unique
identify_collinear
identify_zero_importance
identify_low_importance

These methods find the features to drop according to specified criteria. The identified features are stored in the ops attribute (a Python dictionary) of the FeatureSelector. We can remove the identified features manually or use the remove function in the  FeatureSelector for actually removing the features.
Create the Instance
The FeatureSelector only requires a dataset with observations in the rows and features in the columns (standard structured data). We are working with a classifified machine learning problem so we also pass in training labels.
Can you write Python code for it?

fs = FeatureSelector(data = train, labels = train_labels)


I am working on the file "Feature Selector Usage.ipynb".
The first task is:
Removing Features
Once we have identified the features to remove, we have a number of ways to drop the features. We can access any of the feature lists in the removal_ops dictionary and remove the columns manually. We also can use the remove method, passing in the methods that identified the features we want to remove.
This method returns the resulting data which we can then use for machine learning. The original data will still be accessible in the data attribute of the Feature Selector.
Be careful of the methods used for removing features! It's a good idea to inspect the features that will be removed before using the remove function.
Can you write Python code for it?

train_no_missing = fs.remove(methods = ['missing'])

train_no_missing_zero = fs.remove(methods = ['missing', 'zero_importance'])


I am working on the file "Feature Selector Usage.ipynb".
The first task is:
Running the gradient boosting model requires one hot encoding the features. These features are saved in the one_hot_features attribute of the FeatureSelector. The original features are saved in the base_features.
Can you write Python code for it?

one_hot_features = fs.one_hot_features
base_features = fs.base_features
print('There are %d original features' % len(base_features))
print('There are %d one-hot features' % len(one_hot_features))


I am working on the file "Feature Selector Usage.ipynb".
The first task is:
The data attribute of the FeatureSelector holds the original dataframe. After one-hot encoding, the data_all attribute holds the original data plus the one-hot encoded features.
Can you write Python code for it?


I am working on the file "Feature Selector Usage.ipynb".
The first task is:
There are several categorical columns in the dataset. The FeatureSelector handles these using one-hot encoding when using the feature importance based methods.
Can you write Python code for it?

train = train.drop(columns = ['TARGET'])


I am working on the file "Feature Selector Usage.ipynb".
The first task is:
To plot all of the correlations in the data, we can pass in plot_all = True to the plot_collinear function.
Can you write Python code for it?

fs.plot_collinear(plot_all=True)

fs.identify_collinear(correlation_threshold=0.98)
fs.plot_collinear()


I am working on the file "Feature Selector Usage.ipynb".
The first task is:
To remove the features from all of the methods, pass in method='all'. Before we do this, we can check how many features will be removed using check_removal. This returns a list of all the features that have been idenfitied for removal.
Can you write Python code for it?

all_to_remove = fs.check_removal()
all_to_remove[10:25]


I am working on the file "Feature Selector Usage.ipynb".
The first task is:
We can also plot a histogram of the missing column fraction for all columns in the dataset.
Can you write Python code for it?


I am working on the file "Feature Selector Usage.ipynb".
The first task is:
We can plot a histogram of the number of unique values in each feature of the dataset.
Can you write Python code for it?


I am working on the file "Feature Selector Usage.ipynb".
The first task is:
We can view a heatmap of the correlations above the threhold. The features which will be dropped are on the x-axis.
Can you write Python code for it?


I am working on the file "Feature Selector Usage.ipynb".
The first task is:
We could use these results to select only the 'n' most important features. For example, if we want the top 100 most importance, we could do the following.
Can you write Python code for it?

one_hundred_features = list(fs.feature_importances.loc[:99, 'feature'])
len(one_hundred_features)


I am working on the file "Feature_Selection/ipython_notebook/ex2_Recursive_feature_elimination.ipynb".
The first task is:
(四)原始碼
Python source code: plot_rfe_digits.py
Can you write Python code for it?

print(__doc__)

from sklearn.svm import SVC
from sklearn.datasets import load_digits
from sklearn.feature_selection import RFE
import matplotlib.pyplot as plt

# Load the digits dataset
digits = load_digits()
X = digits.images.reshape((len(digits.images), -1))
y = digits.target

# Create the RFE object and rank each pixel
svc = SVC(kernel="linear", C=1)
rfe = RFE(estimator=svc, n_features_to_select=1, step=1)
rfe.fit(X, y)
ranking = rfe.ranking_.reshape(digits.images[0].shape)

# Plot pixel ranking
plt.matshow(ranking, cmap=plt.cm.Blues)
plt.colorbar()
plt.title("Ranking of pixels with RFE")
plt.show()


I am working on the file "Feature_Selection/ipython_notebook/ex2_Recursive_feature_elimination.ipynb".
The first task is:
可以用方法ranking_來看輸入的特徵權重關係。而方法estimator_可以取得訓練好的分類機狀態。比較特別的是當我們核函數是以線性來做分類時，estimator_下的方法coef_即為特徵的分類權重矩陣。權重矩陣的大小會因為n_features_to_select與資料的分類類別而改變，譬如本範例是十個數字的分類，並選擇以一個特徵來做分類訓練，就會得到45*1的係數矩陣，其中45是從分類類別所需要的判斷式而來，與巴斯卡三角形的第三層數正比。
(三)畫出每個像素所對應的權重順序
取得每個像素位置對於判斷數字的權重順序後，我們把權重順序依照顏色畫在對應的位置，數值愈大代表該像素是較不重要之特徵。由結果來看，不重要之特徵多半位於影像之外圍部份。而所有的訓練影像中，外圍像素多半為空白，因此較不重要。
Can you write Python code for it?

# Plot pixel ranking
plt.matshow(ranking, cmap=plt.cm.Blues)
plt.colorbar()
plt.title("Ranking of pixels with RFE")
plt.show()


I am working on the file "Features of different class of neurons.ipynb".
The first task is:
Class1: Interneuron
An indivisual neuron
Can you write Python code for it?

#loc1 = "/Volumes/Arch/Projects/Computational Anatomy/neuron_nmo/poorthuis/CNG version/060110-LII-III.CNG.swc"
loc1 = "../Generative-Models-of-Neuron-Morphology/Data/Pyramidal/poorthuis/CNG version/060110-LV.CNG.swc"
loc2 = "../Generative-Models-of-Neuron-Morphology/Data/Interneuron/allen cell types/CNG version/Pvalb-IRES-Cre-Ai14-475465561.CNG.swc"
pyramidal = McNeuron.Neuron(file_format = 'swc', input_file=loc1)
inter = McNeuron.Neuron(file_format = 'swc', input_file=loc2)
a = pyramidal.subsample(20.)
McNeuron.visualize.plot_2D(a,show_radius=True)
print len(a.nodes_list)
a.show_features(15,17,30)

btmorph3.visualize.plot_2D(inter,show_radius=False)

len(inter.nodes_list)


I am working on the file "Features of different class of neurons.ipynb".
The first task is:
Global Angle
Global angles are the angles between the vector with the starting point of one compartment and end point of its child, and the vector that connect it to $soma$.
Can you write Python code for it?

f, (ax1, ax2) = plt.subplots(1, 2)
a = pyramidal.angle_global
b = ax1.hist(a[~np.isnan(a)],bins = 50,color = 'g')
ax1.set_xlabel('angle (radian)')
ax1.set_ylabel('density')
#plt.title('Histogram of global angles')

a = inter.angle_global
b = ax2.hist(a[~np.isnan(a)],bins = 50,color = 'm')
ax2.set_xlabel('angle (radian)')

a = inter.angle_global
b = plt.hist(a[~np.isnan(a)],bins = 50,color = 'm')
plt.xlabel('angle (radian)')


I am working on the file "Features of different class of neurons.ipynb".
The first task is:
Histogram of Slope of each segments
By looking at the conneceted compartments, we can calculate the slope of the segment by dividing the diffrernce of radius and difference of the location. For many of them the slope is zero and we ignor them.
Can you write Python code for it?

f, (ax1, ax2) = plt.subplots(1, 2)
e = pyramidal.slope
x = ax1.hist(e[e!=0],bins=40,color = 'g')
ax1.set_xlabel('Value of Slope')
ax1.set_ylabel('density')

e = inter.slope
x = ax2.hist(e[e!=0],bins=40,color = 'm')
ax2.set_xlabel('Value of Slope')
#ax2.set_ylabel('density')


I am working on the file "Features of different class of neurons.ipynb".
The first task is:
Local Angle
Local angles are the angles between the vector with the starting point of one compartment and end point of its child, and the vector that connect it to $its$ $parent$.
Can you write Python code for it?

f, (ax1, ax2) = plt.subplots(1, 2)
a = pyramidal.local_angle
b = ax1.hist(a[~np.isnan(a)],bins = 50,color = 'g')
ax1.set_xlabel('angle (radian)')
ax1.set_ylabel('density')
#plt.title('Histogram of local angles')
a = inter.local_angle
b = ax2.hist(a[~np.isnan(a)],bins = 50,color = 'm')
ax2.set_xlabel('angle (radian)')
#ax2.set_ylabel('density')

a = inter.local_angle
b = plt.hist(a[~np.isnan(a)],bins = 50,color = 'm')
plt.xlabel('angle (radian)')


I am working on the file "Features of different class of neurons.ipynb".
The first task is:
Morphology of the neurons
The first one is pyramidal neuron and second is interneuron
Can you write Python code for it?

ax1 = McNeuron.visualize.plot_2D(pyramidal, show_radius=False)
ax2 = McNeuron.visualize.plot_2D(inter, show_radius=False)


I am working on the file "Features of different class of neurons.ipynb".
The first task is:
Slope
Can you write Python code for it?

f, (ax1, ax2) = plt.subplots(1, 2)
a = pyramidal.slope
b = ax1.hist(a[~np.isnan(a)],bins = 40,color = 'g')

a = inter.slope
b = ax2.hist(a[~np.isnan(a)],bins = 40,color = 'm')

a = inter.slope
b = plt.hist(a[~np.isnan(a)],bins = 40,color = 'm')


I am working on the file "Features of different class of neurons.ipynb".
The first task is:
distance from parent
Can you write Python code for it?

f, (ax1, ax2) = plt.subplots(1, 2)
a = pyramidal.length_to_parent
b = ax1.hist(a[~np.isnan(a)],bins = 40,color = 'g')

a = inter.length_to_parent
b = ax2.hist(a[~np.isnan(a)],bins = 40,color = 'm')

a = inter.length_to_parent
b = a[~np.isnan(a)]
c = plt.hist(b[np.absolute(b)<4],bins = 70,color = 'm')

np.absolute(b)<3


I am working on the file "Fibonacci numbers.ipynb".
The first task is:
Better Test
We've proven these provide the output we desire, but is there another way to test whether our result is a Fibonacci Number?
By definition, "A number is Fibonacci if and only if one or both of (5n^2 + 4) or (5n^2 – 4) is a perfect square"
First, let's define our perfect square function, then run it against our output.
Can you write Python code for it?

import math

# returns true if x is perfect square, else false
def isPerfectSquare(x):
    s = int(math.sqrt(x))
    return s*s == x

def is_PS(n):
    # n is Fibinacci if one of 5*n*n + 4 or 5*n*n - 4 or both
    # is a perferct square
    return isPerfectSquare(5*n*n + 4) or isPerfectSquare(5*n*n - 4)

def isFibonacci(n):
    if (is_PS(n) == True):
        print(n,"is a Fibonacci Number")
    else:
        print(n,"is a not Fibonacci Number")

isFibonacci(r[0])


I am working on the file "Fibonacci numbers.ipynb".
The first task is:
Dynamic Programming Method
Can you write Python code for it?

FibArray = [0,1]

def Fibonacci_DP(n):
    if n<=0:
        print("Incorrect input")
    elif n<=len(FibArray):
        return FibArray[n-1]
    else:
        temp_fib = Fibonacci_DP(n-1)+Fibonacci_DP(n-2)
        FibArray.append(temp_fib)
        return temp_fib


I am working on the file "Fibonacci numbers.ipynb".
The first task is:
Finally, what if you want to use user input for this? Say you are on a virtual call and they want to run the program themselves and only enter a number, and have it provide the result, as well as confirmations that the result it truly Fibonacci?
Finding the nth Fibonacci Number
Can you write Python code for it?

def Fibonacci_R(n):
    if n<=0:
        print("Incorrect input")
    # First Fibonacci number is 0
    elif n==1:
        return 0
    # Second Fibonacci number is 1
    elif n==2:
        return 1
    else:
        return Fibonacci_R(n-1)+Fibonacci_R(n-2)
txt = int(input("To find the nth Fibonacci Number, type in an integer for that position: "))
print("Your integer",txt,"references the Fibonacci Number",Fibonacci_R(txt))


I am working on the file "Fibonacci numbers.ipynb".
The first task is:
Is this number a Fibonacci Number?
Can you write Python code for it?

txt = int(input("Please enter an integer to see if it is a Fibonacci Number "))
import math
def is_PS(n):
    # n is Fibinacci if one of 5*n*n + 4 or 5*n*n - 4 or both
    # is a perferct square
    return isPerfectSquare(5*n*n + 4) or isPerfectSquare(5*n*n - 4)

def isFibonacci(n):
    if (is_PS(n) == True):
        print("Your integer",txt,"is a Fibonacci Number")
    else:
        print("Your integer",txt,"is a not Fibonacci Number")
isFibonacci(txt)


I am working on the file "Fibonacci numbers.ipynb".
The first task is:
Space Optimized Method
Can you write Python code for it?

def Fibonacci_SO(n):
    a = 0
    b = 1
    if n <= 0:
        print("Incorrect input")
    elif n == 1:
        return b
    else:
        for i in range(2,n):
            c = a + b
            a = b
            b = c
        return b


I am working on the file "Fibonacci numbers.ipynb".
The first task is:
The Test
Let's test these
Can you write Python code for it?


I am working on the file "Fibonacci.ipynb".
The first task is:
Dificultades algorítmicas
A continuación se miden los tiempos de ejecución de ambas implementaciones de la sucesión de Fibonacci. Cada una se ejecuta diezmil veces y se reporta el tiempo promedio.
Can you write Python code for it?

import timeit

# cronometro para implementacion recursiva
tr = timeit.Timer("""def fibonacci(k):
    if k <= 1:
        return k
    else:
        return fibonacci(k-1) + fibonacci(k-2)

    fibonacci(25)""")

# cronómetro para implementación con formula
tf = timeit.Timer("""from math import sqrt

def fibonacci_formula(k):
    return (1.0/sqrt(5))*(((1.0+sqrt(5))/2)**k-((1.0-sqrt(5))/2)**k)

fibonacci_formula(25)""")

# ejecutarlas diez mil veces, medir su tiempo
tiempos=[tr.timeit(10000),
         tf.timeit(10000)]

# armar gráfica de barras
plt.bar([0,1], tiempos, align='center', alpha=0.5)
plt.title('10,000 ejecuciones, k=25')
plt.ylabel('milisegundos')
plt.xticks([0,1],['recursivo', 'formula'])
plt.show()

tiempos


I am working on the file "Fibonacci.ipynb".
The first task is:
Fibonacci con fórmula
Can you write Python code for it?

def fibonacci_formula(k):
    return (1.0/sqrt(5))*(((1.0+sqrt(5))/2)**k-((1.0-sqrt(5))/2)**k)

#  el indice empieza en cero
for n in range(25):
    print("F%i=%i" % (n,fibonacci_formula(n)))


I am working on the file "Fibonacci.ipynb".
The first task is:
Polinomio característico
Como $\lambda$ es un valor propio de $A$
$(\lambda I-A)\overrightarrow{X}=\overrightarrow{0}$
un eigenvector que sea su solución depende de que
$\det(\lambda I-A)=\det\left(\left[\begin{array}{cc}
\lambda & 0\
0 & \lambda
\end{array}\right]-\left[\begin{array}{cc}
1 & 1\
1 & 0
\end{array}\right]\right)=\overrightarrow{0}$
Luego
$\det\left(\left[\begin{array}{cc}
\lambda-1 & -1\
-1 & \lambda
\end{array}\right]\right)=(\lambda-1)\lambda-(-1)(-1)=\lambda^{2}-\lambda-1$
Valores propios
Para obtener los valores propios usamos la fórmula general para encontrar valores de $\lambda$ que satisfagan el polinomio característico
$p(\lambda)=\lambda^{2}-\lambda-1$
Acá la fórmula general:
$x=\frac{-b\pm\sqrt{b^{2}-4ac}}{2a}$
$\lambda=\frac{1\pm\sqrt{-1^{2}-4(1)(-1)}}{2(1)}=\frac{1\pm\sqrt{1+4}}{2}$
Por eso
$\begin{array}{c}
\lambda_{1}=\frac{1+\sqrt{5}}{2}\
\lambda_{2}=\frac{1-\sqrt{5}}{2}
\end{array}$
.......eq6
## Vectores propios
$\left[\begin{array}{cc}
\lambda-1 & -1\
-1 & \lambda
\end{array}\right]\overrightarrow{X}=\left[\begin{array}{cc}
\lambda-1 & -1\
-1 & \lambda
\end{array}\right]\left[\begin{array}{c}x_{1}\
x_{2}\end{array}\right]$
Equivale al sistema de ecuaciones
$(\lambda-1)x_{1}-x_{2}=0$....(1)
$-x_{1}+\lambda x_{2}=0$.....(2)
Despejando $x_{1}$ en (2)
$x_{1}=\lambda x_{2}$
que sustituyendo en (1)
$(\lambda-1)\lambda x_{2}-x_{2}=0$
factorizamos $x_{2}$
$x_{2}\left[(\lambda-1)\lambda-1\right]=0$
sea $x_{2}=1$ en (2)
$-x_{1}+\lambda=0$
$x_{1}=\lambda$
Por eso los vectores propios son
$v_{1}=\left[\begin{array}{c}
\lambda_{1}\
1
\end{array}\right],\: v_{2}=\left[\begin{array}{c}
\lambda_{2}\
1
\end{array}\right]$
.......eq7
Matriz de transición P
Con $v_{1}$ y $v_{2}$ formamos la matriz $P$.
Para obtener la matriz invertida $P^{-1}$ Anton y Rorres muestran este teorema:
Si
$A=\left[\begin{array}{cc}
a & b\
c & d
\end{array}\right]$
entonces
$A^{-1}=\frac{1}{ad-bc}\left[\begin{array}{cc}
d & -b\
-c & a
\end{array}\right]$
Teniendo la...
Can you write Python code for it?

from math import sqrt

# eigenvalores
l1=(1+sqrt(5))/2.0
l2=(1-sqrt(5))/2.0

print("lambda1=%s, lambda2=%s" % (l1,l2))

# la matriz de transicion P
P = np.matrix([[l1, l2],
               [1,  1]])

# valores como 1.66533454e-16 son como ceros! esto pasa por no hacerlo analiticamente
(P**-1)*A*P


I am working on the file "Fibonacci.ipynb".
The first task is:
¿Cómo se ven las series de Lucas y Fibonacci juntas?
Can you write Python code for it?

# importamos bibliotecas para plotear
import matplotlib
import matplotlib.pyplot as plt

# para desplegar los plots en el notebook
%matplotlib inline

p = []
for n in range(11):
    p.append([fibonacci(n), lucas(n)])

figura = plt.plot( p )
p


I am working on the file "Field correlations in atom cavity systems latest qutip.ipynb".
The first task is:
Once again, great agreement here!
Can you write Python code for it?

print(qutip.__version__)

import sys
print(sys.version)


I am working on the file "Field correlations in atom cavity systems latest qutip.ipynb".
The first task is:
The two curves they show are for different values of g and pump, we'll try that here:
Can you write Python code for it?

kappa = 1
g = 20*kappa
theta = g
Ep = 0.5*kappa
gamma = 0.1*kappa

Hint = theta*(0.5*sz + a.dag()*a) + 1j*g*(a.dag()*sm - a*sm.dag())
Hpump = 1j*Ep*(a - a.dag())
H = Hint + Hpump

c_ops = [np.sqrt(kappa)*a, np.sqrt(gamma)*sm]

rho0=tensor(fock_dm(2,0),fock_dm(N,0))

g2_strong, G2_strong = coherence_function_g2(H, None, taus, c_ops, a)

plt.plot(taus,np.real(g2_weak),"-.")
plt.plot(taus,np.real(g2_strong))
plt.ylabel(r"$g^{(2)}(\tau)$",fontsize=18)
plt.xlabel(r"$\kappa\tau$",fontsize=18)


I am working on the file "Field correlations in atom cavity systems latest qutip.ipynb".
The first task is:
Very nice agreement here.
Next, look at the other model (EIT Kerr) and compare.
Now for the EIT Kerr approach:
Can you write Python code for it?

# Define four level system and field:
N=4  # photons to track in the field

a=tensor(identity(4), destroy(N))

# the atomic levels:
one = Qobj([[1],[0],[0],[0]])
one = tensor(one, identity(N))
two = Qobj([[0],[1],[0],[0]])
two = tensor(two, identity(N))
three = Qobj([[0],[0],[1],[0]])
three = tensor(three, identity(N))
four = Qobj([[0],[0],[0],[1]])
four = tensor(four, identity(N))

# populations and coherences:
sig11 = one * one.dag()
sig22 = two * two.dag()
sig33 = three * three.dag()
sig44 = four * four.dag()
sig12 = one * two.dag()
sig13 = one * three.dag()
sig23 = two * three.dag()
sig24 = two * four.dag()

# Note, reversing the subscript is equivalent to taking the dagger:
sig13.dag() == three * one.dag()

# Definitions for the EIT model:
taus=np.linspace(0,10,1000)

kappa = 1
gamma = 0.1*kappa
Delta = 0.0
g1 = 6*kappa
g2 = 6*kappa

# For dashed line in plot:
delta = 0.2*kappa
OmegaC = 6*kappa
Ep = 0.7*kappa

Hint = delta*sig33 + Delta*sig44 + 1j*(g1*(a.dag()*sig13 - sig13.dag()*a) + OmegaC*(sig23 - sig23.dag()) + g2*(a.dag()*sig24 - sig24.dag()*a))
Hpump = 1j*Ep*(a - a.dag())
H = Hint + Hpump

c_ops = [np.sqrt(kappa)*a,np.sqrt(gamma + gamma)*sig33,np.sqrt(gamma)*sig44] # TODO: check the s33 and s44 decays.

rho0=tensor(fock_dm(4,0),fock_dm(N,0))

n_op = a.dag()*a
exp_ops = [n_op,sig11,sig22,sig33,sig44]
solution = mesolve(H,rho0,taus, c_ops, exp_ops)
n = solution.expect[0]

# Check the cavity photon number << 4
plt.plot(taus,n)

# Look at the level populations:
for i,v in enumerate(solution.expect[1:]):
    plt.plot(taus,v,label="j={}".format(i+1))

plt.ylim(-0.2,1.2)
plt.legend()
plt.title("Populations")
plt.ylabel(r"$\sigma_{jj}$",fontsize=18)
plt.xlabel(r"$\kappa\tau$",fontsize=18)

g2_eit, G2_eit = coherence_function_g2(H, None, taus, c_ops, a)

plt.plot(taus,np.real(g2_eit))

# Repeat for different conditions:
taus=np.linspace(0,10,1000)

kappa = 1
gamma = 0.1*kappa
Delta = 0.0
g1 = 6*kappa
g2 = 6*kappa

# For solid line:
delta = 4*kappa
OmegaC = 12*kappa
Ep = 0.1*kappa

Hint = delta*sig33 +...


I am working on the file "Field correlations in atom cavity systems.ipynb".
The first task is:
The states will be $\big|m\big\rangle \otimes \big|n\big\rangle$ where $\big|m\big\rangle$ are atomic states 1-4 and $\big|n\big\rangle$ are the photon states. Therefore we need to construct tensor operators:
Can you write Python code for it?

a=tensor(identity(2), destroy(N))
sm=tensor(sigmam(),identity(N))
sz=tensor(sigmaz(),identity(N))

kappa = 1  # the cavity decay rate, all other rates given in terms of kappa:
g = 6*kappa
theta = g
Ep = 0.1*kappa
gamma = 0.1*kappa

Hint = theta*(0.5*sz + a.dag()*a) + 1j*g*(a.dag()*sm - a*sm.dag())
Hpump = 1j*Ep*(a - a.dag())
H = Hint + Hpump

c_ops = [np.sqrt(kappa)*a, np.sqrt(gamma)*sm]

rho0=tensor(fock_dm(2,0),fock_dm(N,0)) # start with empty cavity and in atomic ground state.

n_op = a.dag()*a
solution = mesolve(H,rho0,taus, c_ops, [n_op])
n = solution.expect[0]

plt.plot(taus,n)

# 1st order correlation:
g1 = coherence_function_g1(H, taus, c_ops, a)

plt.plot(taus,np.real(g1))
plt.ylabel(r"$g^{(1)}(\tau)$",fontsize=18)
plt.xlabel(r"$\kappa\tau$",fontsize=18)

# 2nd order correlation
g2_weak = coherence_function_g2(H, taus, c_ops, a)

plt.plot(taus,np.real(g2_weak))


I am working on the file "Field correlations in atom cavity systems.ipynb".
The first task is:
Very nice agreement here.
Next, look at the other model (EIT Kerr) and compare.
Now for the EIT Kerr approach:
Can you write Python code for it?

# Define four level system and field:
N=4  # photons to track in the field

a=tensor(identity(4), destroy(N))

# the atomic levels:
one = Qobj([[1],[0],[0],[0]])
one = tensor(one, identity(N))
two = Qobj([[0],[1],[0],[0]])
two = tensor(two, identity(N))
three = Qobj([[0],[0],[1],[0]])
three = tensor(three, identity(N))
four = Qobj([[0],[0],[0],[1]])
four = tensor(four, identity(N))

# populations and coherences:
sig11 = one * one.dag()
sig22 = two * two.dag()
sig33 = three * three.dag()
sig44 = four * four.dag()
sig12 = one * two.dag()
sig13 = one * three.dag()
sig23 = two * three.dag()
sig24 = two * four.dag()

# Note, reversing the subscript is equivalent to taking the dagger:
sig13.dag() == three * one.dag()

# Definitions for the EIT model:
taus=np.linspace(0,10,1000)

kappa = 1
gamma = 0.1*kappa
Delta = 0.0
g1 = 6*kappa
g2 = 6*kappa

# For dashed line in plot:
delta = 0.2*kappa
OmegaC = 6*kappa
Ep = 0.7*kappa

Hint = delta*sig33 + Delta*sig44 + 1j*(g1*(a.dag()*sig13 - sig13.dag()*a) + OmegaC*(sig23 - sig23.dag()) + g2*(a.dag()*sig24 - sig24.dag()*a))
Hpump = 1j*Ep*(a - a.dag())
H = Hint + Hpump

c_ops = [np.sqrt(kappa)*a,np.sqrt(gamma + gamma)*sig33,np.sqrt(gamma)*sig44] # TODO: check the s33 and s44 decays.

rho0=tensor(fock_dm(4,0),fock_dm(N,0))

n_op = a.dag()*a
exp_ops = [n_op,sig11,sig22,sig33,sig44]
solution = mesolve(H,rho0,taus, c_ops, exp_ops)
n = solution.expect[0]

# Check the cavity photon number << 4
plt.plot(taus,n)

# Look at the level populations:
for i,v in enumerate(solution.expect[1:]):
    plt.plot(taus,v,label="j={}".format(i+1))

plt.ylim(-0.2,1.2)
plt.legend()
plt.title("Populations")
plt.ylabel(r"$\sigma_{jj}$",fontsize=18)
plt.xlabel(r"$\kappa\tau$",fontsize=18)

g2_eit = coherence_function_g2(H, taus, c_ops, a)

plt.plot(taus,np.real(g2_eit))

# Repeat for different conditions:
taus=np.linspace(0,10,1000)

kappa = 1
gamma = 0.1*kappa
Delta = 0.0
g1 = 6*kappa
g2 = 6*kappa

# For solid line:
delta = 4*kappa
OmegaC = 12*kappa
Ep = 0.1*kappa

Hint = delta*sig33 + Delta*sig44 +...


I am working on the file "Figure - FRET vs distance.ipynb".
The first task is:

This function computes the dye position according to the passed parameters:
Can you write Python code for it?

def dye_position(i, l=1.6, λ=0.5, ψ=0):
    # global structural params: r, δ, n, Δφ
    Δφr = Δφ*pi/180
    φ = 2*pi/n  # (radiants) rotation for base-pair
    Dx = r * cos(φ*i) + λ*( r*cos(φ*i + Δφr) - r*cos(φ*i) ) + l*cos(ψ)*cos(φ*i + 0.5*Δφr)
    Dy = r * sin(φ*i) + λ*( r*sin(φ*i + Δφr) - r*sin(φ*i) ) + l*cos(ψ)*sin(φ*i + 0.5*Δφr)
    Dz = i*δ + l*sin(ψ)
    return np.array([Dx, Dy, Dz])


I am working on the file "Figure - FRET vs distance.ipynb".
The first task is:
Dyes Parameters
Can you write Python code for it?

λ = 0.5
ψ = 0
i = 7    # number of bases from reference "base 0"
l = 1.6  # (nm) distance between S and dye position D

dye_position(7)

D_params = dict(l=1, λ=1, ψ=0)
A_params = dict(l=1, λ=0, ψ=-pi/2)

bp = np.arange(0, 1)

PD = dye_position(bp, **D_params)
PA = dye_position(bp, **A_params)

bp1 = np.arange(0, 10.1, 0.02)

PD1 = dye_position(bp1, **D_params)
PA1 = dye_position(bp1, **A_params)

axes = plot_dye(PD, marker='s')
plot_dye(PA, axes, color='r',  marker='s');
plot_dye(PA1, axes, marker='', ls='-', color='r');
plot_dye(PD1, axes, marker='', ls='-', color='b');

bp = np.arange(0, 40, 0.1)

PD = dye_position(bp, l=1.6, λ=0.2, ψ=0)
PA = dye_position(0, l=1.6, λ=0.8, ψ=-pi/2)
R = np.linalg.norm(PD.T - PA, axis=1)
#R

plt.plot(bp, R);
plt.xlabel('Base-pair')
plt.ylabel('Distance (nm)')
plt.ylim(0);

R0 = 6.7  # nm

plt.plot(bp, fret(R, R0));


I am working on the file "Figure - FRET vs distance.ipynb".
The first task is:
FRET vs distance figure
Can you write Python code for it?

%config InlineBackend.figure_format='retina'  # for hi-dpi displays

fig, ax = plt.subplots()
E_alex.plot(ax=ax)
E_mspot.plot(marker='+', mew=1, ls='none', ax=ax)

E_alexi = E_alex.rename(lambda x: int(x[:-1])).to_frame()
E_alexi.columns = ['μs-ALEX']
E_alexi

E_mspoti = E_mspot.rename(lambda x: int(x[:-1]))
#E_mspoti

sns.set(style='ticks', font_scale=1.4)

R0 = 7.3  # nm Forster Radius
PD = dye_position(bp, l=2, λ=0.6, ψ=0)
PA = dye_position(0, l=1.6, λ=0.4, ψ=-pi/2)
R = np.linalg.norm(PD.T - PA, axis=1)

pitch = δ*n
min_groove_pitch = 1.2
min_groove_pitch/pitch * 360

bp = np.arange(0, 30, 0.2)
bpm = np.array([7, 12, 17, 22, 27])

D_params = dict(l=2.4, λ=0.5, ψ=pi)
A_params = dict(l=2, λ=0.5, ψ=-1.2*pi/2)
n = 10.5     # number of bases per turn
Δφ = 131     # (degree) minor-grove angle between the two strands backbones
R0 = 7.5 # nm Forster Radius

D_params = dict(l=1.28, λ=0.61, ψ=0)
A_params = dict(l=1.28, λ=0.39, ψ=-pi/2)
n = 10.5     # number of bases per turn
Δφ = 132     # (degree) minor-grove angle between the two strands backbones
R0 = 6.7 # nm Forster Radius

D_params = dict(l=1.246, λ=1-0.256, ψ=0)
A_params = dict(l=1.246, λ=0.256, ψ=-pi/2)
n = 10.5                 # number of bases per turn
Δφ = 2.31 * (180/np.pi)  # (degree) minor-grove angle between the two strands backbones
R0 = 6.7                 # nm Forster Radius

PD = dye_position(bp, **D_params)
PA = dye_position(0, **A_params)
R = np.linalg.norm(PD.T - PA, axis=1)

fig, ax = plt.subplots()
E_alexi.plot(ax=ax, marker='s', lw=0, label='usALEX')
#E_mspoti.plot(marker='+', mew=2, ms=10, ls='none', ax=ax)
ax.set_ylim(0)
ax.set_xlim(5, 30)
ax.set_xlabel('D-A Separation (base-pairs)')
ax.set_ylabel('FRET Efficiency')
plt.xticks(E_alexi.index)
sns.despine(trim=True, offset=10, ax=ax)
ax.plot(bp, fret(R, R0), color='gray', alpha=0.5);
#savefig('multi-spot E vs distance.png');

PD = dye_position(bp, **D_params)
PA = dye_position(0, **A_params)
R = np.linalg.norm(PD.T - PA, axis=1)

fig, ax = plt.subplots()
E_alexi.plot(ax=ax,...


I am working on the file "Figure - FRET vs distance.ipynb".
The first task is:
Function to plot the dye position:
Can you write Python code for it?

def plot_dye(P, axes=None, **kws):
    kws_ = dict(marker='o', ls='-')
    kws_.update(kws)
    if axes is None:
        fig = plt.figure(figsize=(9, 9))
        ax_xy = plt.subplot2grid((2,2), (1,0))
        ax_xz = plt.subplot2grid((2,2), (0,0))
        ax_yz = plt.subplot2grid((2,2), (0,1))
        ax_3d = fig.add_subplot(224, projection='3d')
    else:
        ax_xy, ax_xz, ax_yz, ax_3d = axes

    ax_xy.plot(P[0], P[1], **kws_)
    ax_xz.plot(P[0], P[2], **kws_)
    ax_yz.plot(P[1], P[2], **kws_)
    for ax in (ax_xy, ax_xz):
        ax.set_xlabel('x (nm)')
    ax_xy.set_ylabel('y (nm)')
    ax_xz.set_xlabel('x (nm)')
    ax_xz.set_ylabel('z (nm)')
    ax_yz.set_xlabel('y (nm)')
    ax_yz.set_ylabel('z (nm)')
    lim = max(1.5, np.abs(P[0]).max(), np.abs(P[1]).max())*1.05

    ax_xy.set_xlim(-lim, lim)
    ax_xy.set_ylim(-lim, lim)
    ax_xz.set_xlim(-lim, lim)
    ax_yz.set_xlim(-lim, lim)

    ax_3d.plot(P[0], P[1], P[2], **kws_)
    return (ax_xy, ax_xz, ax_yz, ax_3d)


I am working on the file "Figure - FRET vs distance.ipynb".
The first task is:
Load data
Can you write Python code for it?

E_mspot = pd.read_csv(
    'results/Multi-spot - dsDNA - Corrected E - all_samples all_ch.csv', index_col=0)
E_mspot.columns.name = 'Channel'
E_mspot

data_file = 'results/usALEX-5samples-E-corrected-all-ph.csv'
data_alex = pd.read_csv(data_file).set_index('sample')
E_alex = data_alex.E_gauss_w
E_alex.index.name = 'Sample'
E_alex


I am working on the file "Figure1.6.ipynb".
The first task is:
Reproducible visualization
In "The Functional Art: An introduction to information graphics and visualization" by Alberto Cairo, on page 12 we are presented with a visualization of UN data time series of Fertility rate (average number of children per woman) per country:
Figure 1.6 Highlighting the relevant, keeping the secondary in the background.
Let's try to reproduce this.
Getting the data
The visualization was done in 2012, but limited the visualization to 2010. This should make it easy, in theory, to get the data, since it is historical. These are directly available as excel spreadsheets now, we'll just ignore the last bucket (2010-2015).
Pandas allows loading an excel spreadsheet straight from a URL, but here we will download it first so we have a local copy.
Can you write Python code for it?


I am working on the file "Figure1.ipynb".
The first task is:
Finally, we plot the exact final position that we obtained from running a full $N$-body simulation as well as our approximation near a neighbourhood of $a_0$ which we got from the variational equations.
Can you write Python code for it?

fig = plt.figure(figsize=(6,4))
ax = plt.subplot(111)
ax.set_xlim(a_grid[0],a_grid[-1])
ax.set_ylim(np.min(x_exact),np.max(x_exact)*1.01)
ax.set_xlabel("initial semi-major axis of outer planet")
ax.set_ylabel("$x$ position of inner planet after 10 orbits")
ax.plot(a_grid, x_exact, "-", color="black", lw=2)
ax.plot(a_grid, x_1st_order, "--", color="green")
ax.plot(a_grid, x_2nd_order, ":", color="blue")
ax.plot(a_0, x, "ro",ms=10);
plt.savefig('paper_test1.pdf',bbox_inches='tight');  # Save to file.


I am working on the file "Figure1.ipynb".
The first task is:
The following code produces an interactive version of this graph where one can change the initial semi-major axis $a_0$ and immediately see the new plot. It uses the ipywidgets tool interact. Move the slider and see how REBOUND accurately calculates the first and second derivate using variational equations.
Can you write Python code for it?

from ipywidgets import interact
def generate_plot(a_0=1.56):
    x, dxda, ddxdda = run_sim_var(a_0)
    x_1st_order = np.zeros(N)
    x_2nd_order = np.zeros(N)
    for i,a in enumerate(a_grid):
        x_1st_order[i] = x + (a-a_0)*dxda
        x_2nd_order[i] = x + (a-a_0)*dxda + 0.5*(a-a_0)*(a-a_0)*ddxdda

    fig = plt.figure(figsize=(6,4))
    ax = plt.subplot(111)
    ax.set_xlim(a_grid[0],a_grid[-1])
    ax.set_ylim(np.min(x_exact),np.max(x_exact)*1.01)
    ax.set_xlabel("initial semi-major axis of outer planet")
    ax.set_ylabel("$x$ position of inner planet after 10 orbits")
    ax.plot(a_grid, x_exact, "-", color="black", lw=2)
    ax.plot(a_grid, x_1st_order, "--", color="green")
    ax.plot(a_grid, x_2nd_order, ":", color="blue")
    ax.plot(a_0, x, "ro",ms=10)
    plt.show()
    return
interact(generate_plot,a_0=(1.4,1.7,0.01));


I am working on the file "Figure1.ipynb".
The first task is:
We now run this simulation 400 times for different initial $a$ in the range [1.4, 1.7] and store the final $x$ coordinate of the inner planet in the array x_exact.
Can you write Python code for it?

N=400
x_exact = np.zeros((N))
a_grid = np.linspace(1.4,1.7,N)
for i,a in enumerate(a_grid):
    x_exact[i] = run_sim(a)


I am working on the file "Figure1.ipynb".
The first task is:
We run one simulation with variational particles at $a_0=1.56$. We then use the derivates we got from the run_sim_var() function to approximate the final position of the inner planet as a function of the outer planet's initial semi-major axis using a Taylor series:
$$x(a) = x(a_0)+\frac{\partial x}{\partial a}\Big|{a_0} (a-a_0)+\frac 12\frac{\partial^2 x}{\partial a^2}\Big|{a_0} (a-a_0)^2 + \ldots$$
We do this to both first and second order.
Can you write Python code for it?

a_0 = 1.56
x, dxda, ddxdda = run_sim_var(a_0)
x_1st_order = np.zeros(N)
x_2nd_order = np.zeros(N)
for i,a in enumerate(a_grid):
    x_1st_order[i] = x + (a-a_0)*dxda
    x_2nd_order[i] = x + (a-a_0)*dxda + 0.5*(a-a_0)*(a-a_0)*ddxdda


I am working on the file "Figure1.ipynb".
The first task is:
We setup a planetary system with two Jupiter mass planets. The following function takes that system, integrates it forward in time by 10 orbits and returns the inner planet's $x$ coordinate at the end of the simulation. The $x$ coordinate changes because the planet orbits the star, but also because the planet interacts with the other planet. The function takes the outer planet's initial semi-major axis, $a$, as a parameter. We setup the system using heliocentric coordinates and therefore specify the primary attribute when adding particles to REBOUND (by default REBOUND uses Jacobi coordinates which are not supported by variational equations).
Can you write Python code for it?

def run_sim(a):
    sim = rebound.Simulation()
    sim.add(m=1.)
    sim.add(primary=sim.particles[0],m=1e-3, a=1)
    sim.add(primary=sim.particles[0],m=1e-3, a=a)

    sim.integrate(2.*np.pi*10.)
    return sim.particles[1].x