I am working on the file "2017 August Demo.ipynb".
The first task is:
Verfication
Check the Assignment Match the Actual ones
Can you write Python code for it?

correct = 0.0
assignment = []
printProgressBar(0, X.shape[0], prefix='Progress:', suffix='Complete', length=50)
for idx, item in enumerate(X):
    center = np.squeeze(np.sum(np.square(item - centroid), axis =1)).argsort()[0]
    if assign[idx] == center:
        correct +=1.0
    assignment.append(center)
    printProgressBar(idx, X.shape[0], prefix='Progress:', suffix='Complete'+' Acc:' + str(correct/(idx+1)), length=50)


I am working on the file "2017 NEON DI JAMB Presentation.ipynb".
The first task is:

CDF and PS Plots illustrating Albedo

Can you write Python code for it?

# Run function and plot
fig = plt.figure(figsize=(10,5))
ax = fig.add_subplot(121)
ind = 0 # Index for Albedo
X1 = cdf_info_array[0,:,ind]
F1 = cdf_info_array[1,:,ind]
plt.title('CDF %s'%file_list[ind][0:-4])
ax.set_ylabel(file_list[ind][0:-4])
ax.set_xlabel('fraction of pixels')
ax.set_xlim([0,1])
ax.plot(F1,X1)

ax = fig.add_subplot(122)
ind = 0
X1 = cdf_info_array[0,:,ind] #replace with ps
F1 = cdf_info_array[1,:,ind]
plt.title('FFT %s'%file_list[ind][0:-4])
plt.ylabel(file_list[ind][0:-4])
ax.plot(F1,X1, color='green')
plt.show()
#sub.plt


I am working on the file "2017 NEON DI JAMB Presentation.ipynb".
The first task is:

CDF and PS Plots illustrating Canopy Height

Can you write Python code for it?

# Run function and plot
fig = plt.figure(figsize=(10,5))
ax = fig.add_subplot(121)
ind = 3 # Index for Canopy Height
X1 = cdf_info_array[0,:,ind]
F1 = cdf_info_array[1,:,ind]
plt.title('CDF %s'%file_list[ind][0:-4])
ax.set_ylabel(file_list[ind][0:-4])
ax.set_xlabel('fraction of pixels')
ax.set_xlim([0,1])
ax.plot(F1,X1)

ax = fig.add_subplot(122)
ind = 0
X1 = cdf_info_array[0,:,ind] #replace with ps
F1 = cdf_info_array[1,:,ind]
plt.title('FFT %s'%file_list[ind][0:-4])
ax.set_ylabel(file_list[ind][0:-4])
ax.set_xlabel('fraction')
ax.plot(F1,X1, color='red')
plt.show()

# Run function and plot
fig = plt.figure(figsize=(10,5))
ax = fig.add_subplot(121)
ind = 14 # Index for NDVI
X1 = cdf_info_array[0,:,ind]
F1 = cdf_info_array[1,:,ind]
plt.title('CDF %s'%file_list[ind][0:-4])
ax.set_ylabel(file_list[ind][0:-4])
ax.set_xlabel('fraction of pixels')
ax.set_xlim([0,1])
ax.plot(F1,X1)

ax = fig.add_subplot(122)
ind = 0
X1 = cdf_info_array[0,:,ind] #replace with ps
F1 = cdf_info_array[1,:,ind]
plt.title('FFT %s'%file_list[ind][0:-4])
ax.set_ylabel(file_list[ind][0:-4])
ax.set_xlabel('fraction')
ax.plot(F1,X1, color='red')
plt.show()


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
1. What is pandas?

pandas main page
pandas installation instructions
Anaconda distribution of Python (includes pandas)
How to use the IPython/Jupyter notebook (video)

2. How do I read a tabular data file into pandas?
Can you write Python code for it?

# read a dataset of Chipotle orders directly from a URL and store the results in a DataFrame
orders = pd.read_table('data/chipotle.tsv')

# examine the first 5 rows
orders.head()


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
3. How do I select a pandas Series from a DataFrame?
Can you write Python code for it?

# read a dataset of UFO reports into a DataFrame
ufo = pd.read_table('data/ufo.csv', sep=',')


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
API reference for datetime properties and methods
Can you write Python code for it?

# convert a single string to datetime format (outputs a timestamp object)
ts = pd.to_datetime('1/1/1999')
ts

# compare a datetime Series with a timestamp
ufo.loc[ufo.Time >= ts, :].head()

# perform mathematical operations with timestamps (outputs a timedelta object)
ufo.Time.max() - ufo.Time.min()

# timedelta objects also have attributes you can access
(ufo.Time.max() - ufo.Time.min()).days

# allow plots to appear in the notebook
%matplotlib inline

# count the number of UFO reports per year
ufo['Year'] = ufo.Time.dt.year
ufo.Year.value_counts().sort_index().head()

# plot the number of UFO reports per year (line plot is the default)
ufo.Year.value_counts().sort_index().plot()


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Bracket notation will always work, whereas dot notation has limitations:

Dot notation doesn't work if there are spaces in the Series name
Dot notation doesn't work if the Series has the same name as a DataFrame method or attribute (like 'head' or 'shape')
Dot notation can't be used to define the name of a new Series (see below)
Can you write Python code for it?

# create a new 'Location' Series (must use bracket notation to define the Series name)
ufo['Location'] = ufo.City + ', ' + ufo.State
ufo.head()


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Documentation for Series
Can you write Python code for it?

# calculate the total annual beer servings for each country
(drinks.beer_servings * people).head()


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Documentation for agg
Can you write Python code for it?

# specifying a column to which the aggregation function should be applied is not required
drinks.groupby('continent').mean()

# allow plots to appear in the notebook
%matplotlib inline

# side-by-side bar plot of the DataFrame directly above
drinks.groupby('continent').mean().plot(kind='bar')


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Documentation for astype
Can you write Python code for it?

# alternatively, change the data type of a Series while reading in a file
drinks = pd.read_csv('data/drinks.csv', dtype={'beer_servings':float})
drinks.dtypes

# read a dataset of Chipotle orders into a DataFrame
orders = pd.read_table('data/chipotle.tsv')
orders.head()

# examine the data type of each Series
orders.dtypes

# convert a string to a number in order to do math
orders.item_price.str.replace('$', '').astype(float).mean()

# string method 'contains' checks for a substring and returns a boolean Series
orders.item_name.str.contains('Chicken').head()

# convert a boolean Series to an integer (False = 0, True = 1)
orders.item_name.str.contains('Chicken').astype(int).head()


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Documentation for concat
Can you write Python code for it?

# reset the DataFrame
train = pd.read_csv('http://bit.ly/kaggletrain')
train.head()

# pass the DataFrame to 'get_dummies' and specify which columns to dummy (it drops the original columns)
pd.get_dummies(train, columns=['Sex', 'Embarked']).head()

# use the 'drop_first' parameter (new in pandas 0.18) to drop the first dummy variable for each feature
pd.get_dummies(train, columns=['Sex', 'Embarked'], drop_first=True).head()


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Documentation for concat
Indexing and selecting data
[Back to top]
19. How do I select multiple rows and columns from a pandas DataFrame?
Can you write Python code for it?

# read a dataset of UFO reports into a DataFrame
ufo = pd.read_csv('data/ufo.csv')
ufo.head(3)


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Documentation for crosstab
Exploring a numeric Series:
Can you write Python code for it?

# calculate various summary statistics
movies.duration.describe()

# many statistics are implemented as Series methods
movies.duration.mean()


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Documentation for describe
[Back to top]
11. How do I use the "axis" parameter in pandas?
Can you write Python code for it?

# read a dataset of alcohol consumption into a DataFrame
drinks = pd.read_csv('data/drinks.csv')
drinks.head()

# drop a column (temporarily)
drinks.drop('continent', axis=1).head()


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Documentation for describe
[Back to top]
5. How do I rename columns in a pandas DataFrame?
Can you write Python code for it?

# read a dataset of UFO reports into a DataFrame
ufo = pd.read_csv('data/ufo.csv')

# examine the column names
ufo.columns

# rename two of the columns by using the 'rename' method
ufo.rename(columns={'Colors Reported':'Colors_Reported', 'Shape Reported':'Shape_Reported'}, inplace=True)
ufo.columns


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Documentation for drop
Can you write Python code for it?

# drop a row (temporarily)
drinks.drop(2, axis=0).head()


# remove multiple columns at once
ufo.drop(['City', 'State'], axis=1, inplace=True)
ufo.head()

# remove multiple rows at once (axis=0 refers to rows)
ufo.drop([0, 1], axis=0, inplace=True)
ufo.head()


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Documentation for fillna
Can you write Python code for it?

# confirm that the missing values were filled in
ufo['Shape Reported'].value_counts().head()


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Documentation for groupby
Can you write Python code for it?

# other aggregation functions (such as 'max') can also be used with groupby
drinks.groupby('continent').beer_servings.max()

# multiple aggregation functions can be applied simultaneously
drinks.groupby('continent').beer_servings.agg(['count', 'mean', 'min', 'max'])


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Documentation for info and memory_usage
Can you write Python code for it?

# use the 'category' data type (new in pandas 0.15) to store the 'continent' strings as integers
drinks['continent'] = drinks.continent.astype('category')
drinks.dtypes

# 'continent' Series appears to be unchanged
drinks.continent.head()

# strings are now encoded (0 means 'Africa', 1 means 'Asia', 2 means 'Europe', etc.)
drinks.continent.cat.codes.head()

# memory usage has been drastically reduced
drinks.memory_usage(deep=True)

# repeat this process for the 'country' Series
drinks['country'] = drinks.country.astype('category')
drinks.memory_usage(deep=True)

# memory usage increased because we created 193 categories
drinks.country.cat.categories


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Documentation for isin
[Back to top]
10. Your pandas questions answered!
Question: When reading from a file, how do I read in only a subset of the columns?
Can you write Python code for it?

# read a dataset of UFO reports into a DataFrame, and check the columns
ufo = pd.read_csv('data/ufo.csv')
ufo.columns

# specify which columns to include by name
ufo = pd.read_csv('data/ufo.csv', usecols=['City', 'State'])

# or equivalently, specify columns by position
ufo = pd.read_csv('data/ufo.csv', usecols=[0, 4])
ufo.columns


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Documentation for isnull
Question: Why are DataFrame slices inclusive when using .loc, but exclusive when using .iloc?
Can you write Python code for it?

# label-based slicing is inclusive of the start and stop
ufo.loc[0:4, :]

# position-based slicing is inclusive of the start and exclusive of the stop
ufo.iloc[0:4, :]


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Documentation for isnull and notnull
Can you write Python code for it?

# count the number of missing values in each Series
ufo.isnull().sum()


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Documentation for loc
Can you write Python code for it?

# set an existing column as the index
drinks.set_index('country', inplace=True)
drinks.head()


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Documentation for loc
[Back to top]
9. How do I apply multiple filter criteria to a pandas DataFrame?
Can you write Python code for it?

# read a dataset of top-rated IMDb movies into a DataFrame
movies = pd.read_csv('data/imdb_1000.csv')
movies.head()

# filter the DataFrame to only show movies with a 'duration' of at least 200 minutes
movies[movies.duration >= 200]


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Documentation for loc and iloc
Can you write Python code for it?

# 'iloc' is simply following NumPy's slicing convention...
ufo.values[0:4, :]

# ...and NumPy is simply following Python's slicing convention
'python'[0:4]

# 'loc' is inclusive of the stopping label because you don't necessarily know what label will come after it
ufo.loc[0:4, 'City':'State']


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Documentation for plot
[Back to top]
15. How do I explore a pandas Series?
Can you write Python code for it?

# read a dataset of top-rated IMDb movies into a DataFrame
movies = pd.read_csv('data/imbd_1000.csv')
movies.head()

# examine the data type of each Series
movies.dtypes


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Documentation for plot
[Back to top]
16. How do I handle missing values in pandas?
Can you write Python code for it?

# read a dataset of UFO reports into a DataFrame
ufo = pd.read_csv('data/ufo.csv')
ufo.tail()


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Documentation for read_csv
Can you write Python code for it?

# replace all spaces with underscores in the column names by using the 'str.replace' method
ufo.columns = ufo.columns.str.replace(' ', '_')
ufo.columns


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Documentation for read_csv
Question: How do I iterate through a Series?
Can you write Python code for it?

# Series are directly iterable (like a list)
for c in ufo.City:
    print(c)


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Documentation for read_table
Can you write Python code for it?

users = pd.read_table('data/u.user')
# examine the first 5 rows
users.head()

users = pd.read_table('data/u.user', sep='|')
# examine the first 5 rows
users.head()

users = pd.read_table('data/u.user', sep='|', header=None)
# examine the first 5 rows
users.head()

user_cols = ['user_id', 'age', 'gender', 'occupation', 'zip_code']
users = pd.read_table('data/u.user', sep='|', header=None, names=user_cols)
# examine the first 5 rows
users.head()


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Documentation for sample
Can you write Python code for it?

# use the 'random_state' parameter for reproducibility
ufo.sample(n=3, random_state=42)

# sample 75% of the DataFrame's rows without replacement
train = ufo.sample(frac=0.75, random_state=99)

# store the remaining 25% of the rows in another DataFrame
test = ufo.loc[~ufo.index.isin(train.index), :]


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Documentation for set_index
Can you write Python code for it?

# Series index is on the left, values are on the right
drinks.continent.head()

# another example of a Series (output from the 'value_counts' method)
drinks.continent.value_counts()


# 'country' is now the index
drinks.index

# 'country' is no longer a column
drinks.columns

# 'country' data is no longer part of the DataFrame contents
drinks.shape

# country name can now be used for selection
drinks.loc['Brazil', 'beer_servings']

# index name is optional
drinks.index.name = None
drinks.head()

# restore the index name, and move the index back to a column
drinks.index.name = 'country'
drinks.reset_index(inplace=True)
drinks.head()


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Documentation for the DataFrame constructor
Can you write Python code for it?

# ensure that PassengerID is the first column by setting it as the index
pd.DataFrame({'PassengerId':test.PassengerId, 'Survived':new_pred_class}).set_index('PassengerId').head()

# write the DataFrame to a CSV file that can be submitted to Kaggle
pd.DataFrame({'PassengerId':test.PassengerId, 'Survived':new_pred_class}).set_index('PassengerId').to_csv('sub.csv')


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Documentation for to_pickle and read_pickle
[Back to top]
23. More of your pandas questions answered!
Question: Could you explain how to read the pandas documentation?
pandas API reference
Question: What is the difference between ufo.isnull() and pd.isnull(ufo)?
Can you write Python code for it?

# read a dataset of UFO reports into a DataFrame
ufo = pd.read_csv('http://bit.ly/uforeports')
ufo.head()

# use 'isnull' as a top-level function
pd.isnull(ufo).head()

# equivalent: use 'isnull' as a DataFrame method
ufo.isnull().head()


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Exploring a non-numeric Series:
Can you write Python code for it?

# count the non-null values, unique values, and frequency of the most common value
movies.genre.describe()


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Generally speaking:

If you have "K" possible values for a categorical feature, you only need "K-1" dummy variables to capture all of the information about that feature.
One convention is to drop the first dummy variable, which defines that level as the "baseline".
Can you write Python code for it?

# drop the first dummy variable ('female') using the 'iloc' method
pd.get_dummies(train.Sex).iloc[:, 1:].head()

# add a prefix to identify the source of the dummy variables
pd.get_dummies(train.Sex, prefix='Sex').iloc[:, 1:].head()

# use 'get_dummies' with a feature that has 3 possible values
pd.get_dummies(train.Embarked, prefix='Embarked').head(10)

# drop the first dummy variable ('C')
pd.get_dummies(train.Embarked, prefix='Embarked').iloc[:, 1:].head(10)


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Goal: Fix the 'duration' for 'The Shawshank Redemption'.
Can you write Python code for it?

# overwrite the relevant cell with the correct duration
top_movies.loc[0, 'duration'] = 150


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Goal: Mark the 'NOT RATED' values as missing values, represented by 'NaN'.
Can you write Python code for it?

# first, locate the relevant rows
movies[movies.content_rating=='NOT RATED'].head()

# then, select the 'content_rating' Series from those rows
movies[movies.content_rating=='NOT RATED'].content_rating.head()

# finally, replace the 'NOT RATED' values with 'NaN' (imported from NumPy)
import numpy as np
movies[movies.content_rating=='NOT RATED'].content_rating = np.nan


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Indexing and selecting data
[Back to top]
18. What do I need to know about the pandas index?
Can you write Python code for it?

# read a dataset of alcohol consumption into a DataFrame
drinks = pd.read_csv('data/drinks.csv')
drinks.head()

# every DataFrame has an index
drinks.index

# every Series also has an index (which carries over from the DataFrame)
drinks.continent.head()

# set 'country' as the index
drinks.set_index('country', inplace=True)


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Logic for duplicated:

keep='first' (default): Mark duplicates as True except for the first occurrence.
keep='last': Mark duplicates as True except for the last occurrence.
keep=False: Mark all duplicates as True.
Can you write Python code for it?

# examine the duplicate rows (ignoring the first occurrence)
users.loc[users.duplicated(keep='first'), :]

# examine the duplicate rows (ignoring the last occurrence)
users.loc[users.duplicated(keep='last'), :]

# examine the duplicate rows (including all duplicates)
users.loc[users.duplicated(keep=False), :]

# drop the duplicate rows (inplace=False by default)
users.drop_duplicates(keep='first').shape

users.drop_duplicates(keep='last').shape

users.drop_duplicates(keep=False).shape


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Methods end with parentheses, while attributes don't:
Can you write Python code for it?

# example method: show the first 5 rows
movies.head()

# example method: calculate summary statistics
movies.describe()

# example attribute: number of rows and columns
movies.shape

# example attribute: data type of each column
movies.dtypes

# use an optional parameter to the describe method to summarize only 'object' columns
movies.describe(include=['object'])


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Note: None of the sorting methods below affect the underlying data. (In other words, the sorting is temporary).
Can you write Python code for it?

# sort the 'title' Series in ascending order (returns a Series)
movies.title.sort_values()

# sort in descending order instead
movies.title.sort_values(ascending=False).head()


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Note: There is no need to convert these pandas objects to NumPy arrays. scikit-learn will understand these objects as long as they are entirely numeric and the proper shapes.
Can you write Python code for it?

# fit a classification model to the training data
from sklearn.linear_model import LogisticRegression
logreg = LogisticRegression()
logreg.fit(X, y)


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
OR
Can you write Python code for it?

# read_csv is equivalent to read_table, except it assumes a comma separator
ufo = pd.read_csv('data/ufo.csv')
type(ufo)

# examine the first 5 rows
ufo.head()

# select the 'City' Series using bracket notation
ufo['Colors Reported']

type(ufo['City'])

# or equivalently, use dot notation - see notes below
ufo.City


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Overview of categorical data in pandas
API reference for categorical methods
[Back to top]
22. How do I use pandas with scikit-learn to create Kaggle submissions?
Can you write Python code for it?

# read the training dataset from Kaggle's Titanic competition into a DataFrame
train = pd.read_csv('http://bit.ly/kaggletrain')
train.head()


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Problem: That statement involves two operations, a __getitem__ and a __setitem__. pandas can't guarantee whether the __getitem__ operation returns a view or a copy of the data.

If __getitem__ returns a view of the data, __setitem__ will affect the 'movies' DataFrame.
But if __getitem__ returns a copy of the data, __setitem__ will not affect the 'movies' DataFrame.
Can you write Python code for it?

# the 'content_rating' Series has not changed
movies.content_rating.isnull().sum()


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Question: How do I iterate through a DataFrame?
Can you write Python code for it?

# various methods are available to iterate through a DataFrame
for index, row in ufo.iterrows():
    print(index, row.City, row.State)


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Question: How do I randomly sample rows from a DataFrame?
Can you write Python code for it?

# sample 3 rows from the DataFrame without replacement (new in pandas 0.16.1)
ufo.sample(n=3)


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Question: When reading from a file, how do I read in only a subset of the rows?
Can you write Python code for it?

# specify how many rows to read
ufo = pd.read_csv('data/ufo.csv', nrows=3)
ufo


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Rules for specifying multiple filter criteria in pandas:

use & instead of and
use | instead of or
add parentheses around each condition to specify evaluation order

Goal: Further filter the DataFrame of long movies (duration >= 200) to only show movies which also have a 'genre' of 'Drama'
Can you write Python code for it?

# CORRECT: use the '&' operator to specify that both conditions are required
movies[(movies.duration >=200) & (movies.genre == 'Drama')]

# INCORRECT: using the '|' operator would have shown movies that are either long or dramas (or both)
movies[(movies.duration >=200) | (movies.genre == 'Drama')].head()


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Rules for using numbers with ix:

If the index is strings, numbers are treated as integer positions, and thus slices are exclusive on the right.
If the index is integers, numbers are treated as labels, and thus slices are inclusive.
Can you write Python code for it?

# rows 'Albania' through 'Andorra' (inclusive), columns in positions 0 through 2 (exclusive)
drinks.ix['Albania':'Andorra', 0:2]

# rows 0 through 2 (inclusive), columns in positions 0 through 2 (exclusive)
ufo.ix[0:2, 0:2]


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Solution: Any time you are attempting to create a DataFrame copy, use the copy method.
Can you write Python code for it?

# explicitly create a copy of 'movies'
top_movies = movies.loc[movies.star_rating >= 9, :].copy()

# pandas now knows that you are updating a copy instead of a view (does not cause a SettingWithCopyWarning)
top_movies.loc[0, 'duration'] = 150

# 'top_movies' DataFrame has been updated
top_movies


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
String handling section of the pandas API reference
[Back to top]
13. How do I change the data type of a pandas Series?
Can you write Python code for it?

# read a dataset of alcohol consumption into a DataFrame
drinks = pd.read_csv('data/drinks.csv')
drinks.head()

# examine the data type of each Series
drinks.dtypes

# change the data type of an existing Series
drinks['beer_servings'] = drinks.beer_servings.astype(float)
drinks.dtypes


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Summary of changes to the sorting API in pandas 0.17
[Back to top]
8. How do I filter rows of a pandas DataFrame by column value?
Can you write Python code for it?

# read a dataset of top-rated IMDb movies into a DataFrame
movies = pd.read_csv('data/imdb_1000.csv')
movies.head()

# examine the number of rows and columns
movies.shape


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
The category data type should only be used with a string Series that has a small number of possible values.
Can you write Python code for it?

# create a small DataFrame from a dictionary
df = pd.DataFrame({'ID':[100, 101, 102, 103], 'quality':['good', 'very good', 'good', 'excellent']})
df

# sort the DataFrame by the 'quality' Series (alphabetical order)
df.sort_values('quality')

# define a logical ordering for the categories
df['quality'] = df.quality.astype('category', categories=['good', 'very good', 'excellent'], ordered=True)
df.quality

# sort the DataFrame by the 'quality' Series (logical order)
df.sort_values('quality')

# comparison operators work with ordered categories
df.loc[df.quality > 'good', :]


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
The ix method is used to select rows and columns by label or integer position, and should only be used when you need to mix label-based and integer-based selection in the same call.
Can you write Python code for it?

# read a dataset of alcohol consumption into a DataFrame and set 'country' as the index
drinks = pd.read_csv('data/drinks.csv', index_col='country')
drinks.head()

# row with label 'Albania', column in position 0
drinks.ix['Albania', 0]

# row in position 1, column with label 'beer_servings'
drinks.ix[1, 'beer_servings']


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
The loc method is used to select rows and columns by label. You can pass it:

A single label
A list of labels
A slice of labels
A boolean Series
A colon (which indicates "all labels")
Can you write Python code for it?

# row 0, all columns
ufo.loc[0, :]

# rows 0 and 1 and 2, all columns
ufo.loc[[0, 1, 2], :]

# rows 0 through 2 (inclusive), all columns
ufo.loc[0:2, :]

# this implies "all columns", but explicitly stating "all columns" is better
ufo.loc[0:2]

# rows 0 through 2 (inclusive), column 'City'
ufo.loc[0:2, 'City']

# rows 0 through 2 (inclusive), columns 'City' and 'State'
ufo.loc[0:2, ['City', 'State']]

# accomplish the same thing using double brackets - but using 'loc' is preferred since it's more explicit
ufo[['City', 'State']].head(3)

# rows 0 through 2 (inclusive), columns 'City' through 'State' (inclusive)
ufo.loc[0:2, 'City':'State']

# accomplish the same thing using 'head' and 'drop'
ufo.head(3).drop('Time', axis=1)

# rows in which the 'City' is 'Oakland', column 'State'
ufo.loc[ufo.City=='Oakland', 'State']

# accomplish the same thing using "chained indexing" - but using 'loc' is preferred since chained indexing can cause problems
ufo[ufo.City=='Oakland'].State


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
This calculation works because:

The sum method for a DataFrame operates on axis=0 by default (and thus produces column sums).
In order to add boolean values, pandas converts True to 1 and False to 0.
Can you write Python code for it?

# use the 'isnull' Series method to filter the DataFrame rows
ufo[ufo.City.isnull()].head()


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Understanding logical operators:

and: True only if both sides of the operator are True
or: True if either side of the operator is True
Can you write Python code for it?

# demonstration of the 'and' operator
print(True and True)
print(True and False)
print(False and False)

# demonstration of the 'or' operator
print(True or True)
print(True or False)
print(False or False)


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
Video series: Introduction to machine learning with scikit-learn
Can you write Python code for it?

# read the testing dataset from Kaggle's Titanic competition into a DataFrame
test = pd.read_csv('http://bit.ly/kaggletest')
test.head()

# create a feature matrix from the testing data that matches the training data
X_new = test.loc[:, feature_cols]
X_new.shape

# use the fitted model to make predictions for the testing set observations
new_pred_class = logreg.predict(X_new)

# create a DataFrame of passenger IDs and testing set predictions
pd.DataFrame({'PassengerId':test.PassengerId, 'Survived':new_pred_class}).head()


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
What does "NaN" mean?

"NaN" is not a string, rather it's a special value: numpy.nan.
It stands for "Not a Number" and indicates a missing value.
read_csv detects missing values (by default) when reading the file, and replaces them with this special value.

Documentation for read_csv
Can you write Python code for it?

# 'isnull' returns a DataFrame of booleans (True if missing, False if not missing)
ufo.isnull().tail()

# 'nonnull' returns the opposite of 'isnull' (True if not missing, False if missing)
ufo.notnull().tail()


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
What is the index used for?

identification
selection
alignment (covered in the next video)
Can you write Python code for it?

# identification: index remains with each row when filtering the DataFrame
drinks[drinks.continent=='South America']

# selection: select a portion of the DataFrame using the index
drinks.loc[23, 'beer_servings']


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
When referring to rows or columns with the axis parameter:

axis 0 refers to rows
axis 1 refers to columns
Can you write Python code for it?

# calculate the mean of each numeric column
drinks.mean()

# or equivalently, specify the axis explicitly
drinks.mean(axis=0)


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
[Back to top]
21. How do I make my pandas DataFrame smaller and faster?
Can you write Python code for it?

# read a dataset of alcohol consumption into a DataFrame
drinks = pd.read_csv('data/drinks.csv')
drinks.head()

# exact memory usage is unknown because object columns are references elsewhere
drinks.info()

# force pandas to calculate the true memory usage
drinks.info(memory_usage='deep')

# calculate the memory usage for each Series (in bytes)
drinks.memory_usage(deep=True)


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
[Back to top]
26. How do I find and remove duplicate rows in pandas?
Can you write Python code for it?

# read a dataset of movie reviewers into a DataFrame
user_cols = ['user_id', 'age', 'gender', 'occupation', 'zip_code']
users = pd.read_table('http://bit.ly/movieusers', sep='|', header=None, names=user_cols, index_col='user_id')
users.head()

users.shape

# detect duplicate zip codes: True if an item is identical to a previous item
users.zip_code.duplicated().tail()

# count the duplicate items (True becomes 1, False becomes 0)
users.zip_code.duplicated().sum()

# detect duplicate DataFrame rows: True if an entire row is identical to a previous row
users.duplicated().tail()

# count the duplicate rows
users.duplicated().sum()


I am working on the file "2017-01-23/pandas.ipynb".
The first task is:
[Back to top]
7. How do I sort a pandas DataFrame or a Series?
Can you write Python code for it?

# read a dataset of top-rated IMDb movies into a DataFrame
movies = pd.read_csv('data/imdb_1000.csv')
movies.head()


I am working on the file "2017-02-15-jm-predict-employee-leaving.ipynb".
The first task is:
And for our final output layer we will use the sigmoid function for our activation function because we want our output to be between 0 and 1.
Can you write Python code for it?

w_3 = tf.Variable(tf.truncated_normal([8, 1], stddev=0.01))
b_3 = tf.Variable(tf.truncated_normal([1], stddev=0.01))

output_layer = tf.nn.sigmoid(tf.add(tf.matmul(layer_2, w_3), b_3))


I am working on the file "2017-02-15-jm-predict-employee-leaving.ipynb".
The first task is:
For our optimizer we will use Adam with a learning rate of 1e-3.
Can you write Python code for it?


I am working on the file "2017-02-15-jm-predict-employee-leaving.ipynb".
The first task is:
Lets define a function that will give us a confusion matrix of our data and show us the F1 Score and the Total Accuracy.
Can you write Python code for it?

def confusion_matrix(pred_data, act_data, threshold=0.7):
    stayed_true = 0
    stayed_false = 0
    left_true = 0
    left_false = 0
    for i in range(len(pred_data)):
        if pred_data[i][0] >= threshold and act_data[i][0] == 1:
            left_true += 1
        elif pred_data[i][0] < threshold and act_data[i][0] == 1:
            left_false += 1
        elif pred_data[i][0] >= threshold and act_data[i][0] == 0:
            stayed_false += 1
        elif pred_data[i][0] < threshold and act_data[i][0] == 0:
            stayed_true += 1
    precision = left_true/np.max([1e-5, (left_true + left_false)])
    recall = left_true/np.max([1e-5, (left_true + stayed_false)])
    f1_score = 2*((precision*recall)/(precision+recall))
    print("Stayed True: {0}\nStayed False: {1}\nLeft True: {2}\nLeft False: {3}".format(stayed_true, stayed_false, left_true, left_false))
    print("Precision = {0}".format(precision))
    print("Recall = {0}".format(recall))
    print("F1 score = {0}".format(f1_score))
    print("Total Accuracy = {0}".format((stayed_true+left_true)/(len(pred_data))) )


I am working on the file "2017-02-15-jm-predict-employee-leaving.ipynb".
The first task is:
Lets do some tensorflow setup and define an array to store our loss function values when we train our model.
Can you write Python code for it?

init = tf.global_variables_initializer()

sess = tf.Session()

sess.run(init)

loss_values = []


I am working on the file "2017-02-15-jm-predict-employee-leaving.ipynb".
The first task is:
Lets set our seeds for the environment and pull in our data.
Can you write Python code for it?

np.random.seed(7)
tf.set_random_seed(7)

init_data = pd.read_csv("./HR_comma_sep.csv")


I am working on the file "2017-02-15-jm-predict-employee-leaving.ipynb".
The first task is:
Next we will define our cost function, which will be the cross entropy loss function.
Can you write Python code for it?

cost = -tf.reduce_mean(tf.multiply(Y_init, tf.log(output_layer)) + (1 - Y_init)*tf.log(1 - output_layer) )


I am working on the file "2017-02-15-jm-predict-employee-leaving.ipynb".
The first task is:
Now lets create our first hidden layer. Typically your hidden layers will use a Rectified Linear Unit(Relu) activation function in these cases, but in our case we will use an Exponential Linear Unit(Elu) activation function for its nice properties of reducing the bias shift effect on our network to have faster learning than Relu and for the fact that it acts like batch normalization without the computational complexity. We will also add the caveat of initalizing our weights and biases with a standard deviation of 0.01.
Can you write Python code for it?

w_1 = tf.Variable(tf.truncated_normal([num_features, 10], stddev=0.01))
b_1 = tf.Variable(tf.truncated_normal([10], stddev=0.01))

layer_1 = tf.nn.elu(tf.add(tf.matmul(X_init, w_1), b_1))


I am working on the file "2017-02-15-jm-predict-employee-leaving.ipynb".
The first task is:
Now lets grab the predictions of our training set data.
Can you write Python code for it?

predictions = sess.run(output_layer, feed_dict={X_init:data})


I am working on the file "2017-02-15-jm-predict-employee-leaving.ipynb".
The first task is:
Now lets set up our training and then print out our final cost value.
Can you write Python code for it?

for epoch in range(num_epochs):
    start_n = 0
    c = None
    while start_n < count:
        sess.run(optimizer, feed_dict={X_init:data[start_n:(start_n + batch_size)], Y_init:data_labels[start_n:(start_n + batch_size)]})
        start_n += batch_size
    c = sess.run(cost, feed_dict={X_init:data, Y_init:data_labels})
    loss_values.append(c)
print("Final cost = {0}".format(sess.run(cost, feed_dict={X_init:data, Y_init:data_labels})) )


I am working on the file "2017-02-15-jm-predict-employee-leaving.ipynb".
The first task is:
Now lets start defining our model. The model we will create will have 2 hidden layers and an output layer. Lets start with defining the inputs and labels.
Can you write Python code for it?

X_init = tf.placeholder(tf.float32, [None, num_features])
Y_init = tf.placeholder(tf.float32, [None, 1])


I am working on the file "2017-02-15-jm-predict-employee-leaving.ipynb".
The first task is:
Now that we have defined our function to print out the metrics that we want to see, lets test out two different thresholds for analyzing our data. We will first use the default 70% threshold that we had set in our function and then we will try a much smaller threshold, like 33%.
Can you write Python code for it?

confusion_matrix(predictions, data_labels)

confusion_matrix(predictions, data_labels, 0.33)


I am working on the file "2017-02-15-jm-predict-employee-leaving.ipynb".
The first task is:
Now we have our stratified samples. However, just to make sure, we can look at our training set percentages.
Can you write Python code for it?


I am working on the file "2017-02-15-jm-predict-employee-leaving.ipynb".
The first task is:
Our second hidden layer will be the same but adjusted for the new input shape.
Can you write Python code for it?

w_2 = tf.Variable(tf.truncated_normal([10, 8], stddev=0.01))
b_2 = tf.Variable(tf.truncated_normal([8], stddev=0.01))

layer_2 = tf.nn.elu(tf.add(tf.matmul(layer_1, w_2), b_2))


I am working on the file "2017-02-15-jm-predict-employee-leaving.ipynb".
The first task is:
Since the feature we are wanting to predict is whether the employee has left or not, we should look at the percentages of left versus stayed in our data set.
Can you write Python code for it?


I am working on the file "2017-02-15-jm-predict-employee-leaving.ipynb".
The first task is:
So it looks like, even with our test set, the 33% is a better threshold to use, to keep our Precision and Recall values close together. Now the last thing to check is to compare our cost values for the training set and test set to make sure we are not overfitting or underfitting the data.
Can you write Python code for it?

print("Cost for train data: {0}".format(sess.run(cost, feed_dict={X_init:data, Y_init:data_labels})) )
print("Cost for test  data: {0}".format(sess.run(cost, feed_dict={X_init:test_data, Y_init:test_data_labels}) ) )
sess.close()


I am working on the file "2017-03-Presentation-Jupyter-Notebook.ipynb".
The first task is:
Configuration / Environment
Can you write Python code for it?

!ipython profile locate

# Otherwise run: !ipython profile create

# Sample output:

# [ProfileCreate] Generating default config file: 'C:\\Users\\randall.white\\.ipython\\profile_default\\ipython_config.py'
# [ProfileCreate] Generating default config file: 'C:\\Users\\randall.white\\.ipython\\profile_default\\ipython_kernel_config.py'


I am working on the file "2017-03-Presentation-Jupyter-Notebook.ipynb".
The first task is:
Embed YouTube videos.
Can you write Python code for it?

from IPython.display import YouTubeVideo
YouTubeVideo('HW29067qVWk')


I am working on the file "2017-03-Presentation-Jupyter-Notebook.ipynb".
The first task is:
HTML
Can you write Python code for it?

from IPython.core.display import display, HTML
display(HTML("<style>.container { width:100% !important; }</style>"))

from IPython.core.display import HTML
css = open('style-table.css').read() + open('style-notebook.css').read()
HTML('<style>{}</style>'.format(css))


I am working on the file "2017-03-Presentation-Jupyter-Notebook.ipynb".
The first task is:
Magic

% or %%
debug, time, timeit, run, and more...
Can you write Python code for it?

%time some_list = [x**x for x in range(9001)]

!sudo python3.6 -m pip install matplotlib

%matplotlib notebook
import matplotlib, numpy, pandas, seaborn
from matplotlib import pyplot as plt

ts = pd.Series(numpy.random.randn(1000), index=pandas.date_range('1/1/2000', periods=1000))
ts = ts.cumsum()
df = pandas.DataFrame(numpy.random.randn(1000, 4), index=ts.index,
                  columns=['A', 'B', 'C', 'D'])
df = df.cumsum()
df.plot(); plt.legend(loc='best')


I am working on the file "2017-03-Presentation-Jupyter-Notebook.ipynb".
The first task is:
Notebook specific configuration values

Example location in Windows:

C:\Users\randall.white.jupyter\jupyter_notebook_config.py

Resources

http://jupyter-notebook.readthedocs.io/en/latest/examples/Notebook/Working%20With%20Markdown%20Cells.html
http://sebastianraschka.com/Articles/2014_ipython_internal_links.html
http://www.datadependence.com/2017/02/speed-up-jupyter-notebook-workflow/
http://jupyter-notebook.readthedocs.io/en/latest/examples/Notebook/What%20is%20the%20Jupyter%20Notebook.html
http://nb.bianp.net/sort/views/
http://nbviewer.jupyter.org/
https://anaconda.org/gallery
Can you write Python code for it?

import time

for i in range(5):
    time.sleep(.5)
    print(i)

time.sleep(10)
print("I'm waiting to see this...")


I am working on the file "2017-04-10/numpy_4_matlab_users.ipynb".
The first task is:
 get the number of elements of an array
| MATLAB  |  Numpy |
|---|---|
|numel(a) |size(a) or a.size|
Can you write Python code for it?


I am working on the file "2017-04-10/numpy_4_matlab_users.ipynb".
The first task is:
2x3 matrix literal
| MATLAB  |  Numpy |
|---|---|
|[ 1 2 3; 4 5 6 ] | array([[1.,2.,3.], [4.,5.,6.]])|
Can you write Python code for it?


I am working on the file "2017-04-10/numpy_4_matlab_users.ipynb".
The first task is:
3x3 identity matrix
| MATLAB  |  Numpy |
|---|---|
|eye(3)|np.eye(3)|
Can you write Python code for it?


I am working on the file "2017-04-10/numpy_4_matlab_users.ipynb".
The first task is:
3x4 two-dimensional array full of 64-bit floating point ones
| MATLAB  |  Numpy |
|---|---|
|ones(3,4)|np.ones((3,4))|
Can you write Python code for it?


I am working on the file "2017-04-10/numpy_4_matlab_users.ipynb".
The first task is:
a with elements less than 0.5 zeroed out
| MATLAB  |  Numpy |
|---|---|
|a .* (a>0.5)  |a * (a>0.5)|
Can you write Python code for it?


I am working on the file "2017-04-10/numpy_4_matlab_users.ipynb".
The first task is:
a with elements less than 0.5 zeroed out
| MATLAB  |  Numpy |
|---|---|
|a(a<0.5)=0 |a[a<0.5]=0|
Can you write Python code for it?


I am working on the file "2017-04-10/numpy_4_matlab_users.ipynb".
The first task is:
a with rows in reverse order
| MATLAB  |  Numpy |
|---|---|
|a(end:-1:1,:) or flipud(a)|a[ ::-1,:]|
Can you write Python code for it?


I am working on the file "2017-04-10/numpy_4_matlab_users.ipynb".
The first task is:
access last element in the 1xn matrix a
| MATLAB  |  Numpy |
|---|---|
|a(2,5)  | a[1,4]|
Can you write Python code for it?


I am working on the file "2017-04-10/numpy_4_matlab_users.ipynb".
The first task is:
aconjugate transpose of a
| MATLAB  |  Numpy |
|---|---|
|a'|a.conj().transpose() or a.conj().T|
Can you write Python code for it?


I am working on the file "2017-04-10/numpy_4_matlab_users.ipynb".
The first task is:
construct a matrix from blocks a, b, c, and d
| MATLAB  |  Numpy |
|---|---|
|[ a b; c d ]    | vstack([hstack([a,b]), hstack([c,d])]) or bmat('a b; c d').A|
Can you write Python code for it?


I am working on the file "2017-04-10/numpy_4_matlab_users.ipynb".
The first task is:
create a column vector
| MATLAB  |  Numpy |
|---|---|
|[1:10]'|np.arange(1.,11.)[:, newaxis]  |
3x4 two-dimensional array full of 64-bit floating point zeros
| MATLAB  |  Numpy |
|---|---|
|zeros(3,4) |np.zeros((3,4))|
Can you write Python code for it?


I am working on the file "2017-04-10/numpy_4_matlab_users.ipynb".
The first task is:
create an increasing vector (see note RANGES)
| MATLAB  |  Numpy |
|---|---|
|0:9|np.arange(10.) or r_[:10.] or r_[:9:10j]|
Can you write Python code for it?


I am working on the file "2017-04-10/numpy_4_matlab_users.ipynb".
The first task is:
create an increasing vector (see note RANGES)
| MATLAB  |  Numpy |
|---|---|
|1:10|np.arange(1.,11.) or np.r_[1.:11.] or np.r_[1:10:10j]|
Can you write Python code for it?


I am working on the file "2017-04-10/numpy_4_matlab_users.ipynb".
The first task is:
element-wise multiply
| MATLAB  |  Numpy |
|---|---|
|a .* b |a * b|
Can you write Python code for it?


I am working on the file "2017-04-10/numpy_4_matlab_users.ipynb".
The first task is:
entire second row of a
| MATLAB  |  Numpy |
|---|---|
|a(2,:)| a[1] or a[1,:]|
Can you write Python code for it?


I am working on the file "2017-04-10/numpy_4_matlab_users.ipynb".
The first task is:
every other row of a, starting with the first
| MATLAB  |  Numpy |
|---|---|
|a(1:2:end,:)|a[ ::2,:]|
Can you write Python code for it?


I am working on the file "2017-04-10/numpy_4_matlab_users.ipynb".
The first task is:
get the number of dimensions of an array
| MATLAB  |  Numpy |
|---|---|
|ndims(a) |ndim(a) or a.ndim|
Can you write Python code for it?


I am working on the file "2017-04-10/numpy_4_matlab_users.ipynb".
The first task is:
get the “size” of the matrix
| MATLAB  |  Numpy |
|---|---|
|size(a) |shape(a) or a.shape|
Can you write Python code for it?


I am working on the file "2017-04-10/numpy_4_matlab_users.ipynb".
The first task is:
matrix multiply
| MATLAB  |  Numpy |
|---|---|
|a * b  |a.dot(b)|
Can you write Python code for it?


I am working on the file "2017-04-10/numpy_4_matlab_users.ipynb".
The first task is:
numpy assigns by reference
| MATLAB  |  Numpy |
|---|---|
|y=x    |y = x.copy()   |
Can you write Python code for it?

a=np.random.rand(2,2)
c = a.copy()
b = a
print(a), print(b)
# changing b now changes a (anf vise veras)
b[1,1] = 0
print(a)
print(b)
a[0,0] = -1
print(a)
print(b)
# changing c does not change a (and vise versa)
a[0,1] = -10
print(a)
print(c)
c[1,0] = 20
print(a)
print(c)


I am working on the file "2017-04-10/numpy_4_matlab_users.ipynb".
The first task is:
numpy slices are by reference
| MATLAB  |  Numpy |
|---|---|
|y=x(2,:)|y = x[1,:].copy()|
Can you write Python code for it?

y=a[1,:]
print(a)
print(y)
a[1,1]=1
print(y)


I am working on the file "2017-04-10/numpy_4_matlab_users.ipynb".
The first task is:
turn array into vector (note that this forces a copy)
| MATLAB  |  Numpy |
|---|---|
|y=x(:)|y = x.flatten()|
Can you write Python code for it?


I am working on the file "2017-04-10/numpy_4_matlab_users.ipynb".
The first task is:
vector of diagonal elements of a
| MATLAB  |  Numpy |
|---|---|
|diag(a)|np.diag(a)|
Can you write Python code for it?

a=np.random.rand(4,4)
b = np.diag(a)
print(a)
print(b)


I am working on the file "2017-04-10/xml_processing.ipynb".
The first task is:
Use untangle
untangle is a simple library which takes an XML document and returns a Python object which mirrors the nodes and attributes in its structure
Can you write Python code for it?

import untangle
obj = untangle.parse('input/menu.xml')
obj.breakfast_menu.food[0].calories


I am working on the file "2017-04-10/xml_processing.ipynb".
The first task is:
Use xmltodict
xmltodict is another simple library that aims at making XML feel like working with JSON.
Can you write Python code for it?

import xmltodict

with open('input/menu.xml') as fd:
    doc = xmltodict.parse(fd.read())

print(doc)


I am working on the file "2017-05-08/widgets_list.ipynb".
The first task is:
Accordion
Can you write Python code for it?

accordion = widgets.Accordion(children=[widgets.IntSlider(), widgets.Text()])
accordion.set_title(0, 'Slider')
accordion.set_title(1, 'Text')
accordion


I am working on the file "2017-05-08/widgets_list.ipynb".
The first task is:
BoundedFloatText
Can you write Python code for it?

widgets.BoundedFloatText(
    value=7.5,
    min=0,
    max=10.0,
    step=0.1,
    description='Text:',
    disabled=False,
    color='black'
)


I am working on the file "2017-05-08/widgets_list.ipynb".
The first task is:
Checkbox
Can you write Python code for it?

widgets.Checkbox(
    value=False,
    description='Check me',
    disabled=False
)


I am working on the file "2017-05-08/widgets_list.ipynb".
The first task is:
Date picker
The date picker widget works in Chrome and IE Edge, but does not currently work in Firefox or Safari because they do not support the HTML date input field.
Can you write Python code for it?

widgets.DatePicker(
    description='Pick a Date'
)


I am working on the file "2017-05-08/widgets_list.ipynb".
The first task is:
FloatProgress
Can you write Python code for it?

widgets.FloatProgress(
    value=7.5,
    min=0,
    max=10.0,
    step=0.1,
    description='Loading:',
    bar_style='info',
    orientation='horizontal'
)


I am working on the file "2017-05-08/widgets_list.ipynb".
The first task is:
FloatSlider
Can you write Python code for it?

widgets.FloatSlider(
    value=7.5,
    min=0,
    max=10.0,
    step=0.1,
    description='Test:',
    disabled=False,
    continuous_update=False,
    orientation='horizontal',
    readout=True,
    readout_format='.1f',
    slider_color='white'
)


I am working on the file "2017-05-08/widgets_list.ipynb".
The first task is:
FloatText
Can you write Python code for it?

widgets.FloatText(
    value=7.5,
    description='Any:',
    disabled=False,
    color='black'
)


I am working on the file "2017-05-08/widgets_list.ipynb".
The first task is:
HTML
Can you write Python code for it?

widgets.HTML(
    value="Hello <b>World</b>",
    placeholder='Some HTML',
    description='Some HTML',
    disabled=False
)


I am working on the file "2017-05-08/widgets_list.ipynb".
The first task is:
HTML Math
Can you write Python code for it?

widgets.HTMLMath(
    value=r"Some math and <i>HTML</i>: $x^2$ and $$\frac{x+1}{x-1}$$",
    placeholder='Some HTML',
    description='Some HTML',
    disabled=False
)


I am working on the file "2017-05-08/widgets_list.ipynb".
The first task is:
IntProgress
Can you write Python code for it?

widgets.IntProgress(
    value=7,
    min=0,
    max=10,
    step=1,
    description='Loading:',
    bar_style='', # 'success', 'info', 'warning', 'danger' or ''
    orientation='horizontal'
)


I am working on the file "2017-05-08/widgets_list.ipynb".
The first task is:
Label
Can you write Python code for it?

widgets.Label(
    value="$$\\frac{n!}{k!(n-k)!} = \\binom{n}{k}$$",
    placeholder='Some LaTeX',
    description='Some LaTeX',
    disabled=False
)


I am working on the file "2017-05-08/widgets_list.ipynb".
The first task is:
Layout widgets
Box
HBox
Can you write Python code for it?

items = [widgets.Label(str(i)) for i in range(4)]
widgets.HBox(items)


I am working on the file "2017-05-08/widgets_list.ipynb".
The first task is:
Numeric widgets
There are 8 widgets distributed with IPython that are designed to display numeric values.  Widgets exist for displaying integers and floats, both bounded and unbounded.  The integer widgets share a similar naming scheme to their floating point counterparts.  By replacing Float with Int in the widget name, you can find the Integer equivalent.
IntSlider
Can you write Python code for it?

widgets.IntSlider(
    value=7,
    min=0,
    max=10,
    step=1,
    description='Test:',
    disabled=False,
    continuous_update=False,
    orientation='horizontal',
    readout=True,
    readout_format='i',
    slider_color='white'
)


I am working on the file "2017-05-08/widgets_list.ipynb".
The first task is:
Play (Animation) widget
The Play widget is useful to perform animations by iterating on a sequence of integers with a certain speed.
Can you write Python code for it?

play = widgets.Play(
#     interval=10,
    value=50,
    min=0,
    max=100,
    step=1,
    description="Press play",
    disabled=False
)
slider = widgets.IntSlider()
widgets.jslink((play, 'value'), (slider, 'value'))
widgets.HBox([play, slider])


I am working on the file "2017-05-08/widgets_list.ipynb".
The first task is:
RadioButtons
Can you write Python code for it?

widgets.RadioButtons(
    options=['pepperoni', 'pineapple', 'anchovies'],
#     value='pineapple',
    description='Pizza topping:',
    disabled=False
)


I am working on the file "2017-05-08/widgets_list.ipynb".
The first task is:
SelectMultiple
Multiple values can be selected with shift and/or ctrl (or command) pressed and mouse clicks or arrow keys.
Can you write Python code for it?

widgets.SelectMultiple(
    options=['Apples', 'Oranges', 'Pears'],
    value=['Oranges'],
    #rows=10,
    description='Fruits',
    disabled=False
)


I am working on the file "2017-05-08/widgets_list.ipynb".
The first task is:
SelectionSlider
Can you write Python code for it?

widgets.SelectionSlider(
    options=['scrambled', 'sunny side up', 'poached', 'over easy'],
    value='sunny side up',
    description='I like my eggs ...',
    disabled=False,
    continuous_update=False,
    orientation='horizontal',
    readout=True,
#     readout_format='i',
#     slider_color='black'
)


I am working on the file "2017-05-08/widgets_list.ipynb".
The first task is:
Sliders can also be displayed vertically.
Can you write Python code for it?

widgets.FloatSlider(
    value=7.5,
    min=0,
    max=10.0,
    step=0.1,
    description='Test:',
    disabled=False,
    continuous_update=False,
    orientation='vertical',
    readout=True,
    readout_format='.1f',
    slider_color='white'
)


I am working on the file "2017-05-08/widgets_list.ipynb".
The first task is:
String widgets
There are 4 widgets that can be used to display a string value.  Of those, the Text and Textarea widgets accept input.  The Label and HTML widgets display the string as either Label or HTML respectively, but do not accept input.
Text
Can you write Python code for it?

widgets.Text(
    value='Hello World',
    placeholder='Type something',
    description='String:',
    disabled=False
)


I am working on the file "2017-05-08/widgets_list.ipynb".
The first task is:
Tabs
Can you write Python code for it?

list = ['P0', 'P1', 'P2', 'P3', 'P4']
children = [widgets.Text(description=name) for name in list]
tab = widgets.Tab(children=children)
tab


I am working on the file "2017-05-08/widgets_list.ipynb".
The first task is:
The following is also valid:
Can you write Python code for it?

widgets.Dropdown(
    options={'One': 1, 'Two': 2, 'Three': 3},
    value=2,
    description='Number:',
)


I am working on the file "2017-07-05.ipynb".
The first task is:
An iterator eventually expires... it runs out of stuff and then raises StopIteration. Push it beyond its limit at your own risk.
Can you write Python code for it?

next(my_first_iterator)

my_second_iterator = iter(my_list)

my_third_iterator = iter(my_list)

next(my_second_iterator)

next(my_second_iterator)

next(my_third_iterator)

my_first_iterator()

dir(my_first_iterator)

my_string = "abcdefg"

for i in my_string:
    print(i)

my_string = "ijklmno"

next(my_string)

dir(str)

my_iter_string = iter(my_string)

next(my_iter_string)

dir(my_iter_string)


I am working on the file "2017-07-05.ipynb".
The first task is:
And now for the coolness... a generator comprehension:
Can you write Python code for it?

my_generator = (x * 100 for x in range(10))

type(my_generator)

for i in my_generator:
    print(i - 3)


I am working on the file "2017-07-05.ipynb".
The first task is:
And now the fancy "syntatic sugar" way of creating the decorator.
Can you write Python code for it?

@timer
def even_slower_func():
    sleep(5)

even_slower_func()


I am working on the file "2017-07-05.ipynb".
The first task is:
Decorators
Can you write Python code for it?

def my_decorator(func):
    def inner():
        print("Running my_decorator")
    return inner

def other_func():
    print("Running other_func")

other_func()


I am working on the file "2017-07-05.ipynb".
The first task is:
Let's check this out. What has the call to iter() actually returned?
Can you write Python code for it?

type(my_list)

type(my_first_iterator)

dir(my_list)

dir(my_first_iterator)

for i in my_list:
    print(i)

for i in my_first_iterator:
    print(i)


I am working on the file "2017-07-05.ipynb".
The first task is:
Now we're actuly going to do something with the func that is passed in. We're going to call it right in the middle of our new timer decorator. We do a little work before it is called -- we take a snapshot of the system clock with time() -- and a little more after -- another call to time(). This is the essence of a decorator.
Can you write Python code for it?

def timer(func):
    def wrapper():
        t1 = time()
        func()
        t2 = time()
        print("{} duration is {}".format(func.__name__, t2 - t1))
    return wrapper

def slow_func():
    sleep(2)


I am working on the file "2017-07-05.ipynb".
The first task is:
Stacked decorators
Can you write Python code for it?

def italic(fn):
    def _():
        return "" + fn() + ""
    return _

def bold(fn):
    def _():
        return "" + fn() + ""
    return _

@bold
@italic
def hello():
    return "hello world"

print(hello())


I am working on the file "2017-07-19.ipynb".
The first task is:
First, the non-functional approach. Notice the pattern from Guido's blog post?'
Can you write Python code for it?

def draw(x, y):

    if isinstance(x, Rock):
        if isinstance(y, Rock):
            return None  # No winner
        if isinstance(y, Paper):
            return y
        if isinstance(y, Scissors):
            return x
        else:
            raise TypeError("Unknown type involved")

    elif isinstance(x, Paper):
        if isinstance(y, Rock):
            return x
        if isinstance(y, Paper):
            return None  # No winner
        if isinstance(y, Scissors):
            return y
        else:
            raise TypeError("Unknown type involved")

    elif isinstance(x, Scissors):
        """ This method left as a exercise for the reader. """
        pass

draw(player1(), player2())


I am working on the file "2017-07-19.ipynb".
The first task is:
Multiple dispatch
There are several multiple dispatch libraries for python. I got confused as to which I had installed into which virtualenv during class. Sorry about that.
https://en.wikipedia.org/wiki/Multiple_dispatch
http://www.artima.com/weblogs/viewpost.jsp?thread=101605
Can you write Python code for it?

import random

# https://github.com/mrocklin/multipledispatch/
from multipledispatch import dispatch

class Thing(object):
    pass

class Paper(Thing):
    pass

class Scissors(Thing):
    pass

class Rock(Thing):
    pass

options = [Paper, Scissors, Rock]

player1 = random.choice(options)

player2 = random.choice(options)

player1

player2


I am working on the file "2017-07-19.ipynb".
The first task is:
Now the more functional, multi-dispatch method
Can you write Python code for it?

@dispatch(Rock, Rock)
def draw(x, y):
    return None

@dispatch(Paper, Scissors)
def draw(x, y):
    return(y)

# This is by no means all of the combinations. Here again they're left as a exercise for the reader.

winner = draw(player1(), player2())

(isinstance(winner, player1) and print("Player1 won... bam!"))

(isinstance(winner, player2) and print("Player2 won... bowzza!"))


I am working on the file "2017-07-19.ipynb".
The first task is:
Remember the general form:
if <cond1>:
    func1()
Can you write Python code for it?

(cond1 and func1())

cond1 = False

(cond1 and func1())

cond1 = True
def func1(): return True

(True and func1())


I am working on the file "2017-07-19.ipynb".
The first task is:
Remember to think of cond1 and cond2 as expressions even when using literals; in other words....
Can you write Python code for it?

x = 1
cond1 = (3 > x)

cond1

(cond1 and func1())

((3 > x) and func1())

((3 < x) and func1())

x = 5
cond2 = (3 > x)

cond2

func1()

((cond1) and func1())

((cond2) and func1())

x = 5
cond1 = (3 > x)

(True and func1())


I am working on the file "2017-07-19.ipynb".
The first task is:
Same thing as a comprehension
Can you write Python code for it?

my_doubled_list = [i * 2 for i in my_list]

print(*my_doubled_list)

def my_filter(x):
    return x > 900

my_filtered_list = filter(my_filter, my_list)

print(*my_filtered_list)

my_doubled_list = map(lambda x: x*2, filter(my_filter, my_list))

print(*my_doubled_list)

my_filtered_list = filter(my_filter, my_list)

type(my_filtered_list)


I am working on the file "2017-07-19.ipynb".
The first task is:
Same thing with a lambda
Can you write Python code for it?

my_doubled_list = map(lambda x: x*2, my_list)


I am working on the file "2017-07-19.ipynb".
The first task is:
my_filtered_list is a filter
and a filter is an iterator
https://docs.python.org/3/library/functions.html#filter
so like any other iterator it can be used only once,
so don't spend it here in the print statement if you want to use it below
print(*my_filtered_list)
Can you write Python code for it?

my_doubled_list = map(lambda x: x*2, my_filtered_list)

print(*my_doubled_list)


I am working on the file "2017-07-19.ipynb".
The first task is:
reduce
Can you write Python code for it?

from functools import reduce

my_list = list(range(100))

print(*my_list)

sum(my_list)

min(my_list)

max(my_list)

my_sum = reduce(lambda x, y: x+y, my_list)

my_sum


I am working on the file "2017-09-13-MeetIT-Torun/klimat.ipynb".
The first task is:
Co to mówi? Coraz ciemniejsze sześciokąty w górnej części wykresu w miarę upływu czasu wskazują na coraz częstsze odchylenia temperatury w górę.
Średnie temperatury w jednym miesiącu
Wybrałem jeden miesiąc (taki jak jest teraz) i sprawdziłem, jak często występowała dana temperatura. Sprawdźmy jak to było porównując wszystkie dostępne dane (1753-2013) z okresem po 1970 oraz 1995 roku (niedługo przed El Niño).
Warto zwrócić uwagę, że jeśli miesiącem zdefiniowanym w MONTH jest wrzesień, zmiana nie jest duża. Wynika to z tego, że ten akurat miesiąc aż tak bardzo się nie ocieplił na przestrzeni lat (co widać tam, gdzie jest tak dużo wykresów :) ). Polecam zmienić wartość zmiennej MONTH i uruchomić dwie komórki poniżej.
Can you write Python code for it?

before_1970 = month_city[month_city['Year'] < RISE_START_YEAR]['AverageTemperature'].copy()
after_1970 = month_city[month_city['Year'] >= RISE_START_YEAR]['AverageTemperature'].copy()
after_1995 = month_city[month_city['Year'] >= 1995]['AverageTemperature'].copy()

fig, ax = plt.subplots()
sns.distplot(before_1970, hist=False, ax=ax)
sns.distplot(after_1970, hist=False, ax=ax)
sns.distplot(after_1995, hist=False, ax=ax)


I am working on the file "2017-09-13-MeetIT-Torun/klimat.ipynb".
The first task is:
Następny wykres przedstawia zmiany średniej temperatury na przestrzeni lat we wszystkich miesiącach. Dokładniej rzecz ujmując nie jest to średnia, tylko krzywa regresji.
Can you write Python code for it?

sns.lmplot(y="AverageTemperature", x="Year", data=climate_city,
           lowess=True, markers=".", col="Month", col_wrap=3, sharex=False, sharey=False)


I am working on the file "2017-09-13-MeetIT-Torun/klimat.ipynb".
The first task is:
Od 1753 roku mamy pełne dane. Widać też, że są to średnie z całego miesiąca.
Na początek sprawdźmy jaka była średnia temperatura w aktualnym miesiącu od 1753 i 1970 roku. W 1970 uśredniona temperatura całego świata zaczęła dramatycznie rosnąć.
Należy wziąć pod uwagę, że naukowcy badający klimat odchylenia od średniej mierzą względem okresu 1850-1900 (o tym więcej poniżej).
Can you write Python code for it?

print("Average temperature in {city} in {from_year}-2013: {temperature}".format(
    city=calendar.month_name[MONTH],
    from_year=START_YEAR,
    temperature=month_city['AverageTemperature'].mean()
))
print("Average temperature in {city} in {from_year}-2013: {temperature}".format(
    city=calendar.month_name[MONTH],
    from_year=RISE_START_YEAR,
    temperature=current_month_city['AverageTemperature'].mean()
))
print("Average temperature in {city} in {from_year}-{to_year}: {temperature} (baseline)".format(
    city=calendar.month_name[MONTH],
    from_year=BASE_YEAR_START,
    to_year=BASE_YEAR_END,
    temperature=baseline_month_city['AverageTemperature'].mean()
))


I am working on the file "2017-09-13-MeetIT-Torun/klimat.ipynb".
The first task is:
Poniżej zaś można zobaczyć zakresy temperatur, jakie zdarzały się w całym zbiorze danych przez te nieco ponad 200 lat.
Can you write Python code for it?


I am working on the file "2017-09-13-MeetIT-Torun/klimat.ipynb".
The first task is:
Poniższy wykres przedstawia rozkład temperatur w całym roku i w miesiącu zdefiniowanym w MONTH.
Can you write Python code for it?

fig, (ax1, ax2) = plt.subplots(ncols=2, sharey=False)

sns.distplot(climate_city['AverageTemperature'], bins=50, rug=True, ax=ax1)
ax1.set_title("Whole year")
sns.distplot(month_city['AverageTemperature'], bins=50, rug=True, ax=ax2)
ax2.set_title("Current month")
plt.show()


I am working on the file "2017-09-13-MeetIT-Torun/klimat.ipynb".
The first task is:
Taka prezentacja danych nie mówi jednak wiele o tym, co jest szczególnie istotne w kontekście globalnego ocieplenia, czyli o trendzie. Sprawdźmy jak to wygląda.
Can you write Python code for it?

from scipy.stats import spearmanr

sns.jointplot(month_city['Year'], month_city['AverageTemperature'],
              kind="hex", stat_func=spearmanr)


I am working on the file "2017-09-13-MeetIT-Torun/klimat.ipynb".
The first task is:
Teraz wczytujemy dane z udostępnionego zbioru i czyścimy je z niepotrzebnych w tej analizie informacji.
Can you write Python code for it?

df = pd.read_csv('./data/climate/GlobalLandTemperaturesByCityPoland.csv')

date_index = pd.to_datetime(df['dt'], format='%Y-%m-%d', errors='ignore')

df['di'] = date_index
df['Year'] = df['di'].dt.year
df['Month'] = df['di'].dt.month
df['Day'] = df['di'].dt.day

poland = df
poland_clean = poland.copy()

poland_clean.drop('Latitude', axis=1, inplace=True)
poland_clean.drop('Longitude', axis=1, inplace=True)
poland_clean.drop('AverageTemperatureUncertainty', axis=1, inplace=True)
poland_clean.drop('Country', axis=1, inplace=True)
poland_clean.drop('dt', axis=1, inplace=True)

city = df[df['City'] == CITY].dropna().copy()
climate_city = city[city['Year'] >= START_YEAR].copy()
month_city = climate_city[climate_city['Month'] == MONTH].copy()
current_month_city = month_city[month_city['Year'] > RISE_START_YEAR]
baseline_month_city = month_city[
    month_city['Year'].between(BASE_YEAR_START, BASE_YEAR_END)].copy()


I am working on the file "2017-09-13-MeetIT-Torun/klimat.ipynb".
The first task is:
W drugim i trzecim przypadku danych oczywiście nie ma zbyt dużo (43 i 19 pomiarów), ale da się zauważyć, że zwiększa się prawdopodobieństwo wystąpienia temperatury w okolicach 19 stopni.
Kostka klimatyczna
Kostka klimatyczna jest koncepcją, która ma ułatwić zrozumienie tego, że globalne ocieplenie, zwłaszcza w początkowej fazie, nie oznacza, że nie będzie zim w ogóle, a tylko to, że będą one rzadziej.
Załóżmy, że pomalowaliśmy ściany kostki na trzy kolory: czerwony, biały i niebieski oznaczające odpowiednio od wysokich, przez średnie, do niskich temperatur. Dostosujmy te przedziały tak, żeby prawdopodobieństwa były analogiczne do szans, że wypadnie ściana kostki zamalowana danym kolorem.
Globalne ocieplenie sprawia, że dodatkowe ściany kostki zostają zamalowane na czerwono.
W tym notebooku zmieniłem trochę tę koncepcję — zamiast ścian kostki, pokazuję liczbę miesięcy w roku, które uznalibyśmy za ciepłe, normalne lub chłodne (ciepłe jak na styczeń, itd.) Zobaczmy, jak to wygląda.
Can you write Python code for it?

def label_bin(row):
    if row['AverageTemperature'] < (row['MonthMeanBaseline'] - row['MonthStdBaseline']):
        return 'Low'
    elif row['AverageTemperature'] > (row['MonthMeanBaseline'] + row['MonthStdBaseline']):
        return 'High'
    return 'Medium'

monthly_frames = []
for i, group in climate_city.groupby('Month'):
    g2 = group.copy()
    g_temp_series = g2[(g2['Year'] >= BASE_YEAR_START) & (g2['Year'] <= BASE_YEAR_END)]['AverageTemperature']
    g2['MonthMeanBaseline'] = g_temp_series.mean()
    g2['MonthStdBaseline'] = g_temp_series.std()
    g2['TemperatureBin'] = g2.apply(label_bin, axis=1)
    monthly_frames.append(g2)

all_with_bins = pd.concat(monthly_frames).sort_index()
city_grouped_with_counted_bins = all_with_bins.groupby(['Year','TemperatureBin'])['City'] \
    .agg(['count']).reset_index()

sns.lmplot(x='Year', y='count', hue='TemperatureBin',
           data=city_grouped_with_counted_bins,
           scatter=False, lowess=True)


I am working on the file "2017-09-13-MeetIT-Torun/klimat.ipynb".
The first task is:
Wybierzmy kilka i zobaczmy, jak się zmieniała dla nich temperatura na przestrzeni lat.
To co widać na poniższym wykresie to znaczący wzrost średnich temperatur od początku XX wieku. Dane kończą się na roku 2013 i do tej chwili średnia temperatura wzrosła o 1 stopień.
Ciekawe jest to, że najchłodniejsze z miast (Białystok) ma średnią temperaturę niższą o ponad 2 stopnie od najcieplejszego (Zielonej Góry).
Can you write Python code for it?

sns.lmplot(x='Year', y='AverageTemperature', hue='City', data=poland[poland['City'].isin(SELECTED_CITIES)],
           scatter=False, lowess=True)


I am working on the file "2017-1/Metodos/Aula 05 - Revisão.ipynb".
The first task is:
Metodos de newton e secante
Can you write Python code for it?

%matplotlib inline
import numpy as np
from matplotlib import pyplot as plt

# Segundo passo: Definir função
def f(x):
    return -x * np.e ** -x + 0.2

def f_linha(x):
    return np.e ** -x * (x - 1)

def p(A, B):
    plt.xlabel('x')
    plt.ylabel('y = f(x)')
    plt.title('Zero de funcoes')
    plt.grid()
    plt.plot(x, y)

    if A != 0 and B != 0:
        [xmin, xmax, ymin, ymax] = plt.axis()
        plt.plot([A, A], [ymin, ymax], 'k-')
        plt.plot([B, B], [ymin, ymax], 'k-')

# Terceiro passo: Visualizar função
x = np.linspace(0.0, 10.0, 100)
y = f(x)

p(1.0, 10.0)
plt.show()

A = 1.0
B = 10.0

xi = 2.0
x_tol = 0.0001
y_tol = 0.0001
x_prev = xi

while True:
    p(0, 0)
    [xmin, xmax, ymin, ymax] = plt.axis()
    plt.plot([xi, xi], [ymin, ymax], '--')
    plt.show()

    print('Aproximacao: %.6lf' %f(xi))

    # Termo de recursao
    xi = xi - f(xi) / f_linha(xi)

    if abs(f(xi)) < y_tol:
        break

    if abs(x_prev - xi) < x_tol:
        break

    x_prev = xi

plt.show(block=True)


I am working on the file "2017-1/Metodos/Aula 05 - Revisão.ipynb".
The first task is:
Posição Falsa

Só troca a funcao $(A + B) / 2$ por $(A * f(B) - B * f(A)) / (f(B) - f(A))$
Can you write Python code for it?

%matplotlib inline
import numpy as np
from matplotlib import pyplot as plt

# Segundo passo: Definir função
def f(x):
    return -x * np.e ** -x + 0.2

def p(A, B):
    plt.xlabel('x')
    plt.ylabel('y = f(x)')
    plt.title('Zero de funcoes')
    plt.grid()
    plt.plot(x, y)

    [xmin, xmax, ymin, ymax] = plt.axis()
    plt.plot([A, A], [ymin, ymax], 'k-')
    plt.plot([B, B], [ymin, ymax], 'k-')

# Terceiro passo: Visualizar função
x = np.linspace(0.0, 10.0, 100)
y = f(x)

p(1.0, 10.0)
plt.show()

A = 1.0
B = 10.0

x_tol = 0.001
x_prev = B
y_tol = 0.0001

while True:
    # Termo de recursao
    xi = (A * f(B) - B * f(A)) / (f(B) - f(A))

    p(A, B)
    [xmin, xmax, ymin, ymax] = plt.axis()
    plt.plot([xi, xi], [ymin, ymax], '--')
    plt.show()

    print('Aproximacao: %.6lf' %f(xi))

    if (f(A) * f(xi)) < 0:
        B = xi
    elif (f(A) * f(xi)) == 0:
        print('Raiz encontrada em %6.f' % xi)
    else:
        A = xi

    if abs(f(xi)) < y_tol:
        break

    if abs(x_prev - xi) < x_tol:
        break

    x_prev = xi

plt.show(block=True)


I am working on the file "2017-1/Metodos/Listas/.ipynb_checkpoints/L04_DaniloBarrosMendes_120114780-checkpoint.ipynb".
The first task is:
Questão 03
Como mencionado no exercício anterior, o método de Gauss-Jordan também pode ser utilizado para encontrar a inversa de matrizes quadradas não-singulares.
Dado um inteiro positivo $N$, e todos os elementos de uma matriz quadrada $A$ de ordem $N$, calcule $A^{-1}$ utilizando o método de Gauss-Jordan. Não utilize funções prontas para realizar o cálculo.
Entrada
A primeira linha de entrada contém um inteiro positivo $N$. As próximas $N$ linhas contém $N$ inteiros cada, representando cada um dos elementos da matriz $A$.
Saída
Na saída devem ser impressas $N$ linhas com $N$ valores cada, representando cada um dos elementos da matriz inversa, conforme o caso de teste abaixo.
Entrada de Teste
3
3.0 3.0 -1.0
2.0 6.0 2.0
8.0 4.0 1.0
Saída de Teste
-0.0263 -0.0921  0.1579
 0.1842  0.1447 -0.1053
-0.5263  0.1579  0.1579
Can you write Python code for it?

def matrix_input(N):
    A = np.zeros((N, N))
    for i in range(N):
        for j in range(N):
            A[i, j] = float(input())

    return A

def array_input(N):
    b = np.zeros(N)
    for i in range(N):
        b[i] = float(input())
    return b

def inverse(A):
    length = len(A[0])
    inverse = np.zeros((length, length))

    for i in range(length):
        inverse[i, i] = 1

    inverse = gauss_jordan(A, inverse)

    return inverse

N = int(input())

A = matrix_input()
inverse(A)


I am working on the file "2017-1/Metodos/Listas/.ipynb_checkpoints/L04_DaniloBarrosMendes_120114780-checkpoint.ipynb".
The first task is:
Questão 04
O Método de Jacobi é um método utilizado para encontrar a solução de sistemas lineares de forma numérica, ou seja, por aproximações sucessivas. O método funiona da seguinte forma: dado um sistema linear $Ax = b$ e uma aproximação inicial para os valores de $x$, a próxima aproximação para os valores de $x$ são calculadas através da equação:
$$
x^{(novo)}i = \frac{b_i \sum{j \neq i}{a_{ij}x_j^{(antigo)}}}{a_{ii}}
$$
Dado um sistema linear, utilize o Método de Jacobi para encontrar as 10 (dez) primeiras aproximações para os elementos do vetor de incógnitas ($x$). Assuma que todos os elementos de $x$ são iguais a $1$ na aproximação inicial. Não utilize funções prontas para realizar o cálculo.
Entrada
A primeira linha da entrada consiste de um valor inteiro $N$ ($2 \leq N \leq 10$), indicando a ordem do sistema linear. A segunda linha consiste de $N$ valores de ponto flutuante, indicando cada um dos termos independentes do sistema linear. As próximas $N$ linhas consistem de $N$ valores de ponto flutuante cada, representando cada um dos elementos da matriz de coeficientes.
Saída
A saída deve conter 10 linhas, a linha $i$ ($1 \leq i \leq N$) deve conter $N$ valores de ponto flutuante com 6 casas decimais de precisão, representando cada um dos elementos da i-ésima aproximação do vetor $x$.
Entrada de Teste
2
8.0 10.0
2.0 2.0
1.0 4.0
Saída de Teste
1.000000  1.000000
3.000000  2.250000
1.750000  1.750000
2.250000  2.062500
1.937500  1.937500
2.062500  2.015625
1.984375  1.984375
2.015625  2.003906
1.996094  1.996094
2.003906  2.000977
Can you write Python code for it?

def jacobi(A, b, x):
    x_novo = np.zeros(len(x))

    for i in range(len(x)):
        r = 0
        for j in range(len(x)):
            if i == j:
                continue
            r += A[i, j] * x[j]
        x_novo[i] = (b[i] - r) / A[i, i]

    x = x_novo
    print(x)
    return x

N = int(input())
A = matrix_input(N)
b = array_input(N)

b = b.T
x = np.ones(2)
for i in range(10):
    x = jacobi(A, b, x)


I am working on the file "2017-1/Metodos/Listas/.ipynb_checkpoints/L04_DaniloBarrosMendes_120114780-checkpoint.ipynb".
The first task is:
Questão 05
Métodos numéricos de integração são geralmente utilizados quando se deseja computar integrais definidas de funções muito complicadas para que tal cálculo seja realizado de forma analítica.
O Método do Ponto Médio é um método de integração numérica que aproxima a área sob a curva por uma soma de retângulos com largura definida e não infinitesimal, diferente da Soma de Riemann.
Seja $f(x)$ a função:
$$ f(x) = xcos(x)^2 $$
Dados os limites $a$ e $b$ e o tamanho da largura dos retângulos $h$, calcule $\int_a^b{f(x)dx}$ utilizando o Método do Ponto Médio.
Entrada
A entrada consiste de uma única linha contendo três valores de ponto flutuante: $a$, $b$ e $h$, respectivamente.
Saída
A saída deve conter um único valor de ponto flutuante com 4 casas decimais de precisão, representando o valor da integral encontrada.
Entrada de Teste
0.0 10.0 1.0
Saída de Teste
27.6559
Can you write Python code for it?

def f(x):
    return x * np.cos(x) ** 2

def mid_point(f, a, b, h):
    N = int((b - a) / h)
    I = h * sum(f(h/2 + a + i * h) for i in range(N))
    return I

a = float(input())
b = float(input())
h = float(input())

print('%.4f' % mid_point(f, a, b, h))


I am working on the file "2017-1/Metodos/Listas/L02_DaniloBarrosMendes_120114780.ipynb".
The first task is:
Questão 02
Em zeros de funções, o Método da Newton é um método iterativo utilizado para encontrar a raíz de uma função, dado um ponto inicial. A cada iteração o método encontra uma nova aproximação para o local da raíz da função desejada.
Dado um ponto inicial $x_0$, utilize o Método de Newton para encontrar as 10 primeiras aproximações para a raíz da função:
$$ f(x) = x^4 - 10x^3 + 8x $$
Entrada
A entrada consiste de um valor real $-10^4 \leq x_0 \leq 10^4$ que indica o ponto inicial.
Saída
Na saída devem ser impressos 10 valores reais com 6 casas de precisão, cada valor representa uma aproximação para a raíz da função $f(x)$ dada. Após cada valor deve ser impressa uma quebra de linha.
Entrada de Teste
$\texttt{8.00}$
Saída de Teste
$\texttt{8.000000}$
$\texttt{15.058824}$
$\texttt{12.524640}$
$\texttt{10.923665}$
$\texttt{10.137268}$
$\texttt{9.932064}$
$\texttt{9.918737}$
$\texttt{9.918683}$
$\texttt{9.918683}$
$\texttt{9.918683}$
Can you write Python code for it?

def g(x):
    return x**4 - 10 * x**3 + 8 * x

def g_linha(x):
    return 4*x**3 - 30*x**2 + 8

x0 = 8.00

x = x0

for i in range(10):
    value = g(x)
    diff = g_linha(x)
    print('%.6f' % x)
    x = x - value / diff


I am working on the file "2017-1/Metodos/Listas/L02_DaniloBarrosMendes_120114780.ipynb".
The first task is:
Questão 03
O Método da Posição Falsa é um método iterativo utilizado para encontrar a raíz de uma função, dado um intervalo inicial. A cada iteração o método encontra uma nova aproximação para o local da raíz da função desejada.
Dado um intervalo inicial $[A,B]$, utilize o Método da Posição Falsa para encontrar as 10 primeiras aproximações para a raíz da função:
$$ f(x) = \frac{x^2 + 3x - 3}{x^2 + 1} $$
Entrada
A entrada consiste de dois valores reais $-10^4 \leq A < B \leq 10^4$ que indicam os limites inferior e superior do intervalo inicial, respectivamente.
Saída
Na saída devem ser impressos 10 valores reais com 6 casas de precisão. Após cada valor deve ser impressa uma quebra de linha. Caso a raíz seja encontrada, esta deve ser a última linha impressa, também com 6 casas decimais de precisão. Caso o intervalo inicial não seja válido, a mensagem "Intervalo invalido" deve ser impressa.
Entrada de Teste
$\texttt{-1.00}$
$\texttt{10.00}$
Saída de Teste
$\texttt{6.318841}$
$\texttt{3.733514}$
$\texttt{1.971825}$
$\texttt{0.908998}$
$\texttt{0.702668}$
$\texttt{0.799252}$
$\texttt{0.791794}$
$\texttt{0.791320}$
$\texttt{0.791290}$
$\texttt{0.791288}$
Can you write Python code for it?

def y(x):
    return (x**2 + 3*x - 3)/(x**2 + 1)

A = -1.00
B = 10.00

if A >= B:
    print("Intervalo invalido")
else:
    xa = A
    xb = B
    ga = y(xa)
    gb = y(xb)

    for i in range(10):
        x_pfalse = (gb * xa - ga * xb) / (gb - ga)
        g_pfalse = y(x_pfalse)

        if g_pfalse < 0:
            xa = x_pfalse
            ga = g_pfalse
        elif g_pfalse > 0:
            xb = x_pfalse
            gb = g_pfalse
        elif g_pfalse == 0:
            print('%.6f' % x_pfalse)
            break

        print('%.6f' % x_pfalse)


I am working on the file "2017-1/Metodos/Listas/L02_DaniloBarrosMendes_120114780.ipynb".
The first task is:
Questão 06
O Método da Busca pela Razão Áurea é um método para encontrar pontos críticos de uma função dentro de um intervalo definido, de forma iterativa. A cada iteração o método encontra uma nova aproximação para o local do ponto crítico (máximo ou mínimo) da função.
Dado um polinômio de quarta ordem e um intervalo inicial $[A, B]$, utilize o Método da Busca pela Razão Áurea para encontrar as 10 primeiras aproximações para o ponto de mínimo do polinômio dado.
Entrada
A entrada é composta por duas linhas. A primeira linha contêm cinco valores reais $-10^4 \leq a,b,c,d,e \leq 10^4$ que indicam os coeficientes do polinômio de quarta ordem abaixo.
$$ p(x) = ax^4 + bx^3 + cx^2 + dx + e $$
A segunda linha consiste de dois valores reais $-10^4 \leq A,B \leq 10^4$ que indicam os limites inferior e superior do intervalo, respectivamente.
Saída
Na saída devem ser impressos 10 valores reais com 6 casas de precisão. Após cada valor deve ser impressa uma quebra de linha.
Entrada de Teste
$\texttt{1.00 0.00 -1.00 0.00 1.00}$
$\texttt{0.00}$
$\texttt{10.00}$
Saída de Teste
$\texttt{3.819660}$
$\texttt{2.360680}$
$\texttt{1.458980}$
$\texttt{0.901699}$
$\texttt{0.557281}$
$\texttt{0.344419}$
$\texttt{0.557281}$
$\texttt{0.688837}$
$\texttt{0.638587}$
$\texttt{0.688837}$
Can you write Python code for it?

a = 1.0
b = 0.0
c = -1.0
d = 0.0
e = 1.0

def h(x):
    return a * x**4 + b * x**3 + c * x**2 + d * x + np.e

A = 0.0
B = 10.0
golden = (1 + 5 ** 0.5) / 2

xa = B - (B - A)/golden
xb = A + (B - A)/golden
fa = h(xa)
fb = h(xb)
print('%.6f' % (xa if fa < fb else xb))

for i in range(9):
    if fa > fb:
        A = xa
        xa = xb
        fa = fb
        xb = A + (B - A)/golden
        fb = h(xb)
    else:
        B = xb
        xb = xa
        fb = fa
        xa = B - (B - A)/golden
        fa = h(xa)
    print('%.6f' % (xa if fa < fb else xb))


I am working on the file "2017-1/Metodos/Listas/L02_DaniloBarrosMendes_120114780.ipynb".
The first task is:
Questão 07
O Método da Descida de Gradiente é um método iterativo utilizado para encontrar o ponto crítico de uma função, dado um ponto inicial. A cada iteração o método encontra uma nova aproximação para o local do ponto crítico da função desejada. Esta nova aproximação pode ser encontrada utilizando duas abordagens: passo contante e passo variável.
Dado um ponto inicial $x_0$, utilize o Método da Descida de Gradiente com Passo Constante para encontrar as 10 primeiras aproximações para o ponto crítico da função:
$$ f(x) = -xe^{-x} $$
Entrada
A entrada consiste de dois números reais. O primeiro valor é $-10^4 \leq x_0 \leq 10^4$ que indica o ponto inicial. O segundo valor é $0 < k \leq 10$ que indica o passo constante $k$ do método.
Saída
Na saída devem ser impressas dez linhas com a seguinte mensagem "$i\texttt{: }f(x_i)\texttt{ = }y_i$", onde $i$ é o número da iteração, $x_i$ é o local da aproximação atual para o ponto crítico da função e $y_i$ é o valor de $f(x)$ no ponto $x_i$.
Entrada de Teste
$\texttt{7.50}$
$\texttt{1.00}$
Saída de Teste
$\texttt{0: f(7.500000) = -0.004148}$
$\texttt{1: f(6.500000) = -0.009772}$
$\texttt{2: f(5.500000) = -0.022477}$
$\texttt{3: f(4.500000) = -0.049990}$
$\texttt{4: f(3.500000) = -0.105691}$
$\texttt{5: f(2.500000) = -0.205212}$
$\texttt{6: f(1.500000) = -0.334695}$
$\texttt{7: f(0.500000) = -0.303265}$
$\texttt{8: f(1.500000) = -0.334695}$
$\texttt{9: f(0.500000) = -0.303265}$
Can you write Python code for it?


I am working on the file "2017-1/Metodos/Listas/L04_DaniloBarrosMendes_120114780.ipynb".
The first task is:
Questão 01
O Método da Eliminação de Gauss é um método analítico de resolução de sistemas lineares.
Dado o sistema linear $Ax = b$, onde:
$$
A =
\begin{bmatrix}
3 & 3 & -1 \
2 & 6 & 2 \
8 & 4 & 1
\end{bmatrix}, b =
\begin{bmatrix}
6 \
20 \
19
\end{bmatrix}
$$
Encontre a matriz aumentada e calcule a matriz triangular superior utilizando o Método da Eliminação de Gauss. Não utilize funções prontas para realizar o cálculo.
Entrada
Este exercício não contém entradas.
Saída
A matriz aumentada, conforme a saída de teste.
Entrada de Teste
Saída de Teste
3.0000  3.0000 -1.0000  6.0000
0.0000  4.0000  2.6667  16.0000
0.0000  0.0000  6.3333  19.0000
Can you write Python code for it?

def  gauss(A, b):
    G = np.append(A, b, 1).astype(float)
    n = len(G[:, 0])

    for i in range(n):
        if G[i,i] == 0:
            return -1
        for j in range(i + 1, n):
            G[j] -= G[i] * G[j,i]/G[i,i]

    return G

A = np.array([[3, 3, -1], [2, 6, 2], [8, 4, 1]])
b = np.array([[6, 20, 19]])
b = b.T

print(gauss(A,b))


I am working on the file "2017-1/Metodos/Listas/L04_DaniloBarrosMendes_120114780.ipynb".
The first task is:
Questão 03
Como mencionado no exercício anterior, o método de Gauss-Jordan também pode ser utilizado para encontrar a inversa de matrizes quadradas não-singulares.
Dado um inteiro positivo $N$, e todos os elementos de uma matriz quadrada $A$ de ordem $N$, calcule $A^{-1}$ utilizando o método de Gauss-Jordan. Não utilize funções prontas para realizar o cálculo.
Entrada
A primeira linha de entrada contém um inteiro positivo $N$. As próximas $N$ linhas contém $N$ inteiros cada, representando cada um dos elementos da matriz $A$.
Saída
Na saída devem ser impressas $N$ linhas com $N$ valores cada, representando cada um dos elementos da matriz inversa, conforme o caso de teste abaixo.
Entrada de Teste
3
3.0 3.0 -1.0
2.0 6.0 2.0
8.0 4.0 1.0
Saída de Teste
-0.0263 -0.0921  0.1579
 0.1842  0.1447 -0.1053
-0.5263  0.1579  0.1579
Can you write Python code for it?


I am working on the file "2017-1/Metodos/Listas/L04_DaniloBarrosMendes_120114780.ipynb".
The first task is:
Questão 04
O Método de Jacobi é um método utilizado para encontrar a solução de sistemas lineares de forma numérica, ou seja, por aproximações sucessivas. O método funiona da seguinte forma: dado um sistema linear $Ax = b$ e uma aproximação inicial para os valores de $x$, a próxima aproximação para os valores de $x$ são calculadas através da equação:
$$
x^{(novo)}i = \frac{b_i \sum{j \neq i}{a_{ij}x_j^{(antigo)}}}{a_{ii}}
$$
Dado um sistema linear, utilize o Método de Jacobi para encontrar as 10 (dez) primeiras aproximações para os elementos do vetor de incógnitas ($x$). Assuma que todos os elementos de $x$ são iguais a $1$ na aproximação inicial. Não utilize funções prontas para realizar o cálculo.
Entrada
A primeira linha da entrada consiste de um valor inteiro $N$ ($2 \leq N \leq 10$), indicando a ordem do sistema linear. A segunda linha consiste de $N$ valores de ponto flutuante, indicando cada um dos termos independentes do sistema linear. As próximas $N$ linhas consistem de $N$ valores de ponto flutuante cada, representando cada um dos elementos da matriz de coeficientes.
Saída
A saída deve conter 10 linhas, a linha $i$ ($1 \leq i \leq N$) deve conter $N$ valores de ponto flutuante com 6 casas decimais de precisão, representando cada um dos elementos da i-ésima aproximação do vetor $x$.
Entrada de Teste
2
8.0 10.0
2.0 2.0
1.0 4.0
Saída de Teste
1.000000  1.000000
3.000000  2.250000
1.750000  1.750000
2.250000  2.062500
1.937500  1.937500
2.062500  2.015625
1.984375  1.984375
2.015625  2.003906
1.996094  1.996094
2.003906  2.000977
Can you write Python code for it?


I am working on the file "2017-1/Metodos/Listas/L04_DaniloBarrosMendes_120114780.ipynb".
The first task is:
Questão 06
O Método dos Trapézios é um método de integração numérica que aproxima a área sob a curva por uma soma de trapézios.
Seja $f(x)$ a função:
$$ f(x) = xsin(x) + x $$
Dados os limites $a$ e $b$ e o tamanho da divisão $h$, calcule $\int_a^b{f(x)dx}$ utilizando o Método dos Trapézios.
Entrada
A entrada consiste de uma única linha contendo três valores de ponto flutuante: $a$, $b$ e $h$, respectivamente.
Saída
A saída deve conter um único valor de ponto flutuante com 4 casas decimais de precisão, representando o valor da integral encontrada.
Entrada de Teste
0.0 10.0 1.0
Saída de Teste
57.0878
Can you write Python code for it?

def f(x):
    return x * np.sin(x) + x

def trapez(f, a, b, h):
    N = int((b - a) / h)
    I = 0
    for i in range(N):
        xi = a + i * h
        xf = xi + h
        I += h * (f(xi) + f(xf)) / 2
    return I

a = float(input())
b = float(input())
h = float(input())

print('%.4f' % trapez(f, a, b, h))


I am working on the file "2017-1/Metodos/Listas/L04_DaniloBarrosMendes_120114780.ipynb".
The first task is:
Questão 07
A Regra de Simpson é uma regra de integração numérica que divide a função em pequenos intervalos e aproxima a função em cada intervalo por uma parábola escolhendo 3 pontos de referência no início, meio e fim do intervalo.
Na regra de Simpson, a integral sobre cada segmento é aproximada pela seguinte fórmula
$$S_i = \frac{h}{6} \left[f(x_i) + 4 f(x_i + h/2) + f(x_i + h)\right]$$
Seja $f(x)$ a função:
$$ f(x) = \sin{x}\sqrt{x} $$
Dados os limites $a$ e $b$ e o tamanho da divisão $h$, calcule $\int_a^b{f(x)dx}$ utilizando a Regra de Simpson.
Entrada
A entrada consiste de uma única linha contendo três valores de ponto flutuante: $a$, $b$ e $h$, respectivamente.
Saída
A saída deve conter um único valor de ponto flutuante com 4 casas decimais de precisão, representando o valor da integral encontrada.
Entrada de Teste
0.0 10.0 1.0
Saída de Teste
3.2048
Can you write Python code for it?

def f(x):
    return np.sin(x) * np.sqrt(x)

def simpson(f, a, b, h):
    N = int((b - a) / h)
    I = 0
    for i in range(N):
        x0 = a + i * h
        x1 = x0 + h / 2
        x2 = x0 + h
        I += (h/6) * (f(x0) + 4 * f(x1) + f(x2))
    return I

a = float(input())
b = float(input())
h = float(input())

print('%.4f' % simpson(f, a, b, h))


I am working on the file "2017-1/Metodos/Listas/L04_DaniloBarrosMendes_120114780.ipynb".
The first task is:
Questão 08
A Regra da Quadratura Gaussiana é uma regra de integração numérica que divide o intervalo de intergração em um conjunto de pequenos intervalos e aproxima a função em cada um desses intervalos por aproximações polinomiais, escolhendo sempre os pontos no intervalo de integração que fornecerão a melhor aproximação possível.
Seja $f(x)$ a função:
$$ f(x) = ln(x + 1)sin(x) $$
Dados os limites $a$ e $b$ e o tamanho da divisão $h$, calcule $\int_a^b{f(x)dx}$ utilizando a Regra da Quadratura Gaussiana com n = 3.
Entrada
A entrada consiste de uma única linha contendo três valores de ponto flutuante: $a$, $b$ e $h$, respectivamente.
Saída
A saída deve conter um único valor de ponto flutuante com 4 casas decimais de precisão, representando o valor da integral encontrada.
Entrada de Teste
0.0 10.0 1.0
Saída de Teste
2.3133
Can you write Python code for it?


I am working on the file "2017-10_class_imbalance/Class Imbalance.ipynb".
The first task is:
Imbalance is Common, and Accuracy is NOT the right metric

In prior example 9% of classes were "virginica" and 91% were "not virginica"
If we predicted all classes were "not virginica" we would have 91% accuracy!
Your classifier may be doing this!

... but the accuracy... it is like a paradox
Can you collect more data?
You might think it’s silly, but collecting more data is almost always overlooked.
Can you collect more data? Take a second and think about whether you are able to gather more data on your problem.
A larger dataset might expose a different and perhaps more balanced perspective on the classes.
More examples of minor classes may be useful later when we look at resampling your dataset.
Try Changing Your Performance Metric
As mentioned prior - accuracy is a paradox and no longer the appropriate measurement.

F1 Score (or F-score): A weighted average of precision and recall. - probably the single best measurement
Precision: A measure of a classifiers exactness.
Recall: A measure of a classifiers completeness

Confusion Matrix -- always check the confusion matrix!

Also check out these metrics

Kappa (or Cohen’s kappa): Classification accuracy normalized by the imbalance of the classes in the data.
ROC Curves: either traditional OR precision/recall ROC
Can you write Python code for it?

X_train, X_dev, y_train, y_dev = train_test_split(df.drop('virginica',axis=1), df.virginica, test_size=0.3,random_state=0)
print("virginica in train set = ", str(y_train.sum()))
print("virginica in dev set = ", str(y_dev.sum()))
print()

logistic = LogisticRegression()
logistic.fit(X_train, y_train)

#Get predicted classes
y_train_pred = logistic.predict_proba(X_train)[:,1]
y_dev_pred = logistic.predict_proba(X_dev)[:,1]

evaluation(y_train,y_train_pred,'training set')
print()
evaluation(y_dev,y_dev_pred,'testing set')


I am working on the file "2017-10_class_imbalance/Class Imbalance.ipynb".
The first task is:
Try Different Algorithms

Some algorithms are less suseptible to class imbalances such as tree based methods -- careful not to overfit
Can you write Python code for it?

#random forest
from sklearn.ensemble import RandomForestClassifier
rfc = RandomForestClassifier()
rfc.fit(X_train, y_train)

#Get predicted classes
y_train_pred = rfc.predict_proba(X_train)[:,1]
y_dev_pred = rfc.predict_proba(X_dev)[:,1]

evaluation(y_train,y_train_pred,'training set')
print()
evaluation(y_dev,y_dev_pred,'testing set')


I am working on the file "2017-10_class_imbalance/Class Imbalance.ipynb".
The first task is:
Try Penalized Models

You can build in a penalty proportional to the imbalance in the cost function
In NN or other algorithms you can define explicitly
In sklearn you can use the weighted function
In this case it will be similar to getting one TP wrong is equal to getting ~9 TN wrong
Can you write Python code for it?

X_train, X_dev, y_train, y_dev = train_test_split(df.drop('virginica',axis=1), df.virginica, test_size=0.3,random_state=0)
print("virginica in train set = ", str(y_train.sum()))
print("virginica in dev set = ", str(y_dev.sum()))
print()

#logistic regression has class_weight - to penalize the cost function to be balanced
logistic = LogisticRegression(class_weight='balanced')
logistic.fit(X_train, y_train)

#Get predicted classes
y_train_pred = logistic.predict_proba(X_train)[:,1]
y_dev_pred = logistic.predict_proba(X_dev)[:,1]

evaluation(y_train,y_train_pred,'training set')
print()
evaluation(y_dev,y_dev_pred,'testing set')


I am working on the file "2017-10_class_imbalance/Class Imbalance.ipynb".
The first task is:
Try Resampling Your Dataset
Risk if undersample

Risk if oversample

Some Rules of Thumb

Consider testing under-sampling when you have an a lot data (tens- or hundreds of thousands of instances or more)
Consider testing over-sampling when you don’t have a lot of data (tens of thousands of records or less)
Consider testing random and non-random (e.g. stratified) sampling schemes.
Consider testing different resampled ratios (e.g. you don’t have to target a 1:1 ratio in a binary classification problem, try other ratios)
Can you write Python code for it?

#undersample
virginica = df[df.virginica == 1].copy()
not_virginica = df[df.virginica == 0 ].sample(frac=0.5).copy()
df_undersample = pd.concat([virginica,not_virginica])

X_train, X_dev, y_train, y_dev = train_test_split(df_undersample.drop('virginica',axis=1), df_undersample.virginica, test_size=0.3,random_state=0)
print("virginica in train set = ", str(y_train.sum()))
print("virginica in dev set = ", str(y_dev.sum()))

logistic = LogisticRegression()
logistic.fit(X_train, y_train)

#Get predicted classes
y_train_pred = logistic.predict_proba(X_train)[:,1]
y_dev_pred = logistic.predict_proba(X_dev)[:,1]

evaluation(y_train,y_train_pred,'training set')
print()
evaluation(y_dev,y_dev_pred,'testing set')


I am working on the file "2017-12-04-DIANA-HEP/notebooks/bqplot.ipynb".
The first task is:
Every component of the figure is an independent widget
Can you write Python code for it?

xs = bq.LinearScale()
ys = bq.LinearScale()
x = np.arange(100)
y = np.cumsum(np.random.randn(2, 100), axis=1) #two random walks

line = bq.Lines(x=x, y=y, scales={'x': xs, 'y': ys}, colors=['red', 'green'])
xax = bq.Axis(scale=xs, label='x', grid_lines='solid')
yax = bq.Axis(scale=ys, orientation='vertical', tick_format='0.2f', label='y', grid_lines='solid')

fig = bq.Figure(marks=[line], axes=[xax, yax], animation_duration=1000)
display(fig)

# update data of the line mark
line.y = np.cumsum(np.random.randn(2, 100), axis=1)

xs = bq.LinearScale()
ys = bq.LinearScale()
x, y = np.random.rand(2, 20)
scatt = bq.Scatter(x=x, y=y, scales={'x': xs, 'y': ys}, default_colors=['blue'])
xax = bq.Axis(scale=xs, label='x', grid_lines='solid')
yax = bq.Axis(scale=ys, orientation='vertical', tick_format='0.2f', label='y', grid_lines='solid')

fig = bq.Figure(marks=[scatt], axes=[xax, yax], animation_duration=1000)
display(fig)

#data updates
scatt.x = np.random.rand(20) * 10
scatt.y = np.random.rand(20)