I am working on the file "01-dataset-management.ipynb".
The first task is:
Get the dataset resource
The dataset resource is retrieved by display name. Because multiple datasets can have the same display name, we retrieve the most recent updated one.
Can you write Python code for it?

dataset = vertex_ai.TabularDataset.list(
    filter=f"display_name={DATASET_DISPLAY_NAME}",
    order_by="update_time")[-1]

print("Dataset resource name:", dataset.resource_name)
print("Dataset BigQuery source:", dataset.gca_resource.metadata['inputConfig']['bigquerySource']['uri'])


I am working on the file "01-dataset-management.ipynb".
The first task is:
Setup Google Cloud project
Can you write Python code for it?

PROJECT = '[your-project-id]' # Change to your project id.
REGION = 'us-central1' # Change to your region.

if PROJECT == "" or PROJECT is None or PROJECT == "[your-project-id]":
    # Get your GCP project id from gcloud
    shell_output = !gcloud config list --format 'value(core.project)' 2>/dev/null
    PROJECT = shell_output[0]

print("Project ID:", PROJECT)
print("Region:", REGION)


I am working on the file "01-dataset-preprocessing.ipynb".
The first task is:
3. CSV Preprocessing Pipeline Execution
The CSV Criteo data preprocessing pipeline performs the following steps.

Read CSV files from Cloud Storage.
Convert the CSV files to parquet format and write it Cloud Storage.
Fit a pre-defined NVTabular workflow to the training data split to calculate transformation statistics.
Transform the training and validation data splits using the fitted workflow.
Output transformed parquet files to Cloud Storage.

<img src="./images/preprocessing_pipeline_csv.png" alt="Pipeline"/>
Converting CSV files to Parquet with NVTabular
The Criteo dataset is provided in TSV format, but the recommended data format to run the NVTabular preprocessing task and get the best possible performance is Parquet; a compressed, column-oriented file structure format. While NVTabular also supports reading from CSV files, reading
Parquet files can be 2x faster than reading CSV files.
To convert the Criteo CSV data to Parquet, the following steps are performed:

Create a nvt.Dataset object the CSV data using the create_csv_dataset method in src/preprocessing/task.py.
Convert the CSV data to Parquet, and write it to Cloud Storahe using the convert_csv_to_parquet method in src/preprocessing/task.py.

The pipeline uses the convert_csv_to_parquet_op component, which is implemented in src/pipelines/components.py which submits a Vertex AI training job to convert the files.
Create pipeline parameters
NVTabular provides an option to shuffle the dataset before storing to disk.
The uniformly shuffled dataset enables the data loader to read in contiguous chunks of data that are already randomized across the entire dataset.
NVTabular provides the option to control the number of chunks that are combined into a batch, allowing the end user flexibility when trading off between performance and true randomization.
This mechanism is critical when dealing with datasets that exceed CPU memory and per-epoch shuffling is desired during training.
Full shuffling of such a dataset can exceed training time for the epoch by...
Can you write Python code for it?

# List of path(s) to criteo file(s) or folder(s) in GCS.
# Training files
TRAIN_PATHS = ['gs://<PATH TO CSV FILES>'] # Change to the GCS path where CSV files are located
# Validation files
VALID_PATHS = ['gs://<PATH TO CSV FILES>'] # Change to the GCS path where CSV files are located

num_output_files_train = 24 # Number of output files after converting CSV to Parquet
num_output_files_valid = 1 # Number of output files after converting CSV to Parquet

csv_parameter_values = {
    'train_paths': json.dumps(TRAIN_PATHS),
    'valid_paths': json.dumps(VALID_PATHS),
    'num_output_files_train': num_output_files_train,
    'num_output_files_valid': num_output_files_valid,
    'shuffle': json.dumps(None) # select PER_PARTITION, PER_WORKER, FULL, or None.
}


I am working on the file "01-dataset-preprocessing.ipynb".
The first task is:
Build Container Docker Image
The following command will build the Docker container image to the NVTabular preprocessing steps of the pipeline and push it to the Google Container Registry.
Note that building the Docker container image take up to 20 minutes.
Can you write Python code for it?

FILE_LOCATION = './src'
! gcloud builds submit --config src/cloudbuild.yaml --substitutions _DOCKERNAME=$DOCKERNAME,_IMAGE_URI=$IMAGE_URI,_FILE_LOCATION=$FILE_LOCATION --timeout=2h --machine-type=e2-highcpu-8


I am working on the file "01-dataset-preprocessing.ipynb".
The first task is:
Change the following variables according to your definitions.
Can you write Python code for it?

# Project definitions
PROJECT_ID = '<YOUR PROJECT ID>' # Change to your project ID.
REGION = '<LOCATION OF RESOURCES>' # Change to your region.

# Service Account address
VERTEX_SA = f'vertex-sa@{PROJECT_ID}.iam.gserviceaccount.com' # Change to your service account with Vertex AI Admin permitions.

# Bucket definitions
BUCKET = '<YOUR BUCKET NAME>' # Change to your bucket. All the files will be stored here.


I am working on the file "01-dataset-preprocessing.ipynb".
The first task is:
Compile KFP pipeline
Can you write Python code for it?

from src.pipelines.preprocessing_pipelines import preprocessing_csv

csv_compiled_pipeline_path = f'{PREPROCESS_CSV_PIPELINE_NAME}.json'
compiler.Compiler().compile(
       pipeline_func=preprocessing_csv,
       package_path=csv_compiled_pipeline_path
)


I am working on the file "01-dataset-preprocessing.ipynb".
The first task is:
Initialize Vertex SDK client
Can you write Python code for it?

# Initialize Vertex AI API
vertex_ai.init(
    project=PROJECT_ID,
    location=REGION,
    staging_bucket=os.path.join(WORKSPACE, 'stg')
)


I am working on the file "01-grundlagen/pandas.ipynb".
The first task is:
.iloc()
Can you write Python code for it?

# .iloc slicing at specific location - BY POSITION in the table
# Recall:
# dfg[a:b] by rows
# dfg[[col]] or df[[col1, col2]] by columns
# df.loc[a:b, x:y], by index and column values + location
# df.iloc[3:5,0:2], numeric position in table

dfg.iloc[1:4,3:5] # 2nd to 4th row, 4th to 5th column


I am working on the file "01-grundlagen/pandas.ipynb".
The first task is:
Attributes & general statistics of a Pandas DataFrame
Can you write Python code for it?

dfg.shape # 251 business days last year

dfg.columns

dfg.size

# Generelle statistische Daten mit describe()

dfg.describe()

# Boolean indexing
dfg['Open'][dfg['Open']>1130] # check what dates the opening

# Check where Open, High, Low and Close where greater than 1130
dfg[dfg>1000].drop('Volume',axis=1)

# If you want the values in an np array
dfg.values


I am working on the file "01-grundlagen/pandas.ipynb".
The first task is:
More Basic Statistics
Can you write Python code for it?

# We can change the index sorting
dfg.sort_index(axis=0, ascending=True).head() # starts a year ago

# sort by value
dfg.sort_values(by='Open')[0:10]


I am working on the file "01-grundlagen/pandas.ipynb".
The first task is:
More Statistics and Operations
Can you write Python code for it?

# mean by column, also try var() for variance
dfg.mean()

# Verainz für jede Spalte
dfg.var()

dfg[0:5].mean(axis = 1) # Mittelwert der Zeilen für die ersten 5 Zeilen


I am working on the file "01-grundlagen/pandas.ipynb".
The first task is:
Pandas Series Objekt
Wie einnp.array, aber es können Datentypen kombiniert werden und eine Series hat einen eigenen Index
Anmerkung: Jede Spalte in einem Dataframe ist eine Series
Can you write Python code for it?

print(df1[['L','A']])

print(type(df1['L']))

df1

# Spalten umbenennen
df1 = df1.rename(columns = {'L':'Umbenannt'})
df1

# Löschen von Spalten
del df1['C']
df1

# oder drop Spalten
df1.drop('A',axis=1,inplace=True) # does not change df1 if we don't set inplace=True

df1

df1

# oder drop Zeilen
df1.drop(0)

# Beispiel: Anzeige einer Spalte
df1['B']

# Anzeigen verschiedener Spalten
df1[['B','Umbenannt']]


I am working on the file "01-grundlagen/pandas.ipynb".
The first task is:
PlotCorrelation
Load several stocks
Can you write Python code for it?

# Reload
dfg = pd.read_csv('data/googl.csv').drop('Unnamed: 0',axis=1) # Google stock data
dfa = pd.read_csv('data/apple.csv').drop('Unnamed: 0',axis=1) # Apple stock data
dfm = pd.read_csv('data/microsoft.csv').drop('Unnamed: 0',axis=1) # Google stock data
dfn = pd.read_csv('data/nike.csv').drop('Unnamed: 0',axis=1) # Apple stock data
dfb = pd.read_csv('data/boeing.csv').drop('Unnamed: 0',axis=1) # Apple stock data

dfb.head()

# Rename columns
dfg = dfg.rename(columns = {'Close':'GOOG'})
#print (dfg.head())

dfa = dfa.rename(columns = {'Close':'AAPL'})
#print (dfa.head())

dfm = dfm.rename(columns = {'Close':'MSFT'})
#print (dfm.head())

dfn = dfn.rename(columns = {'Close':'NKE'})
#print (dfn.head())

dfb = dfb.rename(columns = {'Close':'BA'})

dfb.head(2)

# Lets merge some tables
# They will all merge on the common column Date

df = dfg[['Date','GOOG']].merge(dfa[['Date','AAPL']])
df = df.merge(dfm[['Date','MSFT']])
df = df.merge(dfn[['Date','NKE']])
df = df.merge(dfb[['Date','BA']])

df.head()

df['Date'] = pd.to_datetime(df['Date'])
df = df.set_index('Date')
df.head()

df.plot()

df['2017'][['NKE','BA']].plot()

# show a correlation matrix (pearson)
crl = df.corr()
crl

crl.sort_values(by='GOOG',ascending=False)

s = crl.unstack()
so = s.sort_values(ascending=False)
so[so<1]

df.mean()

sim=df-df.mean()
sim.tail()

sim[['MSFT','BA']].plot()


I am working on the file "01-grundlagen/pandas.ipynb".
The first task is:
Slicing
[10 Minutes]
In the 10 min Pandas Guide, you will see many ways to view, slice a dataframe

view/slice by rows, eg df[1:3], etc.

view by index location, see df.iloc (iloc)

view by ranges of labels, ie index label 2 to 5, or dates feb 3 to feb 25, see df.loc (loc)

view a single row by the index df.xs (xs) or df.ix (ix)

filtering rows that have certain conditions

add column

add row

How to change the index

and more...
Can you write Python code for it?

print (df1[0:2])  # ok

df1

df1.iloc[1,1]

df1


I am working on the file "01-grundlagen/pandas.ipynb".
The first task is:
Teil 1
Einfache Erzeugung und Veränderung von Panda Objekten
Key Points: Pandas has two / three main data types:
* Series (similar to numpy arrays, but with index)
* DataFrames (table or spreadsheet with Series in the columns)
* Panels (3D version of DataFrame, not as common)
Es ist einfach einen DataFrame zu erzeugen
Wir verwenden pd.DataFrame(**inputs**) und können jeden Datentyp als Argument verwenden
Function: pd.DataFrame(data=None, index=None, columns=None, dtype=None, copy=False)
Input data can be a numpy ndarray (structured or homogeneous), dict, or DataFrame.
Dict can contain Series, arrays, constants, or list-like objects as the values.
Can you write Python code for it?

# Wir probieren es mit einem array aus
np.random.seed(0) # setze seed für Nachvollziebarkeit

a1 = np.array(np.random.randn(3))
a2 = np.array(np.random.randn(3))
a3 = np.array(np.random.randn(3))

print (a1)
print (a2)
print (a3)

# Wir erzeugen einen ersten Dataframe mit einem np.array - der Dataframe hat nur eine Spalte
df0 = pd.DataFrame(a1)
print(type(df0))
df0

# DataFrame aus einer Liste der np.arrays

df0 = pd.DataFrame([a1, a2, a3])
df0

# beachte, dass keine Spalten Labels vorhanden sind, nur ganzzahlige Werte,
# und der Index wird automatisch gesetzt

# DataFrame von einem 2D np.array
# 9 Normalverteilte Zufallszahlen np.random.randn()
# mit reshape als 3x3 Matrix
ax = np.random.randn(9).reshape(3,3)
ax

# DataFrame von einem 2D np.array
# 24 Normalverteilte Zufallszahlen np.random.randn()
# mit reshape als 3x8 Matrix
at = np.random.randn(24).reshape(3,8)
at

# Setzen der Spalten Labels in ax
df0 = pd.DataFrame(ax,columns=['rand_normal_1','Random Again','Third'],
                   index=[100,200,99]) # wir können Spalten Bezeichnungen und einen Index zuweisen, dies muss aber in der Größe übereinstimmen
#Ausgabe von df0
df0

# Setzen der Spalten Labels in ax
df_0 = pd.DataFrame(at,columns=['rand_1','rand_2','rand_3', 'rand_4', 'rand_5','rand_6', 'rand_7', 'rand_8'],
                   index=[0,1,2]) # wir können Spalten Bezeichnungen und einen Index zuweisen, dies muss aber in der Größe übereinstimmen
# Ausgabe von df_0
df_0

# DataFrame aus einem Dictionary
# arrays a1 und a2
dict1 = {'A':a1, 'B':a2}
df1 = pd.DataFrame(dict1)
#Ausgabe df1
df1

# Man kann leicht eine weitere Spalte hinzufügen (so wie man Werte in ein dictionary hinzufügt)
df1['C']=a3
# Ausgabe df1
df1

# Wir können eine zusätzliche Spalte mit Text und Zahlen hinzufügen
df1['L'] = ["Etwas", 3.2, "Worte"]
df1


I am working on the file "01-grundlagen/pandas.ipynb".
The first task is:
Teil 2
Beispiel Finance: Große Data Frames
Wir laden Daten im CSV Format.
See https://www.quantshare.com/sa-43-10-ways-to-download-historical-stock-quotes-data-for-free
Can you write Python code for it?

!ls data/

# Wir können Daten aus dem Web herunterladen. Hierfür nutzen wir pd.read_csv
# Ein CSV file ist ein comma seperated file

base_url = 'https://google.com/finance?output=csv&q='

dfg = pd.read_csv('data/googl.csv').drop('Unnamed: 0',axis=1) # Google Aktien
dfa = pd.read_csv('data/apple.csv').drop('Unnamed: 0',axis=1) #Apple Aktien

dfg

dfg.head() # zeige die ersten fünf Werte an

dfg.tail(3) # zeige die letzten 3 Werte an

dfg.columns # returns columns, can be used to loop over

dfg.index # return


I am working on the file "01-grundlagen/pandas.ipynb".
The first task is:
Umwandeln des Index in pandas datetime Objekt
Can you write Python code for it?

dfg['Date'][0]

type(dfg['Date'][0])

# Date wird als String gelistet, sollte deshalb in datetime umgewandelt werden
dfg.index = pd.to_datetime(dfg['Date']) # setzen des neuen index

dfg.drop(['Date'],axis=1,inplace=True)

dfg.head()

print(type(dfg.index[0]))
dfg.index[0]

dfg.index

dfg['2017-08':'2017-06']


I am working on the file "01-hello_world/00-history.ipynb".
The first task is:

Compiling (1951)
Can you write Python code for it?

!gcc hello.c -o hello

!file hello

!wc -c < hello


I am working on the file "01-hello_world/00-history.ipynb".
The first task is:

Compiling Java code (for the Java virtual machine) (1991)
Can you write Python code for it?

!cat HelloWorld.java

!javac HelloWorld.java

!javap -c HelloWorld.class

!java HelloWorld


I am working on the file "01-hello_world/01-intro.ipynb".
The first task is:
C++
Can you write Python code for it?

# %load quicksort.cpp
# http://codereview.stackexchange.com/questions/77782/quick-sort-implementation

#include <iostream>
using namespace std;

void print(int *a, int n)
{
    int i=0;
    while(i<n){
        cout<<a[i]<<",";
        i++;
    }
}

void swap(int i,int j, int *a){
    int temp = a[i];
    a[i] = a[j];
    a[j] = temp;
}

void quicksort(int *arr, int left, int right){
    int min = (left+right)/2;
    cout<<"QS:"<<left<<","<<right<<"\n";

    int i = left;
    int j = right;
    int pivot = arr[min];

    while(left<j || i<right)
    {
        while(arr[i]<pivot)
        i++;
        while(arr[j]>pivot)
        j--;

        if(i<=j){
            swap(i,j,arr);
            i++;
            j--;
        }
        else{
            if(left<j)
                quicksort(arr, left, j);
            if(i<right)
                quicksort(arr,i,right);
            return;
        }
    }
}

int main() {
    int arr[8] = {110, 5, 10,3 ,22, 100, 1, 23};
    quicksort(arr, 0, (sizeof(arr)/sizeof(arr[0]))-1);
    print(arr, (sizeof(arr)/sizeof(arr[0])));
    return 0;
}


I am working on the file "01-hello_world/01-intro.ipynb".
The first task is:
Python
Can you write Python code for it?

# %load quicksort.py
# http://cs231n.github.io/python-numpy-tutorial/#python

def quicksort(arr):
    if len(arr) <= 1:
        return arr
    pivot = arr[len(arr) // 2]
    left = [x for x in arr if x < pivot]
    middle = [x for x in arr if x == pivot]
    right = [x for x in arr if x > pivot]
    return quicksort(left) + middle + quicksort(right)

print(quicksort([3,6,8,10,1,2,1]))


I am working on the file "01-hello_world/01-intro.ipynb".
The first task is:
R
Can you write Python code for it?

# %load quicksort.r
# http://www.jason-french.com/blog/2014/07/26/recursion-in-r/

#!/usr/bin/env Rscript
# Author:  Jason A. French

quickSort <- function(vect) {
  # Args:
  #  vect: Numeric Vector

  # Stop if vector has length of 1
  if (length(vect) <= 1) {
      return(vect)
  }
  # Pick an element from the vector
  element <- vect[1]
  partition <- vect[-1]
  # Reorder vector so that integers less than element
  # come before, and all integers greater come after.
  v1 <- partition[partition < element]
  v2 <- partition[partition >= element]
  # Recursively apply steps to smaller vectors.
  v1 <- quickSort(v1)
  v2 <- quickSort(v2)
  return(c(v1, element, v2))
}

quickSort(c(4, 65, 2, -31, 0, 99, 83, 782, 1))


I am working on the file "01-hello_world/02-installation.ipynb".
The first task is:
Automatically activating and deactivating VEs
pyenv-virtualenv (running) automatically activate and deactivate the VE when you arrive to and leave a Python project that previously has been configured with:
pyenv local <VE_name>
Example
We configure a Python project to automatically activate a VE when we access to it.
Can you write Python code for it?

%%bash

echo "Showing versions of Python"
pyenv version

echo "Ensure the test VE does not exit"
pyenv uninstall -f my_python_project__system

echo "Create the test VE"
pyenv virtualenv 3.8.5 my_python_project__system

echo "Create and go into the test Python project"
rm -rf /tmp/my_python_project
mkdir /tmp/my_python_project
cd /tmp/my_python_project

echo "Hook the VE and the project"
pyenv local my_python_project__system

echo "Inside of the project the Python's version is ..."
python --version

echo "Outside of the project the Python's version is ..."
cd
python --version


I am working on the file "01-hello_world/02-installation.ipynb".
The first task is:
Creating a VE
pyenv virtualenv <Python_version> <VE_name>
VEs are stored in ~/.pyenv/versions/<VE_name>/.
Listing VEs
pyenv virtualenvs
Activating a VE
pyenv activate <VE_name>`
<!-- Now, `(<VE_name>)` should preceed your current prompt. -->

To activate a VE you must run virtualenv-initin your shell. You can run it directly from the shell with:
pyenv virtualenv-init
or you can include:
eval "$(pyenv virtualenv-init -)"
into the .bashrc file.
Deactivating a VE
pyenv deactivate
Delete a VE
pyenv uninstall <VE_name>
Example
Now, we create, activate, desativate, and remove a VE.
Can you write Python code for it?

%%bash

pyenv install 3.8.5

%%bash

pyenv virtualenv 3.8.5 socket_programming__385

%%bash

pyenv virtualenvs

%%bash

eval "$(pyenv init -)" # <- This should be in .bashrc

pyenv activate socket_programming__385
pyenv virtualenvs
pyenv deactivate

%%bash

pyenv virtualenvs


I am working on the file "01-hello_world/03-hello_world.ipynb".
The first task is:

2. Interacting with Jupyter Notebook
This interface (what you are reading now) is know as Jupyter Notebook, an interactive document, which is a mix of Markdown and Python code executed by IPython:
Can you write Python code for it?

print("Hello world!") # Modify me and push <SHIFT> + <RETURN>


I am working on the file "01-hello_world/03-hello_world.ipynb".
The first task is:

3. Interacting with the Python interpreter
Run python in a shell and type: print("Hello world!") <enter> quit().
```
$ python
Python 2.7.10 (default, Oct 23 2015, 19:19:21)
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

print("Hello world!")
Hello world!
quit()
$
```

Alternatively, instead of python we can use ipython, which provides dynamic object introspection, command completion, access to the system shell, etc.
```
$ ipython
Python 3.5.1rc1 (v3.5.1rc1:948ef16a6951, Nov 22 2015, 11:29:13)
Type "copyright", "credits" or "license" for more information.
IPython 5.1.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.
In [1]: print("Hello world!")
Hello world!
In [2]: help(print)
Help on built-in function print in module builtins:
print(...)
    print(value, ..., sep=' ', end='\n', file=sys.stdout, flush=False)
Prints the values to a stream, or to sys.stdout by default.
Optional keyword arguments:
file:  a file-like object (stream); defaults to the current sys.stdout.
sep:   string inserted between values, default a space.
end:   string appended after the last value, default a newline.
flush: whether to forcibly flush the stream.

(type: <q> to exit)
In [3]: quit() # <ctrl> + <d> also works in Unixes
$
```
Interpreted?
Python is an interpreted programming language. When we run a Python program, we are executing the translation to bytecode of each Python statement of our program over the Python Virtual Machine (PVM). The .pyc files that appear after running a collection of modules as a script for the first time, contains the bytecode of such modules. This is used by Python to speed up their future executions.
Can you write Python code for it?

def hello():
    print('Hello world!')
import dis

dis.dis(hello)


I am working on the file "01-hello_world/03-hello_world.ipynb".
The first task is:

4. Running Python programs (modules) as scripts
Can you write Python code for it?

!cat hello_world.py

# Check the code (optional)
!pyflakes3 hello_world.py

!./hello_world.py # Specific of Unix

!python hello_world.py

%run hello_world.py # Specific of Ipython


I am working on the file "01-introduction.ipynb".
The first task is:
PCA
TODO
Can you write Python code for it?

from sklearn.datasets import fetch_olivetti_faces
faces = fetch_olivetti_faces()

plt.subplots(2, 3)
for i in range(6):
    plt.subplot(2, 3, i + 1)
    plt.axis('off')
    plt.imshow(faces.images[10 * i].reshape(64, 64), cmap=plt.cm.gray)

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

faces.data = StandardScaler().fit_transform(faces.data)
pca = PCA().fit(faces.data)

plt.subplots(2, 3)
for i in range(6):
    plt.subplot(2, 3, i + 1)
    plt.axis('off')
    plt.imshow(pca.components_[i].reshape(64, 64), cmap=plt.cm.gray_r)


I am working on the file "01-introduction.ipynb".
The first task is:
The model is ready and can be used to predict on real data

Classification
Prepare the data

Just like regression.
Can you write Python code for it?

from sklearn.datasets import fetch_mldata
mnist = fetch_mldata('MNIST original')

X = pd.DataFrame(mnist.data)
y = pd.Series(mnist.target)

f, axes = plt.subplots(3, 4, figsize=(8, 8));
for i in range(3):
    for j in range(4):
        axes[i, j].axis('off')
        if i == 2 and j >= 2:
            continue
        num = i * 4 + j
        axes[i, j].set_title(num)
        axes[i, j].matshow(X.values[y == num][0].reshape(28, 28))


I am working on the file "01-introduction.ipynb".
The first task is:
Train the model on the training set
Can you write Python code for it?

from sklearn.linear_model import LinearRegression
model = LinearRegression().fit(X_train, y_train)


from sklearn.linear_model import SGDClassifier
model = SGDClassifier().fit(X_train, y_train)


I am working on the file "01-introduction.ipynb".
The first task is:
Train/Test Split

In order to reguralize the model, we split the dataset into training and test sets.
Training and test sets are similar in shape except in size.
Usually training set is the bigger part of the dataset (e.g. 80%) and the rest is test set (e.g. 20%).
Can you write Python code for it?

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X.values.reshape(X.shape[0], 1),
                                                    y.values.reshape(y.shape[0], 1),
                                                    test_size=.3)

print('X_train.shape =', X_train.shape)
print('X_test.shape =', X_test.shape)
print('y_train.shape =', y_train.shape)
print('y_test.shape =', y_test.shape)


I am working on the file "01-introduction.ipynb".
The first task is:
Truely Classified Digits
Can you write Python code for it?

y_pred = model.predict(X_test)
f, axes = plt.subplots(1, 5, figsize=(12, 4))
samples = random.sample(list(y_test[y_test == y_pred].iteritems()), 5)
for i_zero, (i, p) in enumerate(samples):
    axes[i_zero].axis('off')
    axes[i_zero].matshow(X_test.loc[i].values.reshape(28, 28))
    axes[i_zero].set_title('pred = %d' % y_pred[i - 60000], color='green')


I am working on the file "01-introduction.ipynb".
The first task is:
We separate dataset into features and labels matrices.
Features could be vectors (data rows has only 1 feature), Matrices (having multiple features) or Tensors (each feature is itself another matrix or tensor)
Labels could be vectors, matrices or tensors too.
Usually features are matrices and we declare it by X and lables are vectors declared by y.
We declare the dataset size as the number of data items in it.
Can you write Python code for it?

X = pd.DataFrame(iris.data, columns=iris.feature_names)
y = pd.Series(iris.target)

print('X.shape =', X.shape)
print('y.shape =', y.shape)

X.head()

y.head()

plt.figure(figsize=(6, 6));
plt.scatter(X.values[:, 0], X.values[:, 1], c=y, cmap=plt.cm.rainbow);

# Correlation plots are a good way to understand the dataset
pd.plotting.scatter_matrix(X, figsize=(10, 10));


I am working on the file "01-introduction.ipynb".
The first task is:
What is Data?

A dataset consists of multiple data rows.
Each row describes an item with its features.
Think of features as properties of a sample. E.g. an apple has color, size, kind, ...
Can you write Python code for it?

# Load Iris dataset
from sklearn.datasets import load_iris
iris = load_iris()

iris.feature_names

print(iris.DESCR)


I am working on the file "01-introduction/Linear Programming.ipynb".
The first task is:
2. Setup the variables:
for now we will make them manually, but there are convenience methods for when you need to make millions at a time
Can you write Python code for it?

x = LpVariable('x',lowBound=0, upBound=10, cat='Continuous')
y = LpVariable('y',lowBound=0, upBound=10, cat='Continuous')
z = LpVariable('z',lowBound=0, upBound=10, cat='Continuous')


I am working on the file "01-introduction/Linear Programming.ipynb".
The first task is:
3. Setup the objective
Can you write Python code for it?


I am working on the file "01-introduction/Linear Programming.ipynb".
The first task is:
4. Setup the constraints
Can you write Python code for it?


I am working on the file "01-introduction/Linear Programming.ipynb".
The first task is:
5. stuff the objective and the constraint into the problem
To add constraints and objectives to the problem, we literally just add them to it
Can you write Python code for it?

#add the objective
prob+= objective

#add the constraints
prob+=constraint


I am working on the file "01-introduction/Linear Programming.ipynb".
The first task is:
7. Get the results
Can you write Python code for it?

#get a single variables value
print(x.varValue)

#or get all the variables
for v in prob.variables():
    print(v, v.varValue)


I am working on the file "01-introduction/Linear Programming.ipynb".
The first task is:
It is an LpAffineExpression. You can actually print LpAffineExpressions to see what you have programmed. Be careful with this on larger problems
Can you write Python code for it?


I am working on the file "01-introduction/Linear Programming.ipynb".
The first task is:
lets try a small problem we can easily solve by hand
$$minimize\: f(x,y,z)=5x+10y+6z \
subject\, to\
x+y+z \geq 20 \
0\leq x,y,z \leq 10
$$
In school we may have learned how to solve these types or problems by writing them in canonical form and throwing some linear algebra at them. PuLP is a library that removes this need, we can code our problem almost exactly as stated above in PuLP and it will do the hard work for us. What PuLP actually does is format the problem into a standard language that is used by many numerical solvers.
1. Setup the problem
Can you write Python code for it?


I am working on the file "01-introduction/Linear Programming.ipynb".
The first task is:
like the LpAffineExpression class, we can print the problem to see what PuLP has generated. This is very useful for small problems, but can print thousands of lines for large problems. Its always a good idea to start small.
Can you write Python code for it?


I am working on the file "01-introduction/Linear Programming.ipynb".
The first task is:
what does this create?
Can you write Python code for it?


I am working on the file "01-introduction/exercise.ipynb".
The first task is:
Here, the two scoes start having a big difference between their average values (tick lines). The train set is small so we are not using too many data to train the classifier. Increasing the fraction of the train size provide a larger number of points that improve the performance of the classifier, both on the training and test sets, since we gave it more statistics.
In particular, we see that the mean score (for both sets) as a function of the training set undergoes a vivid rise at the beginning, up to train size fraction of 0.4, when after stays almost constant. This means that increasing the train set size does not always improve significantly the accuracy of the results (on average) after a given point. Note that, however, that the standard deviation of the test set score increases, especially when we reach high values of the train set size. In this case, we are thus trying to have many data to fit the model but few to predict the accuracy in the test set. As a consequence, we are more proned to overfitting the data in the train set. The test set curve seems to show a maximum on 0.5, which means that a somehow optimal split is attained dividing in half the dataset. This is reasonable, as already commented in the part "Splitting the data between a training set and test."
Varying n_neighbors
Can you write Python code for it?

train_performance_nn = []
test_performance_nn = []

# We fix the ratio between train and test size at 0.5

range_nn = range(1,26,1)

for i in range(500):
    train_performance = []
    test_performance = []
    for nn in range_nn:
        X_train, X_test, y_train,y_test = train_test_split(X, y, train_size=0.5)
        kN_classifier = KNeighborsClassifier(n_neighbors=nn)
        kN_classifier.fit(X_train, y_train)
        # store the performance of the kN_classifier on the train set
        train_performance.append(kN_classifier.score(X_train, y_train))
        # store the performance of the kN_classifier on the test set
        test_performance.append(kN_classifier.score(X_test, y_test))
        #
    train_performance_nn.append(train_performance)
    test_performance_nn.append(test_performance)

train_performance_nn = np.array(train_performance_nn)
test_performance_nn = np.array(test_performance_nn)

fig = plt.figure()
ax = fig.add_subplot(111)

mean_train_performance_nn = np.mean(train_performance_nn, axis=0)
std_train_performance_nn = np.std(train_performance_nn, axis=0)
mean_test_performance_nn = np.mean(test_performance_nn, axis=0)
std_test_performance_nn = np.std(test_performance_nn, axis=0)

ax.plot(range_nn, mean_train_performance_nn, color='g', label='Train', lw=4)
ax.plot(range_nn, mean_train_performance_nn + std_train_performance_nn, range_nn, mean_train_performance_nn - std_train_performance_nn, color='g', alpha=0.5, ls="--")
ax.plot(range_nn, mean_test_performance_nn, color='r', label='Test', lw=4)
ax.plot(range_nn, mean_test_performance_nn + std_test_performance_nn, range_nn, mean_test_performance_nn - std_test_performance_nn, color='r', alpha=0.5, ls="--")

ax.set_xlabel("n_neighbors")
ax.set_ylabel("Accuracy")
plt.legend(loc='best')
# plt.plot(ks, np.array(accuracies_test).mean(axis=0), label='Test', c='r', lw=4)
# plt.plot(ks, np.array(accuracies_train).mean(axis=0), label='Train', c='b', lw=4)
# plt.xlabel('k or inverse model complexity')
# plt.ylabel('accuracy')
#...


I am working on the file "01-introduction/exercise.ipynb".
The first task is:
In the boxplots, we summarize the distribuzion of the scores for the training and test set. The upper and lower bound of the boxes mark the first and third quartile, while the vertical lines outside the boxes represent the 5th and 95 percentiles. Data represented as dots are named outliers that fall outside the range between 5th and 95th percentiles. Black horizontal line represent the means, while dotted grey lines the medians.
As we can see, the values of the means (and medians) are pretty far away the each others between the training set and test set. Moreover, the 95th percentile of the test set is just sligthly above the 5th percentile of the traning set. Overall, these mean that the scores on the training set are sistematically higher then the ones in the test set, given their distribution. Of course, some values of the scores on the test set are higher than the lowest ones on the training set, due to fluctuations in the score values.
Varying the splits between datasets (10-90, 20-80, 30-70, ...)
Can you write Python code for it?

train_performance_splits = []
test_performance_splits = []

range_ft = np.linspace(0.1,0.9,9)

for i in range(500):
    train_performance = []
    test_performance = []
    for ft in range_ft:
        # choose each time a different random number to randomly split the dataset into train and test set
        X_train, X_test, y_train,y_test = train_test_split(X, y, train_size=ft)
        kN_classifier = KNeighborsClassifier(n_neighbors=5)
        kN_classifier.fit(X_train, y_train)
        # store the performance of the kN_classifier on the train set
        train_performance.append(kN_classifier.score(X_train, y_train))
        # store the performance of the kN_classifier on the test set
        test_performance.append(kN_classifier.score(X_test, y_test))
        #
    train_performance_splits.append(train_performance)
    test_performance_splits.append(test_performance)

train_performance_splits = np.array(train_performance_splits)
test_performance_splits = np.array(test_performance_splits)

fig = plt.figure()
ax = fig.add_subplot(111)

mean_train_performance_splits = np.mean(train_performance_splits, axis=0)
std_train_performance_splits = np.std(train_performance_splits, axis=0)
mean_test_performance_splits = np.mean(test_performance_splits, axis=0)
std_test_performance_splits = np.std(test_performance_splits, axis=0)

ax.plot(range_ft, mean_train_performance_splits, color='g', label='Train', lw=4)
ax.plot(range_ft, mean_train_performance_splits + std_train_performance_splits, range_ft, mean_train_performance_splits - std_train_performance_splits, color='g', alpha=0.5, ls="--")
ax.plot(range_ft, mean_test_performance_splits, color='r', label='Test', lw=4)
ax.plot(range_ft, mean_test_performance_splits + std_test_performance_splits, range_ft, mean_test_performance_splits - std_test_performance_splits, color='r', alpha=0.5, ls="--")

ax.set_xlabel("Train set fraction")
ax.set_ylabel("Accuracy")
plt.legend(loc='best')
# plt.plot(ks, np.array(accuracies_test).mean(axis=0), label='Test', c='r', lw=4)
#...


I am working on the file "01-introduction/exercise.ipynb".
The first task is:
Question 1
In the lecture we used the nearest neighbour classifier to classify points from a toy dataset into either "red" or "blue" classes. We investigated how the performance changes as a function of model complexity and what this means for the performance of our classifier on unseen data. Instead of using a linear model as in the lecture, use a k-nearest neighbour model.

plot your dataset
split your dataset into a training and testing set. Comment on how you decided to split your data.
evaluate the performance of the classifier on your training dataset.
evaluate the performance of the classifier on your testing dataset.
repeat the above two steps for varying splits (10-90, 20-80, 30-70, ...) and comment
  on what you see. Is there a "best" way to split your data?
comment on why the two performance estimates agree or disagree.
plot the accuracy of the classifier as a function of n_neighbors.
comment on the similarities and differences between the performance on the testing and training dataset.
is a KNeighbor Classifier with 4 or 10 neighbors more complicated?
find the best setting of n_neighbors for this dataset.
why is this the best setting?

Use make_blobs(n_samples=400, centers=23, random_state=42) to create a simple dataset and use the KNeighborsClassifier classifier to answer the above questions.
Can you write Python code for it?

from sklearn.datasets import make_blobs
from sklearn.neighbors import KNeighborsClassifier

labels = ["b", "darkorange"]
X, y = make_blobs(n_samples=400, centers=23, random_state=42)
y = np.take(labels, (y < 10))


I am working on the file "01-introduction/exercise.ipynb".
The first task is:
Question 2
This is a regression problem. It mostly follows the setup of the classification problem so you should be able to reuse some of your work.

plot your dataset
fit a kNN regressor with varying number of n_neighbors and compare each regressors predictions to the location of the training and testing points.
plot the mean squared error of the classifier as a function of n_neighbors for both training and testing datasets.
comment on the similarities and differences between the performance on the testing and training dataset.
find the best setting of n_neighbors for this dataset.
why is this the best setting?
can you explain why the mean square error on the training dataset plateaus between ~n_neihgors=5 to 15 at the value that it does?

Use make_regression() to create the dataset and use KNeighborsRegressor to answer the above questions. Take a look at scikit-learn's metrics module to compute the mean squared error.
Can you write Python code for it?

def make_regression(n_samples=100, noise_level=0.8, random_state=2):
    rng = check_random_state(random_state)
    X = np.linspace(-2, 2, n_samples)
    y = 2 * X + np.sin(5 * X) + rng.randn(n_samples) * noise_level

    return X.reshape(-1, 1), y

# Your solution


I am working on the file "01-introduction/exercise.ipynb".
The first task is:
Splitting the data between a training set and test.
The split operation can be done using different ratios of the train set and test set. I will comment on it later. By now, I decided to take half of the data as training test and the other half as test test. Intuitively, we can already argue that this choice may be the best tradeoff between having a bigger portion of the data as training set (thus facing the risk of overfitting) or the way around, i.e. having a training set too small in order to accurately estimate good parameters that will be able to reproduce the trend observed for unseen data, i.e. for the test set.
By now, I fixed the number of neighbors ("n_neighbors") of the model to 5, thus being able to evaluate the performance (as asked in the next steps) for the fixed hyperparameter "n_neighbors"
Can you write Python code for it?

from sklearn.model_selection import train_test_split

train_performance = []
test_performance = []

for i in range(500):
    X_train, X_test, y_train,y_test = train_test_split(X, y, train_size=0.5)
    kN_classifier = KNeighborsClassifier(n_neighbors=5)
    kN_classifier.fit(X_train, y_train)
    # store the performance of the kN_classifier on the train set
    train_performance.append(kN_classifier.score(X_train, y_train))
    # store the performance of the kN_classifier on the test set
    test_performance.append(kN_classifier.score(X_test, y_test))

train_test_performances = [train_performance, test_performance]

fig = plt.figure()
ax = fig.add_subplot(111)

medianprops = dict(linestyle=':', linewidth=2.5, color='grey')
meanlineprops = dict(linestyle='-', linewidth=2.5, color='black')

bp = ax.boxplot(train_test_performances, medianprops=medianprops, whis=[5,95], meanprops=meanlineprops, meanline=True, showmeans=True)

bp['boxes'][0].set(color = 'g', lw = 2)
bp['boxes'][1].set(color = 'r', lw = 2)

ax.set_xticklabels(['Train', 'Test'])
ax.set_ylabel("Accuracy")


I am working on the file "01-introduction/vitamin-problem.ipynb".
The first task is:
2: The formula must have at least 10% Iron, Zinc, or Magnesium
Can you write Python code for it?

#The formula must have at least 10% Iron, Zinc, or Magnesium
prob += u['Iron'] + u['Zinc'] + u['Magnesium'] >= .1


I am working on the file "01-introduction/vitamin-problem.ipynb".
The first task is:
3: The formula must have at least 20% Vitamin A, Vitamin C, or Vitamin D
Can you write Python code for it?

#The formula must have at least 20% Vitamin A, Vitamin C, or Vitamin D
prob += u['Vitamin A'] + u['Vitamin C'] + u['Vitamin D'] >= .2


I am working on the file "01-introduction/vitamin-problem.ipynb".
The first task is:
4: Each vitamin that is used must account for at least 5% of the total
Can you write Python code for it?

#Each vitamin that is used must account for at least 5% of the total
for v in vitamins:
    #if we don't use this vitamin then the percent must be zero
    prob += u[v] <= b[v]
    #likewise if we do use this vitamin, then the percent must not be zero
    prob += u[v] >= .05 -100*(1-b[v]) # > .05 or > .05 -100


I am working on the file "01-introduction/vitamin-problem.ipynb".
The first task is:
6: If the formula contains Magnesium it must also contain Calcium and Zinc
Can you write Python code for it?

#If the formula contains Magnesium it must also contain Calcium and Zinc
prob += 2*b['Magnesium'] <= b['Calcium'] + b['Zinc']


I am working on the file "01-introduction/vitamin-problem.ipynb".
The first task is:
A new high performance and optimally cheap multivitamin. Saving the world one mixed integer program at a time
You have been tasked with developing a new superior multivitamin. You have been given free reign to select the ingredients and their relative amounts in the vitamin but you have been asked to keep the cost of the raw materials as low as possible.
Further complicating things, you must abide by some restrictions in the formula.

The formula should not have more than 20% of any one vitamin
The formula must have at least 10% Iron, Zinc, or Magnesium
The formula must have at least 20% Vitamin A, Vitamin C, or Vitamin D
Each vitamin that is used must account for at least 5% of the total
The formula may contain as few as 5 vitamins but no more than 10 vitamins
If the formula contains Magnesium it must also contain Calcium and Zinc
The formula must have one of the B vitamins either B6 or B12 but not both

The possible Ingredients and their per mg cost
{{HTML(df.to_html(index=False)) }}
You might see a simple greedy strategy to solve this, but the vitamin B constrains and the Magnesium/Zinc/Calcium constraints make things a bit more complicated. Instead of trial and error we can try and create this vitamin by writing a fairly simple mixed integer linear program. First we need to come up with an expression that captures our ultimate goal, in this case, to minimize the cost of raw materials.
We can calculate the total cost of the formula by adding up the individual cost of each vitamin in it.
$$\text{Total Cost} = \text{ sum over all the vitamins (cost of vitamin) * (percent of vitamin) } $$
First lets load the data we have and organize it so we can easily grab the cost of particular vitamin
Can you write Python code for it?

df = pd.read_csv('vitamin_costs.csv')
vitamins = df.vitamin.values
vitamin_cost = df.set_index('vitamin').to_dict()['cost']
df.head()


I am working on the file "01-introduction/vitamin-problem.ipynb".
The first task is:
Finally, while it wasn't stated as a constraint, our u variables are supposed to be percents, so they must add up to 100%
Can you write Python code for it?

#the percentages must add up to 100
prob += lpSum([ u[v] for v in vitamins]) == 1.0


I am working on the file "01-introduction/vitamin-problem.ipynb".
The first task is:
The next thing we need to do is create an instance of the pulp.LpProblem class. This creates a problem variable that will hold the cost and constraints and tells PuLP that we want to minimize our cost. We pass in a name and either LpMinimize or LpMaximize for either minimizing our cost or maximizing it.
Can you write Python code for it?

prob = LpProblem('super-awesome-vitamin', LpMinimize)


I am working on the file "01-introduction/vitamin-problem.ipynb".
The first task is:
These were fairly straightforward, the code for the constraints and the description of the constraint are almost identical. However the next constraint is a bit stranger.
"4. Each vitamin that is used must account for at least 5% of the total"
We basically need the u variables to be at least 5% or 0%. To model this we need to introduce some new variables that will track in simple yes/no manner if the vitamin is included in the final formula. Once we have these variables we will link them with the u variables somehow and satisfy the rest of the constraints. For now lets just look at the code.
Can you write Python code for it?

#binary variable that captures if this vitamin will be used in the formula
b = LpVariable.dicts('use',vitamins,0,1, LpBinary)


I am working on the file "01-introduction/vitamin-problem.ipynb".
The first task is:
We can now solve the problem and relax knowing we have made the best possible multivitamin (with our highly customized and formalized definition of "best")
Can you write Python code for it?

LpStatus[prob.solve()]

print('total cost: $%.2f'%prob.objective.value())

for v in vitamins:
    if value(u[v]) >0:
        print( '%s %.0f%% at unit cost of: $%.2f' %(v, 100*value(u[v]), vitamin_cost[v]))


I am working on the file "01-iris.ipynb".
The first task is:
Fitting a model
Let's try with KNN. Fit the training data and test on testing dataset
Can you write Python code for it?

model = KNeighborsClassifier()

model.fit(x_train, y_train)

y_hat = model.predict(x_test)


I am working on the file "01-iris.ipynb".
The first task is:
Getting data
Load the dataset from scikit-learn and perform a test train split
Can you write Python code for it?

iris = datasets.load_iris()
X, Y = iris.data, iris.target
x_train, x_test, y_train, y_test = train_test_split(iris.data, iris.target)


I am working on the file "01-iris.ipynb".
The first task is:
Validating the model fit
Can you write Python code for it?

conf_matrix = confusion_matrix(y_test, y_hat, labels=[0,1,2])
plt.imshow(conf_matrix, interpolation="nearest")
accuracy_score(y_test, y_hat) #94.73! not bad!


I am working on the file "01-iterators-and-iterables-review.ipynb".
The first task is:
Iterables
In Python, an Iterable is an instance of any class that defines the __iter__() magic method. Iterator is a subclass of Iterable.
iter(iterable) is the same as iterable.__iter__(), and should always return an iterator for the iterable. This iterator can then be iterated over, in order to retrieve the elements of the iterable.
Iterables (including all iterators) can be one-time use only, but they can also be reusable. When they are reusable, __iter__() will return a brand-new iterator that starts at the first item, and advancing the iterator will have no effect on the original iterable.
Can you write Python code for it?

# Example iterable.

import collections

class RangeIterable(collections.Iterable):
    """Naive implementation of an builtins.range() iterable."""
    def __init__(self, stop):
        super().__init__()
        self.stop = stop

    def __repr__(self):
        return f"{self.__class__.__name__}({self.stop!r})"

    def __iter__(self):
        return RangeIterator(stop=self.stop)

range_iterable = RangeIterable(2)
range_iterable

import traceback

try:
    next(range_iterable)
except TypeError:
    traceback.print_exc()

iter(range_iterable)

iter(range_iterable) is range_iterable

iter(range_iterable) is iter(range_iterable)

next(iter(range_iterable))


I am working on the file "01-iterators-and-iterables-review.ipynb".
The first task is:
Syntax for iteration: for-loops
Python code can, and often does, manually iterate through iterators and iterables, via iter() and next(). However, the Python language syntax has built-in support for automatically iterating over the elements of an iterable. The for-loop, which is common in most modern programming languages (though Python's for-loop is really a for-each loop), runs through an iterable one element at a time, making use of the current element in a block of code.
python
for item in iterable:
    # do something with item, such as
    print(item)
If we ignore the semantics of continue, break, and else, a for-loop generally looks like
python
for TARGET in ITER:
    BLOCK
and is syntactic sugar for something like
python
iterable = (ITER)
iterator = iter(iterable)
running = True
while running:
    try:
        TARGET = next(iterator)
    except StopIteration:
        running = False
    else:
        BLOCK
Note that the for-loop construct has special handling for StopIteration. The for-loop construct is intimately aware of the iterator protocol, and knows to catch StopIteration and interpret it as the end of iteration.
Can you write Python code for it?

for item in RangeIterable(2):
    print(item)

def manual_simplified_for_loop(iterable, function):
    iterator = iter(iterable)
    running = True
    while running:
        try:
            item = next(iterator)
        except StopIteration:
            running = False
        else:
            function(item)

manual_simplified_for_loop(RangeIterable(2), print)


I am working on the file "01-load-vectorize-count.ipynb".
The first task is:
Load Data
Can you write Python code for it?

train_df = pd.read_csv("data/train.tsv", sep="\t")

train_df.sample(10)

# Load the Test Dataset
# Note that it's missing the Sentiment Column.  That's what we need to Predict
#
test_df = pd.read_csv("data/test.tsv", sep="\t")
test_df.head()


I am working on the file "01-load-vectorize-count.ipynb".
The first task is:
Reasons for Testing Accuracy Being Lower?

One Hypothesis, Since we are just Adding up ALL of the scores for each word in the review, the length of the reivew could have an impact.  Let's look at length of reviews in train and test dataset
Can you write Python code for it?

import matplotlib.pyplot as plt

%matplotlib inline

train_df.review.str.len().hist(log=True)

test_df.review.str.len().hist(log=True)


I am working on the file "01-load-vectorize-count.ipynb".
The first task is:
Vectorize Data (a.k.a. covert text to numbers)
Computers don't understand Texts, so we need to convert texts to numbers before we could do any math on it and see if we can build a system to classify a review as Positive or Negative.
Ways to vectorize data:

Bag of Words
TF-IDF
Word Embeddings (Word2Vec)

Bag of Words
Take each sentence and count how many occurances of a particular word.
Can you write Python code for it?

## Doing it by Hand

def bag_of_words_vocab(reviews):
    """Returns words in the reviews"""
    # all_words = []
    # for review in reviews:
    #    for word in review.split():
    #       all_words.append(word)
    ## List comprehension method of the same lines above
    all_words = [word.lower() for review in reviews for word in review.split(" ")]
    return Counter(all_words)

words_vocab = bag_of_words_vocab(train_df.review)

words_vocab.most_common(20)


I am working on the file "01-loading/.ipynb_checkpoints/Pandas Large Data Optimization-checkpoint.ipynb".
The first task is:
A quick glance reveals many columns where there are few unique values relative to the overall ~172,000 games in our data set.
We start to convert day_of_week to category using the .astype(method).
Can you write Python code for it?

dow = gl_obj.day_of_week
print(dow.head())

dow_cat = dow.astype('category')
print(dow_cat.head())

# We can see the integer values associated to column
dow_cat.head().cat.codes

# We compare the memory usage
print(mem_usage(dow))
print(mem_usage(dow_cat))


I am working on the file "01-loading/.ipynb_checkpoints/Pandas Large Data Optimization-checkpoint.ipynb".
The first task is:
First Look
Can you write Python code for it?

gl.describe()

# Reference http://www.markhneedham.com/blog/2017/07/05/pandas-find-rows-where-columnfield-is-null/
# Columns with null values

null_columns=gl.columns[gl.isnull().any()]
gl[null_columns].isnull().sum()

# Every row that contains at least one null value
print(gl[gl.isnull().any(axis=1)][null_columns].head())


I am working on the file "01-loading/.ipynb_checkpoints/Pandas Large Data Optimization-checkpoint.ipynb".
The first task is:
Optimize Numeric Columns with subtypes
We can use the function pd.to_numeric() to downcast our numeric types
Can you write Python code for it?

# We're going to be calculating memory usage a lot,
# so we'll create a function to save us some time!

def mem_usage(pandas_obj):
    if isinstance(pandas_obj,pd.DataFrame):
        usage_b = pandas_obj.memory_usage(deep=True).sum()
    else: # we assume if not a df it's a series
        usage_b = pandas_obj.memory_usage(deep=True)
    usage_mb = usage_b / 1024 ** 2 # convert bytes to megabytes
    return "{:03.2f} MB".format(usage_mb)

mem_usage(gl)

gl_int = gl.select_dtypes(include=['int64'])
converted_int = gl_int.apply(pd.to_numeric,downcast='unsigned')

print(mem_usage(gl_int))
print(mem_usage(converted_int))

compare_ints = pd.concat([gl_int.dtypes,converted_int.dtypes],axis=1)
compare_ints.columns = ['before','after']
compare_ints.apply(pd.Series.value_counts)


I am working on the file "01-loading/.ipynb_checkpoints/Pandas Large Data Optimization-checkpoint.ipynb".
The first task is:
Selecting Types While Reading the Data In
we can specify the optimal column types when we read the data set in. The pandas.read_csv() function has a few different parameters that allow us to do this. The dtype parameter accepts a dictionary that has (string) column names as the keys and NumPy type objects as the values.
Can you write Python code for it?

dtypes = optimized_gl.drop('date',axis=1).dtypes
dtypes.head()

dtypes_col = dtypes.index
dtypes_col

dtypes_type=[i.name for i in dtypes.values]

column_types = dict(zip(dtypes_col, dtypes_type))

#Preview of first 10
{k:v for k,v in list(column_types.items())[:10]}


I am working on the file "01-loading/.ipynb_checkpoints/Pandas Large Data Optimization-checkpoint.ipynb".
The first task is:
Under the hood, pandas groups the columns into block of values of the same type:
- ObjectBlock, contains string
- FloatBlock (ndarray)
- IntBlock (ndarray)
Because each data type is stored separately, we examine the memory usage by data type.
Average memory usage for data type
Can you write Python code for it?

gl.dtypes.value_counts()

for dtype in gl.dtypes.unique(): #['float','int64','object']:
    selected_dtype = gl.select_dtypes(include=[dtype])
    mean_usage_b = selected_dtype.memory_usage(deep=True).mean()
    mean_usage_mb = mean_usage_b / 1024 ** 2
    print("Average memory usage for {} columns: {:03.2f} MB".format(dtype,mean_usage_mb))


I am working on the file "01-loading/.ipynb_checkpoints/Pandas Large Data Optimization-checkpoint.ipynb".
The first task is:
We can see a drop from 7.9 to 1.5 MB. Now we have 5 uint8 and 1 unit32 instead of 6 int64.
Lets do the same thing with float columns
Can you write Python code for it?

gl_float = gl.select_dtypes(include=['float'])
converted_float = gl_float.apply(pd.to_numeric,downcast='float')

print(mem_usage(gl_float))
print(mem_usage(converted_float))

compare_floats = pd.concat([gl_float.dtypes,converted_float.dtypes],axis=1)
compare_floats.columns = ['before','after']
compare_floats.apply(pd.Series.value_counts)


I am working on the file "01-loading/.ipynb_checkpoints/Pandas Large Data Optimization-checkpoint.ipynb".
The first task is:
We’ll convert using pandas.to_datetime() function, using the format parameter to tell it that our date data is stored YYYY-MM-DD.
Can you write Python code for it?

optimized_gl['date'] = pd.to_datetime(date,format='%Y%m%d')

print(mem_usage(optimized_gl))
optimized_gl.date.head()


I am working on the file "01-loading/.ipynb_checkpoints/Pandas Large Data Optimization-checkpoint.ipynb".
The first task is:
When we convert columns to category, it's important to be aware of trade-off:
- we can't perform numerical computation on category
- we should use the category type primarily for object column where less than 50% of the values are unique
Can you write Python code for it?

converted_obj = pd.DataFrame()

for col in gl_obj.columns:
    num_unique_values = len(gl_obj[col].unique())
    num_total_values = len(gl_obj[col])
    if num_unique_values / num_total_values < 0.5:
        converted_obj.loc[:,col] = gl_obj[col].astype('category')
    else:
        converted_obj.loc[:,col] = gl_obj[col]

print(mem_usage(gl_obj))
print(mem_usage(converted_obj))

compare_obj = pd.concat([gl_obj.dtypes,converted_obj.dtypes],axis=1)
compare_obj.columns = ['before','after']
compare_obj.apply(pd.Series.value_counts)

# Now we combine with the rest of our dataframe (numeric columns)
optimized_gl[converted_obj.columns] = converted_obj

mem_usage(optimized_gl)


I am working on the file "01-loading/.ipynb_checkpoints/Pandas Large Data Optimization-checkpoint.ipynb".
The first task is:
You can see that the size of strings when stored in a pandas series are identical to their usage as separate strings in Python.
Optimizing object types using categoricals
The category type uses integer values under the hood to represent the values in a column, rather than the raw values. Pandas uses a separate mapping dictionary that maps the integer values to the raw ones. This arrangement is useful whenever a column contains a limited set of values. When we convert a column to the category dtype, pandas uses the most space efficient int subtype that can represent all of the unique values in a column.
Can you write Python code for it?

#Where we migh be able to reduce memory?
gl_obj = gl.select_dtypes(include=['object']).copy()
gl_obj.describe()


I am working on the file "01-loading/.ipynb_checkpoints/[Optimization]Pandas Large Data Optimization-checkpoint.ipynb".
The first task is:
Analyzing baseball games
how game length has varied over the years
Can you write Python code for it?

import matplotlib.pyplot as plt
optimized_gl['year'] = optimized_gl.date.dt.year
game_lengths = optimized_gl.pivot_table(index='year', values='length_minutes')
game_lengths.reset_index().plot.scatter('year','length_minutes')
plt.show()


I am working on the file "01-loading/.ipynb_checkpoints/[Optimization]Pandas Large Data Optimization-checkpoint.ipynb".
The first task is:
Convert date
Can you write Python code for it?

date = optimized_gl.date
print(mem_usage(date))
date.head()


I am working on the file "01-loading/.ipynb_checkpoints/[Optimization]Pandas Large Data Optimization-checkpoint.ipynb".
The first task is:
Subtype
Under the hood pandas represents numeric values as NumPy ndarrays and stores them in a continuous block of memory.
This approach:
- consumes less space
- allows us to access quickly
Many types in pandas have multiple subtypes that can use fewer bytes to represent each value. For example, the float type has the float16, float32, and float64 subtypes.
To discovare the range of values of given dtype we can use
https://docs.scipy.org/doc/numpy-1.10.0/reference/generated/numpy.iinfo.html
Can you write Python code for it?

import numpy as np
int_types = ["uint8", "int8", "int16"]
for it in int_types:
    print(np.iinfo(it))


I am working on the file "01-loading/.ipynb_checkpoints/[Optimization]Pandas Large Data Optimization-checkpoint.ipynb".
The first task is:
Under the hood, pandas groups the columns into block of values of the same type:
- ObjectBlock, contains string
- FloatBlock (ndarray)
- IntBlock (ndarray)
Because each data type is stored separately, we examine the memory usage by data type.
Average memory usage for data type
Can you write Python code for it?


I am working on the file "01-loading/.ipynb_checkpoints/[Optimization]Pandas Large Data Optimization-checkpoint.ipynb".
The first task is:
We can see a drop from 7.9 to 1.5 MB. Now we have 5 uint8 and 1 unit32 instead of 6 int64.
Lets do the same thing with float columns
Can you write Python code for it?


I am working on the file "01-loading/.ipynb_checkpoints/[Optimization]Pandas Large Data Optimization-checkpoint.ipynb".
The first task is:
When we convert columns to category, it's important to be aware of trade-off:
- we can't perform numerical computation on category
- we should use the category type primarily for object column where less than 50% of the values are unique
Can you write Python code for it?


I am working on the file "01-loading/.ipynb_checkpoints/[Optimization]Pandas Large Data Optimization-checkpoint.ipynb".
The first task is:
You can see that the size of strings when stored in a pandas series are identical to their usage as separate strings in Python.
Optimizing object types using categoricals
The category type uses integer values under the hood to represent the values in a column, rather than the raw values. Pandas uses a separate mapping dictionary that maps the integer values to the raw ones. This arrangement is useful whenever a column contains a limited set of values. When we convert a column to the category dtype, pandas uses the most space efficient int subtype that can represent all of the unique values in a column.
Can you write Python code for it?


I am working on the file "01-loading/[Optimization]Pandas Large Data Optimization.ipynb".
The first task is:
All our float columns were converted from float64 to float32, give us a 50 reduction in memory usage.
We apply this optimizations on the entire df
Can you write Python code for it?

optimized_gl = gl.copy()

optimized_gl[converted_int.columns] = converted_int
optimized_gl[converted_float.columns] = converted_float

print(mem_usage(gl))
print(mem_usage(optimized_gl))


I am working on the file "01-loading/[Optimization]Pandas Large Data Optimization.ipynb".
The first task is:
Analyzing baseball games
how game length has varied over the years
Can you write Python code for it?

import matplotlib.pyplot as plt
optimized_gl['year'] = optimized_gl.date.dt.year
game_lengths = optimized_gl.pivot_table(index='year', values='length_minutes')
game_lengths.reset_index().plot.scatter('year','length_minutes')
plt.show()

from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"

11
2


I am working on the file "01-loading/[Optimization]Pandas Large Data Optimization.ipynb".
The first task is:
Convert date
Can you write Python code for it?


I am working on the file "01-loading/[Optimization]Pandas Large Data Optimization.ipynb".
The first task is:
In order to have the best benefit, we have to optimize the object types.
Comparing Numeric and String Storage
The object type represents values using Python string objects, partly due to the lack of support for missing string values in NumPy. Because Python is a high-level, interpreted language, it doesn’t have fine grained-control over how values in memory are stored.
This limitation causes strings to be stored in a fragmented way that consumes more memory and is slower to access. Each element in an object column is really a pointer that contains the “address” for the actual value’s location in memory.
Object types as using a variable amount of memory. While each pointer takes up 1 byte of memory, each actual string value uses the same amount of memory that string would use if stored individually in Python. Let’s use sys.getsizeof() to prove that out, first by looking at individual strings, and then items in a pandas series.
Can you write Python code for it?

from sys import getsizeof

s1 = 'working out'
s2 = 'memory usage for'
s3 = 'strings in python is fun!'
s4 = 'strings in python is fun!'

for s in [s1, s2, s3, s4]:
    print(getsizeof(s))

obj_series = pd.Series(['working out',
                          'memory usage for',
                          'strings in python is fun!',
                          'strings in python is fun!'])
obj_series.apply(getsizeof)


I am working on the file "01-loading/[Optimization]Pandas Large Data Optimization.ipynb".
The first task is:
Now we can use the dictionary, along with a few parameters for the date to read in the data with the correct types in a few lines:
Can you write Python code for it?

read_and_optimized = pd.read_csv('..\data\game_logs.csv',dtype=column_types,parse_dates=['date'],infer_datetime_format=True)

print(mem_usage(read_and_optimized))
read_and_optimized.head()


I am working on the file "01-loading/[Optimization]Pandas Large Data Optimization.ipynb".
The first task is:
Optimize Numeric Columns with subtypes
We can use the function pd.to_numeric() to downcast our numeric types
Can you write Python code for it?

%%writefile mem_usage.py
# We're going to be calculating memory usage a lot,
# so we'll create a function to save us some time!

def mem_usage(pandas_obj):
    if isinstance(pandas_obj,pd.DataFrame):
        usage_b = pandas_obj.memory_usage(deep=True).sum()
    else: # we assume if not a df it's a series
        usage_b = pandas_obj.memory_usage(deep=True)
    usage_mb = usage_b / 1024 ** 2 # convert bytes to megabytes
    return "{:03.2f} MB".format(usage_mb)

mem_usage(gl)M

gl_int = gl.select_dtypes(include=['int64'])
converted_int = gl_int.apply(pd.to_numeric,downcast='unsigned')

print(mem_usage(gl_int))
print(mem_usage(converted_int))

compare_ints = pd.concat([gl_int.dtypes,converted_int.dtypes],axis=1)

compare_ints = pd.concat([gl_int.dtypes,converted_int.dtypes],axis=1)
compare_ints.columns = ['before','after']
compare_ints.apply(pd.Series.value_counts)


I am working on the file "01-loading/[Optimization]Pandas Large Data Optimization.ipynb".
The first task is:
Original Article
https://www.dataquest.io/blog/pandas-big-data/
Can you write Python code for it?

gl.dtypes.head()

# Select only the column with same type
gl.select_dtypes(include=['object']).head()

#Exact amount of memory usage of df
gl.info(memory_usage='deep')


I am working on the file "01-loading/[Optimization]Pandas Large Data Optimization.ipynb".
The first task is:
Selecting Types While Reading the Data In
we can specify the optimal column types when we read the data set in. The pandas.read_csv() function has a few different parameters that allow us to do this. The dtype parameter accepts a dictionary that has (string) column names as the keys and NumPy type objects as the values.
Can you write Python code for it?


I am working on the file "01-loading/[Optimization]Pandas Large Data Optimization.ipynb".
The first task is:
Subtype
Under the hood pandas represents numeric values as NumPy ndarrays and stores them in a continuous block of memory.
This approach:
- consumes less space
- allows us to access quickly
Many types in pandas have multiple subtypes that can use fewer bytes to represent each value. For example, the float type has the float16, float32, and float64 subtypes.
To discovare the range of values of given dtype we can use
https://docs.scipy.org/doc/numpy-1.10.0/reference/generated/numpy.iinfo.html
Can you write Python code for it?


I am working on the file "01-loading/[Optimization]Pandas Large Data Optimization.ipynb".
The first task is:
Under the hood, pandas groups the columns into block of values of the same type:
- ObjectBlock, contains string
- FloatBlock (ndarray)
- IntBlock (ndarray)
Because each data type is stored separately, we examine the memory usage by data type.
Average memory usage for data type
Can you write Python code for it?


I am working on the file "01-loading/[Optimization]Pandas Large Data Optimization.ipynb".
The first task is:
We can see a drop from 7.9 to 1.5 MB. Now we have 5 uint8 and 1 unit32 instead of 6 int64.
Lets do the same thing with float columns
Can you write Python code for it?


I am working on the file "01-loading/[Optimization]Pandas Large Data Optimization.ipynb".
The first task is:
When we convert columns to category, it's important to be aware of trade-off:
- we can't perform numerical computation on category
- we should use the category type primarily for object column where less than 50% of the values are unique
Can you write Python code for it?


I am working on the file "01-loading/[Optimization]Pandas Large Data Optimization.ipynb".
The first task is:
You can see that the size of strings when stored in a pandas series are identical to their usage as separate strings in Python.
Optimizing object types using categoricals
The category type uses integer values under the hood to represent the values in a column, rather than the raw values. Pandas uses a separate mapping dictionary that maps the integer values to the raw ones. This arrangement is useful whenever a column contains a limited set of values. When we convert a column to the category dtype, pandas uses the most space efficient int subtype that can represent all of the unique values in a column.
Can you write Python code for it?


I am working on the file "01-notebook.ipynb".
The first task is:
Anschließend kann der StanfordParser damit geladen werden.
Can you write Python code for it?

parser = StanfordParser(
    jar, ger_model,
    model_path="edu/stanford/nlp/models/lexparser/" +
        "germanPCFG.ser.gz"
)


I am working on the file "01-notebook.ipynb".
The first task is:
Englische Dependenzbäume
Dieses zweite Beispiel zeigt die Funktionsweise des Dependenzparsers und die des englischen Modells.
Analog müssen die Pfade zu CoreNLP und Modell definiert werden:
Can you write Python code for it?

jar = os.path.join(
    PATH_TO_CORE,
    "stanford-corenlp-3.8.0.jar"
)
model = os.path.join(
    PATH_TO_CORE,
    "stanford-corenlp-3.8.0-models.jar"
)


I am working on the file "01-notebook.ipynb".
The first task is:
Für eine syntaktische Analyse verwenden Sie einfach raw_parse.
Can you write Python code for it?

tree_list = list(parser.raw_parse('Dies ist mein erster Satz.'))
tree_list[0].pretty_print(unicodelines=True)


I am working on the file "01-notebook.ipynb".
The first task is:
Möchte man mehr als nur einen Baum in derselben Zelle anzeigen, bietet display mehr Flexibilität.
Can you write Python code for it?

from IPython.display import display

display(np1)
display(np2)


I am working on the file "01-notebook.ipynb".
The first task is:
Probieren Sie auch die Default-Darstellung dieser Dependency-Graphen in Jupyter aus.
(Sie müssen dafür evtl. graphviz installieren.)
Can you write Python code for it?


I am working on the file "01-notebook.ipynb".
The first task is:
Stanford Parser
Wenn Sie den Stanford Parser erfolgreich installiert haben, können Sie auch folgende Zellen ausprobieren.
Can you write Python code for it?

from nltk.parse.stanford import StanfordParser, StanfordDependencyParser
import os


I am working on the file "01-pandas_introduction.ipynb".
The first task is:
4. Exploration
Some useful methods:
head and tail
Can you write Python code for it?


I am working on the file "01-pandas_introduction.ipynb".
The first task is:
6. The group-by operation
Some 'theory': the groupby operation (split-apply-combine)
Can you write Python code for it?


I am working on the file "01-pandas_introduction.ipynb".
The first task is:
7. Working with time series data
Can you write Python code for it?

no2 = pd.read_csv('data/20000101_20161231-NO2.csv', sep=';', skiprows=[1], na_values=['n/d'], index_col=0, parse_dates=True)


I am working on the file "01-pandas_introduction.ipynb".
The first task is:

EXERCISE:

Calculate the average survival ratio for all passengers (note: the 'Survived' column indicates whether someone survived (1) or not (0)).

Can you write Python code for it?


I am working on the file "01-pandas_introduction.ipynb".
The first task is:

EXERCISE:

Calculate the average survival ratio for all passengers.

Can you write Python code for it?


I am working on the file "01-pandas_introduction.ipynb".
The first task is:

EXERCISE:

Calculate this survival ratio for all passengers younger that 25 (remember: filtering/boolean indexing).

Can you write Python code for it?


I am working on the file "01-pandas_introduction.ipynb".
The first task is:

EXERCISE:

Make a bar plot to visualize the average Fare payed by people depending on their age. The age column is devided is separate classes using the `pd.cut` function as provided below.

Can you write Python code for it?

df['AgeClass'] = pd.cut(df['Age'], bins=np.arange(0,90,10))

# %load snippets/01-pandas_introduction82.py


I am working on the file "01-pandas_introduction.ipynb".
The first task is:

EXERCISE:

What is the maximum Fare that was paid? And the median?

Can you write Python code for it?

# %load snippets/01-pandas_introduction31.py

# %load snippets/01-pandas_introduction32.py


I am working on the file "01-pandas_introduction.ipynb".
The first task is:

EXERCISE: The evolution of the yearly averages with, and the overall mean of all stations

Use `resample` and `plot` to plot the yearly averages for the different stations.

The overall mean of all stations can be calculated by taking the mean of the different columns (`.mean(axis=1)`).

Can you write Python code for it?


I am working on the file "01-pandas_introduction.ipynb".
The first task is:

EXERCISE: The typical diurnal profile for the different stations

Similar as for the month, you can now group by the hour of the day.

Can you write Python code for it?


I am working on the file "01-pandas_introduction.ipynb".
The first task is:
A range of methods:
Can you write Python code for it?


I am working on the file "01-pandas_introduction.ipynb".
The first task is:
Add a column indicating week/weekend
Can you write Python code for it?


I am working on the file "01-pandas_introduction.ipynb".
The first task is:
Also a DataFrame has a values attribute, but attention: when you have heterogeneous data, all values will be upcasted:
Can you write Python code for it?


I am working on the file "01-pandas_introduction.ipynb".
The first task is:
And many more methods are available.
Can you write Python code for it?


I am working on the file "01-pandas_introduction.ipynb".
The first task is:
Apart from importing your data from an external source (text file, excel, database, ..), one of the most common ways of creating a dataframe is from a dictionary of arrays or lists.
Note that in the IPython notebook, the dataframe will display in a rich HTML view:
Can you write Python code for it?

data = {'country': ['Belgium', 'France', 'Germany', 'Netherlands', 'United Kingdom'],
        'population': [11.3, 64.3, 81.3, 16.9, 64.9],
        'area': [30510, 671308, 357050, 41526, 244820],
        'capital': ['Brussels', 'Paris', 'Berlin', 'Amsterdam', 'London']}
df_countries = pd.DataFrame(data)
df_countries


I am working on the file "01-pandas_introduction.ipynb".
The first task is:
Application of the groupby concept on the titanic data
We go back to the titanic passengers survival data:
Can you write Python code for it?

df = pd.read_csv("data/titanic.csv")

df.head()


I am working on the file "01-pandas_introduction.ipynb".
The first task is:
Attributes of a Series: index and values
The Series has also an index and values attribute, but no columns
Can you write Python code for it?


I am working on the file "01-pandas_introduction.ipynb".
The first task is:
Attributes of the DataFrame
A DataFrame has besides a index attribute, also a columns attribute:
Can you write Python code for it?


I am working on the file "01-pandas_introduction.ipynb".
The first task is:
Boolean indexing (filtering)
Often, you want to select rows based on a certain condition. This can be done with 'boolean indexing' (like a where clause in SQL) and comparable to numpy.
The indexer (or boolean mask) should be 1-dimensional and the same length as the thing being indexed.
Can you write Python code for it?


I am working on the file "01-pandas_introduction.ipynb".
The first task is:
But also a lot of pandas specific methods, e.g.
Can you write Python code for it?


I am working on the file "01-pandas_introduction.ipynb".
The first task is:
Fancy indexing, like indexing with a list or boolean indexing:
Can you write Python code for it?


I am working on the file "01-pandas_introduction.ipynb".
The first task is:
Getting some basic summary statistics about the data with describe:
Can you write Python code for it?


I am working on the file "01-pandas_introduction.ipynb".
The first task is:
However, in many cases your data has certain groups in it, and in that case, you may want to calculate this statistic for each of the groups.
For example, in the above dataframe df, there is a column 'key' which has three possible values: 'A', 'B' and 'C'. When we want to calculate the sum for each of those groups, we could do the following:
Can you write Python code for it?

for key in ['A', 'B', 'C']:
    print(key, df[df['key'] == key]['data'].sum())


I am working on the file "01-pandas_introduction.ipynb".
The first task is:
Luckily, if we have a well formed csv file, we don't need many of those arguments:
Can you write Python code for it?


I am working on the file "01-pandas_introduction.ipynb".
The first task is:
Now we can groupby the hour of the day and the weekend (or use pivot_table):
Can you write Python code for it?

# %load snippets/01-pandas_introduction104.py

# %load snippets/01-pandas_introduction105.py

# %load snippets/01-pandas_introduction106.py

# %load snippets/01-pandas_introduction107.py


I am working on the file "01-pandas_introduction.ipynb".
The first task is:
Now, we can calculate the mean of each month over the different years:
Can you write Python code for it?

# %load snippets/01-pandas_introduction98.py

# %load snippets/01-pandas_introduction99.py


I am working on the file "01-pandas_introduction.ipynb".
The first task is:
One-dimensional data: Series (a column of a DataFrame)
A Series is a basic holder for one-dimensional labeled data.
Can you write Python code for it?


I am working on the file "01-pandas_introduction.ipynb".
The first task is:
Or we can use some more advanced time series features -> see further in this notebook!
5. Selecting and filtering data

ATTENTION!:

One of pandas' basic features is the labeling of rows and columns, but this makes indexing also a bit more complex compared to numpy.

 We now have to distuinguish between:

selection by **label**

selection by **position**

Can you write Python code for it?


I am working on the file "01-pandas_introduction.ipynb".
The first task is:
Recap: aggregating functions
When analyzing data, you often calculate summary statistics (aggregations like the mean, max, ...). As we have seen before, we can easily calculate such a statistic for a Series or column using one of the many available methods. For example:
Can you write Python code for it?


I am working on the file "01-pandas_introduction.ipynb".
The first task is:
Selecting by position with iloc works similar as indexing numpy arrays:
Can you write Python code for it?


I am working on the file "01-pandas_introduction.ipynb".
The first task is:
The default plot (when not specifying kind) is a line plot of all columns:
Can you write Python code for it?


I am working on the file "01-pandas_introduction.ipynb".
The first task is:
The different indexing methods can also be used to assign data:
Can you write Python code for it?

df.loc['Braund, Mr. Owen Harris', 'Survived'] = 100

df


I am working on the file "01-pandas_introduction.ipynb".
The first task is:
This becomes very verbose when having multiple groups. You could make the above a bit easier by looping over the different values, but still, it is not very convenient to work with.
What we did above, applying a function on different groups, is a "groupby operation", and pandas provides some convenient functionality for this.
Groupby: applying functions per group
The "group by" concept: we want to apply the same function on subsets of your dataframe, based on some key to split the dataframe in subsets
This operation is also referred to as the "split-apply-combine" operation, involving the following steps:

Splitting the data into groups based on some criteria
Applying a function to each group independently
Combining the results into a data structure

<img src="img/splitApplyCombine.png">
Similar to SQL GROUP BY
Instead of doing the manual filtering as above
df[df['key'] == "A"].sum()
df[df['key'] == "B"].sum()
...

pandas provides the groupby method to do exactly this:
Can you write Python code for it?

df.groupby('key').sum()

df.groupby('key').aggregate(np.sum)  # 'sum'


I am working on the file "01-pandas_introduction.ipynb".
The first task is:
This does not say too much ..
We can select part of the data (eg the latest 500 data points):
Can you write Python code for it?


I am working on the file "01-pandas_introduction.ipynb".
The first task is:
To check the data types of the different columns:
Can you write Python code for it?


I am working on the file "01-pandas_introduction.ipynb".
The first task is:
Unlike the NumPy array, though, this index can be something other than integers:
Can you write Python code for it?

df = df.set_index('Name')
df

age = df['Age']
age

age['Dooley, Mr. Patrick']


I am working on the file "01-pandas_introduction.ipynb".
The first task is:
We can access series values via the index, just like for NumPy arrays:
Can you write Python code for it?


I am working on the file "01-pandas_introduction.ipynb".
The first task is:
We will come back to these example, and build them up step by step.
2. Pandas: data analysis in python
For data-intensive work in Python the Pandas library has become essential.
What is pandas?

Pandas can be thought of as NumPy arrays with labels for rows and columns, and better support for heterogeneous data types, but it's also much, much more than that.
Pandas can also be thought of as R's data.frame in Python.
Powerful for working with missing data, working with time series data, for reading and writing your data, for reshaping, grouping, merging your data, ...

It's documentation: http://pandas.pydata.org/pandas-docs/stable/
 When do you need pandas?
When working with tabular or structured data (like R dataframe, SQL table, Excel spreadsheet, ...):

Import data
Clean up messy data
Explore data, gain insight into data
Process and prepare your data for analysis
Analyse your data (together with scikit-learn, statsmodels, ...)

ATTENTION!:

Pandas is great for working with heterogeneous and tabular 1D/2D data, but not all types of data fit in such structures!

When working with array data (e.g. images, numerical algorithms): just stick with numpy

When working with multidimensional labeled data (e.g. climate data): have a look at [xarray](http://xarray.pydata.org/en/stable/)

2. The pandas data structures: DataFrame and Series
A DataFrame is a tablular data structure (multi-dimensional object to hold labeled data) comprised of rows and columns, akin to a spreadsheet, database table, or R's data.frame object. You can think of it as multiple Series object which share the same index.

Can you write Python code for it?


I am working on the file "01-pandas_introduction.ipynb".
The first task is:
You can access the underlying numpy array representation with the .values attribute:
Can you write Python code for it?


I am working on the file "01-pandas_introduction.ipynb".
The first task is:
but with the power of numpy arrays. Many things you can do with numpy arrays, can also be applied on DataFrames / Series.
Eg element-wise operations:
Can you write Python code for it?


I am working on the file "01-pandas_introduction.ipynb".
The first task is:
df[] provides some convenience shortcuts
For a DataFrame, basic indexing selects the columns.
Selecting a single column:
Can you write Python code for it?


I am working on the file "01-pandas_introduction.ipynb".
The first task is:
to answering questions about this data in a few lines of code:
Does the air pollution show a decreasing trend over the years?
Can you write Python code for it?


I am working on the file "01-python-intro/aula-01/Aula 01.ipynb".
The first task is:
'owner' é um dicionário dentro do dicionário do repositório (sim, é possível guardar dicionários dentro de dicionários)
Também podemos acessar as informações diretamente sem recorrer à variável intermediária dono:
Can you write Python code for it?


I am working on the file "01-python-intro/aula-01/Aula 01.ipynb".
The first task is:
200 é o código de resposta que informa que a requisição foi bem sucedida (ou OK).
PS: No curso de Desenvolvimento Web com Django será explicado com calma como funciona o protocolo HTTP.
Agora vamos analisar os cabeçalhos dessa resposta:
Can you write Python code for it?


I am working on the file "01-python-intro/aula-01/Aula 01.ipynb".
The first task is:
A função range(inicio, fim) cria listas de valores no intervalo de inicio até fim - 1. Ao lidar com intervalos no python o primeiro número é sempre incluso e o último excluído. Segue alguns exemplos de uso da função range()
Can you write Python code for it?

list(range(10))  # números de 0 a 9

list(range(10, 20))  # números de 10 a 19

list(range(10, 21))  # números de 10 a 20


I am working on the file "01-python-intro/aula-01/Aula 01.ipynb".
The first task is:
Agora vamos fazer uma requisição HTTP ao Github para baixar o conteúdo do repositório deste curso (através da URL https://github.com/lamenezes/python-intro):
Can you write Python code for it?

url = 'https://github.com/lamenezes/python-intro'  # não é necessário declarar variáveis em python
response = requests.get(url)  # e nem especificar seu tipo
response.status_code


I am working on the file "01-python-intro/aula-01/Aula 01.ipynb".
The first task is:
Caso você queira imprimir os valores é preciso usar notas.values():
Can you write Python code for it?

for valor in notas.values():
    print(valor)


I am working on the file "01-python-intro/aula-01/Aula 01.ipynb".
The first task is:
Como visto acima o notas.items() retorna uma lista de chaves e valores. Por esse motivo a cada iteração temos acesso a cada chave e valor do dicionário notas, tornando possível essa iteração mais simples e semântica.
Agora que já temos uma noção das estruturas de dados utilizadas, podemos continuar a visualizar as informações dos repositórios do github:
PS: relaxe, pois nas próximas aulas veremos a fundo essas estruturas!
De volta à API do github
Anteriormente realizamos uma requisição à API do github para pegar os repositórios do usuário lamenezes e armazenamos a resposta na variável response. Essa variável ainda está disponível para nós, o que nos permite continuar nosso exemplo anterior:
Can you write Python code for it?

repositorios = response.json()  # lista de dicionários com dados de cada repositório
repo = repositorios[11]  # vamos analisar o décimo primeiro repositório
repo

repo['full_name']

repo['description']

repo['created_at']  # data de criação

repo['html_url']  # URL da página principal do repositório


I am working on the file "01-python-intro/aula-01/Aula 01.ipynb".
The first task is:
Como visto anteriormente a biblioteca requests permite o envio de requisições HTTP de maneira extremamante fácil. Para obter mais informações sobre a requests acesse a doumentação oficial.
Agora que já aprendemos o básico podemos partir para um exemplo mais interessante.
Acessando os dados da API do github
O Github oferece uma API que fornece dados de seus repositórios públicos. São dadas informações como quantos projetos estão cadastrados na plataforma, quantos contribuidores trabalharam em quais repositórios, a quais organizações essas fazem parte e muito mais. (mais informações na documentação oficial)
O acesso aos dados dessa API é feito através do envio de requisições HTTP para diferentes caminhos na URL base https://api.github.com/. Caso você queira testar a API, pode acessar o link anterior pelo navegador e mudar a URL manualmente, porém nosso foco aqui será usar a biblioteca requests para pegar esses dados.
Todas as respostas da API são em formato JSON. O JSON siginifca JavaScript Object Notation e pronuncia-se "djêizon" e é um formato fácil de processar e simples de utilizar. Para mais informações sobre o JSON acesse a página oficial. Falaremos mais sobre JSON no decorrer do curso, mas fique tranquilo que não segredo há algum nesse assunto.
Vamos testar a API e pegar os repositórios de um usuário específico através da URL https://api.github.com/users/lamenezes/repos:
Can you write Python code for it?

response = requests.get('https://api.github.com/users/lamenezes/repos')
response.status_code

response.text[:1000]  # pegando os primeiros 1000 caracters do conteúdo da resposta


I am working on the file "01-python-intro/aula-01/Aula 01.ipynb".
The first task is:
Dicionários
Dicionários são estruturas de dados utilizadas para armazenar conjuntos de chaves e valores. Em algumas linguagens também são conhecidos como mapaemento.
A diferança de uso entre lista e dicionáro é que o último é acessado por chaves e não pela posição. As chaves podem ser inteiros, strings, booleanos e tuplas (este será visto na próxima aula).
A seguir temos alguns simples exemplos de uso de diconários.
Criar um dict é simples:
Can you write Python code for it?

notas = {'joao': 5, 'maria': 9, 'ana': 7}
notas


I am working on the file "01-python-intro/aula-01/Aula 01.ipynb".
The first task is:
Exercicio
Crie uma lista de preços de produtos em dólar (somente números) e imprima seu valor convertido para reais.
Can you write Python code for it?

taxa_dolar = 3.53  # mude esse valor caso o valor do dólar tenha mudado
preços = ...  # python 3 permite declarar variáveis com acentos
for preço in preço:
    print(...)


I am working on the file "01-python-intro/aula-01/Aula 01.ipynb".
The first task is:
Exercicios

Quantos repositórios o usuário lamenezes possui?

Lembrando que esses repositórios estão armazenados na lista repositorios
Can you write Python code for it?


I am working on the file "01-python-intro/aula-01/Aula 01.ipynb".
The first task is:
Fazemos assim para verificar se um elemento faz parte de uma lista:
Can you write Python code for it?

'foobar' in lista

'abc' in lista

-0.5 in lista


I am working on the file "01-python-intro/aula-01/Aula 01.ipynb".
The first task is:
Imprima as URLs de todos os repositórios
Can you write Python code for it?


I am working on the file "01-python-intro/aula-01/Aula 01.ipynb".
The first task is:
Iterar uma lista é simples:
Can you write Python code for it?

for numero in numeros:
    print(numero)


I am working on the file "01-python-intro/aula-01/Aula 01.ipynb".
The first task is:
O atributo headers de response (response.headers) nos trouxe os cabeçalhos em forma de um dicionário (ou dict). O dict é uma estrutura de dados do Python utilizada para armazenar informações na forma de chave e valor envoltas por chaves {}. Dicts serão explicados mais para frente.
Por motivos técnicos a saída foi apresentada em um formato difícil de ler, para melhorá-la podemos fazer um pequeno workaround:
Can you write Python code for it?


I am working on the file "01-python-intro/aula-01/Aula 01.ipynb".
The first task is:
O for do Python, diferentemente de outras linguagens como C e Java, faz o controle dos índices internamente.
Vamos fazer mais um exemplo para deixar claro:
Can you write Python code for it?

numeros = range(1, 11)  # cria uma lista de números de 1 a 10
list(numeros)

for numero in numeros:
    print(numero ** 2)  # numero elevado a segunda potência


I am working on the file "01-python-intro/aula-01/Aula 01.ipynb".
The first task is:
Para acessar os elementos basta usar sua chave:
Can you write Python code for it?


I am working on the file "01-python-intro/aula-01/Aula 01.ipynb".
The first task is:
Para acessar somente as chaves fazemos:
Can you write Python code for it?


I am working on the file "01-python-intro/aula-01/Aula 01.ipynb".
The first task is:
Para alterar algum valor fazemos:
Can you write Python code for it?

notas['joao'] = 6.5
notas

notas['ana'] = 7.5
notas


I am working on the file "01-python-intro/aula-01/Aula 01.ipynb".
The first task is:
Para iterar tanto a chave quanto o valor use a função notas.items() como mostrado a seguir:
Can you write Python code for it?

for chave, valor in notas.items():
    print(chave, valor)


I am working on the file "01-python-intro/aula-01/Aula 01.ipynb".
The first task is:
Para os próximos exercícios será necessáro pegar os repositórios do usuário gvanrossum. Use a biblioteca requests e faça uma requição à API do github (https://api.github.com/) como demostrado anteriormente.
Can you write Python code for it?

response = ...
response.status_code  # status_code deve ser 200


I am working on the file "01-python-intro/aula-01/Aula 01.ipynb".
The first task is:
Para saber o tamanho de uma lista basta usar a função len():
Can you write Python code for it?

len(lista)

len(lista[2])  # o segundo elemento da lista é uma lista com 3 elementos!

len(numeros)


I am working on the file "01-python-intro/aula-01/Aula 01.ipynb".
The first task is:
Para ter uma lista contendo as chaves e valores:
Can you write Python code for it?


I am working on the file "01-python-intro/aula-01/Aula 01.ipynb".
The first task is:
Qual o link do perfil do dono dos repositórios?
Can you write Python code for it?


I am working on the file "01-python-intro/aula-01/Aula 01.ipynb".
The first task is:
Quantos forks o repositório gvanrossum/asyncio possui?
Can you write Python code for it?


I am working on the file "01-python-intro/aula-01/Aula 01.ipynb".
The first task is:
Também temos os dados do dono (owner) do repositório:
Can you write Python code for it?


I am working on the file "01-python-intro/aula-02/Aula 02.ipynb".
The first task is:
Além de índices o Python também suporta fatiamento (slicing). O fatiamento oferece uma maneira simples de acessar substrings:
Can you write Python code for it?

palavra[:2]  # pega os caracteres da posição 0 (incluído) até 2 (excluído)

palavra[4:]  # pega os caracteres da posição 4 (incluído) até o final

palavra[-2:]  # pega os caracteres da "segunda última" até o final


I am working on the file "01-python-intro/aula-02/Aula 02.ipynb".
The first task is:
Complete a função que corrige um número de celular digitado.

Se for do padrão antigo (8 caracteres) adicionar um 9 no começo.
Se o celular enviado já possuir 9 deverá verificar se o primeiro caracter é 9. Se não for troque-o para 9.
Se conter menos que 8 ou mais que 9 levante uma exceção.
Can you write Python code for it?

def corrige_celular(celular):
    if len(celular) == 8:
        ...  # corrige e retorna o celular
    elif len(celular) == 9:
        ...  # verifica se o celular está correto e retorna número corrigido se necessario
    else:
        raise ValueError('Celular "{0}" é inválido'.format(celular))

assert corrige_celular("733080280") == "933080280"
assert corrige_celular("820229289") == "920229289"
assert corrige_celular("634950619") == "934950619"
assert corrige_celular("620143587") == "920143587"
assert corrige_celular("27439428") == "927439428"
assert corrige_celular("11856634") == "911856634"


I am working on the file "01-python-intro/aula-02/Aula 02.ipynb".
The first task is:
Controle de fluxo
if...elif...else
Já foi mostrado em alguns exemplos anteriores o uso do if...elif...else, mas deixarei aqui alguns exemplos e explicações para não haver duvidas:
A função input em um script para o mesmo e espera a entrada de um usuário. Já no Jupyter Notebook essa função cria um campo de entrada
Can you write Python code for it?

nota = float(input())

if nota < 0 or nota > 10:
    raise ValueError('Nota {} inválida'.format(nota))  # levanta uma exceção do tipo ValueError
elif nota < 5:
    print('{}... reprovado'.format(nota))
elif 5 <= nota <= 10:  # o python permite comparação encadeada
    print('Passou com {}!'.format(nota))
elif nota == 10:
    print('Parabens pelo 10.')


I am working on the file "01-python-intro/aula-02/Aula 02.ipynb".
The first task is:
E também permitem operações aritméticas:
Can you write Python code for it?


I am working on the file "01-python-intro/aula-02/Aula 02.ipynb".
The first task is:
Exemplos
Calcular uma subsequência de Fibonacci até certo valor:
Can you write Python code for it?

a, b = 0, 1  # atribuição múltipla
while b < 100:
    print(b)
    a, b = b, a + b


I am working on the file "01-python-intro/aula-02/Aula 02.ipynb".
The first task is:
Exercicio
Crie o código que corrija as seguintes strings com o uso de substrings. (sem redefinir as strings):
Exemplo:
Can you write Python code for it?

string = 'Fafofa'  # Farofa
string[:2] + 'r' + string[3:]

string = 'Phython'  # Python
...  # digite o código aqui

string = 'Gambiara'  # Gambiarra
...  # digite o código aqui

string = 'Mundo Olá,'  # Olá, Mundo
...  # digite o código aqui


I am working on the file "01-python-intro/aula-02/Aula 02.ipynb".
The first task is:
Exercício 2
Implemente uma função que calcule os encargos trabalhistas dado um salário, sabendo que são descontados:
- 11% para o Imposto de Renda
- 8% para o INSS
- 5% para o sindicato
A função deve imprimir:

Salário bruto e líquido
Quanto foi pago INSS
Quanto foi pago ao sindicato
Can you write Python code for it?

def calcula_encargos(salario):
    imposto_renda = ...
    valor_INSS = ...
    valor_sindicato = ...
    salario_liquido = ...

    # não precisa mexer daqui para baixo!
    # A impressão dos dados calculados já está pronta! :-)
    FORMATO = (
        "+ Salário Bruto:   R$ {salario:.2f}\n"
        "- IR(11%)      :   R$ {imposto_renda:.2f}\n"
        "- INSS(8%)     :   R$ {valor_INSS:.2f}\n"
        "- Sindicato(5%):   R$ {valor_sindicato:.2f}\n"
        "= Salário Líquido: R$ {salario_liquido:.2f}\n"
    )
    print(FORMATO.format(salario=salario, imposto_renda=imposto_renda,
                         valor_INSS=valor_INSS, valor_sindicato=valor_sindicato,
                         salario_liquido=salario_liquido))

# imprime os encargos dos salário de 1000, 1500, 2000, ..., 5000
salarios = range(1000, 5001, 500)
for salario in salarios:
    calcula_encargos(salario)


I am working on the file "01-python-intro/aula-02/Aula 02.ipynb".
The first task is:
Exercícios
Exercício 1
Sabendo que a área de uma circunferência é calculada pela fórmula a seguir (sendo A a área e r o raio):
$a = \pi * r ^ 2 $
Complete a seguinte função (usando a biblioteca math) para calcular a circunferência de um círculo dado um raio:
Can you write Python code for it?


I am working on the file "01-python-intro/aula-02/Aula 02.ipynb".
The first task is:
Funcões úteis de strings
string.capitalize() - Transformar o primeiro caracter para caixa alta.
Can you write Python code for it?


I am working on the file "01-python-intro/aula-02/Aula 02.ipynb".
The first task is:
Geralmente essa notação é ativada automaticamente para números muito pequenos:
Can you write Python code for it?