I am working on the file "Db2 Jupyter Macros.ipynb".
The first task is:
If/Else/Endif Command
If you need to add conditional logic to your macro then you should use the if/else/endif commands. The format of the if statement is:
if variable condition value
   statements
else
   statements
endif
The else portion is optional, but the block must be closed with the endif command. If statements can be nested up to 9 levels deep:
if condition 1
   if condition 2
      statements
   else
      if condition 3
         statements
      end if
   endif
endif
If the condition in the if clause is true, then anything following the if statement will be executed and included in the final SQL statement. For instance, the following code will create a SQL statement based on the value of parameter 1:
if {1} = null
   SELECT * FROM EMPLOYEE
else
   SELECT {1} FROM EMPLOYEE
endif
Conditions
The if statement requires a condition to determine whether or not the block should be executed. The condition uses the following format:
if {variable} condition {variable} | constant | null
Variable can be a number from 1 to 9 which represents the argument in the macro list. So {1} refers to the first argument. The variable can also be the name of a named parameter or global variable.
The condition is one of the following comparison operators:
- =, ==: Equal to
- <: Less than
- >: Greater than
- <=,=<: Less than or equal to
- >=, =>: Greater than or equal to
- !=, <> : Not equal to
The variable or constant will have quotes stripped away before doing the comparison. If you are testing for the existence of a variable, or to check if a variable is empty, use the keyword null.
Can you write Python code for it?

%%sql macro showif
if {argc} = 0
   echo No parameters supplied
   if {option} <> null
      echo The optional parameter option was set: {option}
   endif
else
   if {argc} = "1"
       echo One parameter was supplied
   else
       echo More than one parameter was supplied: {argc}
   endif
endif


I am working on the file "Db2 Jupyter Macros.ipynb".
The first task is:
Macros can also have parameters supplied to them. The parameters are included after the name of the macro. Here is a simple macro which will use the first parameter as the name of the column we want returned from the EMPLOYEE table.
Can you write Python code for it?

%%sql macro emptable
SELECT {1} FROM EMPLOYEE


I am working on the file "Db2 Jupyter Macros.ipynb".
The first task is:
Named parameters are useful when there are many options within the macro and you don't want to keep track of which position it is in. In addition, if you have a variable number of parameters, you should use named parameters for the fixed (required) parameters and numbered parameters for the optional ones.
Macro Coding Overview
Macros can contain any type of text, including SQL commands. In addition to the text, macros can also contain the following keywords:

echo - Display a message
exit - Exit the macro immediately
if/else/endif - Conditional logic
var - Set a variable
display - Turn the display of the final text on

The only restriction with macros is that macros cannot be nested. This means I can't call a macro from within a macro. The sections below explain the use of each of these statement types.
Echo Option
The -e option will result in the final SQL being display after the macro substitution is done.
%%sql -e
%showemp(...)
Can you write Python code for it?

%%sql macro showdisplay
SELECT * FROM EMPLOYEE FETCH FIRST ROW ONLY


I am working on the file "Db2 Jupyter Macros.ipynb".
The first task is:
Now include the optional parameter.
Can you write Python code for it?

%sql %showif(option="Yes there is an option")


I am working on the file "Db2 Jupyter Macros.ipynb".
The first task is:
One additional option is available for variable substitution. If the first character of the variable name or parameter number is the ^ symbol, it will uppercase the entire string.
Can you write Python code for it?

%%sql macro showif
if {option} <> null
   echo The optional parameter option was set: {^option}
endif

%sql %showif(option="Yes there is an option")


I am working on the file "Db2 Jupyter Macros.ipynb".
The first task is:
The actual SQL that is generated is not shown by default. If you do want to see the SQL that gets generated, you can use the -e (echo) option to display the final SQL statement. The following example will display the generated SQL. Note that the echo setting is only used to display results for the current cell that is executing.
Can you write Python code for it?


I am working on the file "Db2 Jupyter Macros.ipynb".
The first task is:
The count of the total number of parameters passed is found in the {argc} variable. You can use this variable to decide whether or not the user has supplied the proper number of arguments or change which code should be executed.
Can you write Python code for it?

%%sql macro showvar
echo The number of unnamed parameters is {argc}. The where clause is *{where}*.


I am working on the file "Db2 Jupyter Macros.ipynb".
The first task is:
The echo command will show each line as a separate box.
Can you write Python code for it?


I am working on the file "Db2 Jupyter Macros.ipynb".
The first task is:
The name of the macro follows the %%sql macro command and is case sensitive. To use the macro, we can place it anywhere in the %sql block. This first example uses it by itself.
Can you write Python code for it?


I am working on the file "Db2 Jupyter Macros.ipynb".
The first task is:
The remainder of this notebook will explore the advanced features of macros.
Macro Parameters
Macros can have up to 9 parameters supplied to them. The parameters are numbered from 1 to 9, left to right in the argument list for the macro. For instance, the following macro has 5 paramters:
%emptable(lastname,firstnme,salary,bonus,'000010')
Parameters are separated by commas, and can contain strings as shown using single or double quotes. When the parameters are used within a macro, the quotes are not included as part of the string. If you do want to pass the quotes as part of the parameter, use square brackets [] around the string. For instance, the following parameter will not have quotes passed to the macro:
python
%sql %abc('no quotes')
To send the string with quotes, you could surround the parameter with other quotes "'hello'" or use the following technique if you use multiple quotes in your string:
python
%sql %abc (['quotes'])
To use a parameter within your macro, you enclose the parameter number with braces {}. The next command will illustrate the use of the five parameters.
Can you write Python code for it?

%%sql macro emptable
display on
SELECT {1},{2},{3},{4}
FROM EMPLOYEE
WHERE EMPNO = '{5}'


I am working on the file "Db2 Jupyter Macros.ipynb".
The first task is:
The string following the variable name can include quotes and these will not be removed. Only quotes that are supplied in a parameter to a macro will have the quotes removed.
Can you write Python code for it?

%%sql macro runit
var hello This is a long string without quotes
var hello2 'This is a long string with quotes'
echo {hello} <br> {hello2}

%sql %runit


I am working on the file "Db2 Jupyter Macros.ipynb".
The first task is:
This example illustrates two concepts. The MACRO command will replace any existing macro with the same name. Since we already have an emptable macro, the macro body will be replaced with this code. In addition, macros only exist for the duration of your notebook. If you create another Jupyter notebook, it will not contain any macros that you may have created. If there are macros that you want to share across notebooks, you should create a separate notebook and place all of the macro definitions in there. Then you can include these macros by executing the %run command using the name of the notebook that contains the macros.
The following SQL shows the use of the macro with parameters.
Can you write Python code for it?


I am working on the file "Db2 Jupyter Macros.ipynb".
The first task is:
Unnamed parameters are included in the count of arguments while named parameters are ignored.
Can you write Python code for it?


I am working on the file "Db2 Jupyter Macros.ipynb".
The first task is:
We could use the square brackets as an alternative way of passing the parameter.
Can you write Python code for it?


I am working on the file "Db2 Jupyter Macros.ipynb".
The first task is:
We just have to make sure that the quotes are part of the parameter now.
Can you write Python code for it?


I am working on the file "Db2 Jupyter Macros.ipynb".
The first task is:
When passing parameters to a macro, the program will automatically create variables based on whether they are positional parameters (1, 2, ..., n) or named parameters. The following macro will be used to show how parameters are passed to the routine.
Can you write Python code for it?

%%sql macro showvar
echo parm1={1}
parm2={2}
message={message}


I am working on the file "Db2 Jupyter Macros.ipynb".
The first task is:
You need to supply the quotes in the script or macro when using variables since quotes are stripped from any strings that are supplied.
Can you write Python code for it?

%%sql macro showvar
echo parm1={1}
parm2={2}
message='{message}'

%sql %showvar(1,2,message="Hello World")


I am working on the file "Db2 Regular Expressions.ipynb".
The first task is:
Adding the NOT modifier in front of the REGEXP function
gives us the stations that do not begin with West.
Can you write Python code for it?

%%sql
SELECT STATION FROM CENTRAL_LINE
  WHERE NOT REGEXP_LIKE(STATION,'^West')


I am working on the file "Db2 Regular Expressions.ipynb".
The first task is:
All of these numbers fit the pattern and should be valid.
When one of the capturing parenthesis matches, it will
remember the string that it matched. For instance, in the
first example (123-456-789), the first match will find the
string '123'. The second match will find '456' and so on. We
can refer to these matched strings with the special control
characters \n where n represents the capturing parenthesis
location. So \1 would refer to the '123' that was found. \2
would be for '456'.
The regular expression will be updated so that the last part
of the pattern needs to be numeric (0-9) but can't be same
as the first match.
Can you write Python code for it?


I am working on the file "Db2 Regular Expressions.ipynb".
The first task is:
Back to Top

Capturing Parenthesis
The previous example used something called a negative
lookaround with capturing parenthesis. When you place a
pattern within a set of brackets (...) the string that
matches this pattern is "remembered". The strings that are
matched can be used in subsequent parts of your regular
expression. This allows a form of programming within your
regular expression!
Each set of parentheses that are matched are associated with
a number, starting at one and incrementing for each
subsequent pattern match. For instance, the following
pattern will have three matches:

^([0-9]{3})-([0-9]{3})-([0-9]{3})$

This is similar to the SSN example used earlier on in this
section. The difference in this example is that each block
of numbers is exactly the same (3 digits). This pattern will
match any sequence of numbers in the format 123-456-789.
Can you write Python code for it?


I am working on the file "Db2 Regular Expressions.ipynb".
The first task is:
Back to Top

Combining Patterns
As we found out in the previous section, there is an OR
operator that you can use to select between two patterns.
How do you request that multiple patterns be present? First
we must understand how matching occurs when we have multiple
strings that need to be matched that have an unknown number
of characters between them.
For instance, how do we create a pattern that looks for
"ing" followed by "way" somewhere in the string? Regular
expression recognize the "." (period) character as matching
anything. Following the pattern you can add a modifier that
specifies how many times you want the pattern matched:

* - Match zero or more times
? - Match zero or one times
+ - Match one or more times
{m} - Match exactly m times
{m,} - Match as least a minimum of m times
{m,n} - Match at least a minimum of m times and no more than n times

The following regular expression searches for a pattern with
"ing" followed by any characters and then "way".
Can you write Python code for it?

%%sql
SELECT STATION FROM CENTRAL_LINE
  WHERE REGEXP_LIKE(STATION,'(ing)*.(way)')


I am working on the file "Db2 Regular Expressions.ipynb".
The first task is:
Back to Top

Matching patterns across multiple lines
So far the examples have dealt with strings that do not
contain newline characters (or carriage feeds). In some
applications, data from an input panel may include multiple
lines which may contain hard line feeds. What this means is
that there are actually multiple lines in the data, but from
a database perspective, there is only one line in the
VARCHAR field. You can modify the behavior of the Regular
Expression search by instructing it to honor the CRLF
characters as line delimiters.
The following SQL will insert a single line with multiple
CRLF characters in it to simulate a multi-line text string.
Can you write Python code for it?

%%sql -q
DROP TABLE LONGLINE;

CREATE TABLE LONGLINE (NAME VARCHAR(255));

INSERT INTO LONGLINE
  VALUES 'George' || CHR(10) || 'Katrina';


I am working on the file "Db2 Regular Expressions.ipynb".
The first task is:
Back to Top

Logical OR Operator
Regular expressions can match more than one pattern. The OR
operator (|) is used to define alternative patterns that can
match in a string. The following example searches for
stations that have "ing" in their name as well as "hill".
Can you write Python code for it?

%%sql
SELECT STATION FROM CENTRAL_LINE
  WHERE REGEXP_LIKE(STATION,'way|ing')


I am working on the file "Db2 Regular Expressions.ipynb".
The first task is:
Back to Top

Regular Expression Commands
There are six regular expression functions within DB2
including:

REGEXP_COUNT - Returns a count of the number of times that a regular expression pattern is matched in a string.
REGEXP_EXTRACT - Returns one occurrence of a substring of a string that matches the regular expression pattern.
REGEXP_INSTR - Returns the starting or ending position of the matched substring, depending on the value of the return option argument.
REGEXP_LIKE - Returns a Boolean value indicating if the regular expression pattern is found in a string. The function can be used only where a predicate is supported.
REGEXP_MATCH_COUNT - Returns a count of the number of times that a regular expression pattern is matched in a string.
REGEXP_REPLACE - Returns a modified version of the source string where occurrences of the regular expression pattern found in the source string are replaced with the specified replacement string.
REGEXP_SUBSTR - Returns one occurrence of a substring of a string that matches the regular expression pattern.

Each one of these functions follows a similar calling
sequence:

 REGEXP_FUNCTION(source, pattern, flags, start_pos, codeunits)

The arguments to the function are:

Source - string to be searched
Pattern - the regular expression that contains what we are searching for
Flag - settings that control how matching is done
Start_pos - where to start in the string
Codeunits - which type unit of measurement start_pos refers to (for Unicode)

The source can be any valid Db2 string including CHAR,
VARCHAR, CLOB, etc. Start_pos is the location in the source
string that you want to start searching from, and codeunits
tells Db2 whether to treat the start_pos as an absolute
location (think byte location) or a character location which
takes into account the unicode size of the character string.
Codeunits can be specified as CODEUNITS16, CODEUNITS32, or
OCTETS. CODEUNITS16 specifies that start is expressed in
16-bit UTF-16 code units. CODEUNITS32...
Can you write Python code for it?

%%sql
SELECT STATION FROM CENTRAL_LINE
  WHERE REGEXP_LIKE(STATION,'Ruislip')


I am working on the file "Db2 Regular Expressions.ipynb".
The first task is:
Back to Top

Special Patterns
The previous example used the [0-9] syntax to request that
only numbers be found in the pattern. There are some
predefined patterns that define these common patterns. The
first argument is Posix format (if it exists), the second is
the escape character equivalent, and the final one is the
raw pattern it represents.
|Posix      | Escape   | Pattern         | Meaning
|:----------| :--------| :---------------|:------------------------
|[:alnum:]  |          | [A-Za-z0-9]     | Alphanumeric characters
|           | \w       | [A-Za-z0-9_]    | Alphanumeric characters plus "_"
|           | \W       | [^A-Za-z0-9_]   | Non-word characters
|[:alpha:]  | \a       | [A-Za-z]        | Alphabetic characters
|[:blank:]  | \s, \t   |                 | Space and tab
|           | \b       |                 | Word boundaries
|[:cntrl:]  |          | [\x00-\x1F\x7F] | Control characters
|[:digit:]  | \d       | [0-9]           | Digits
|           | \D       | [^0-9]          | Non-digits
|[:graph:]  |          | [\x21-\x7E]     | Visible characters
|[:lower:]  | \l       | [a-z]           | Lowercase letters
|[:print:]  | \p       | [\x20-\x7E]     | Visible characters and the space character
|[:punct:]  |          | [][!"#\$%&'()*+,./:;<=>?@\^_`{|}~-] | Punctuation characters
|[:space:]  | \s       | [ \t\r\n\v\f]   | Whitespace characters
|           | \S       | [^ \t\r\n\v\f]  | Non-whitespace characters
|[:upper:]  | \u       | [A-Z]           | Uppercase letters
|[:xdigit:] | \x       | [A-Fa-f0-9]     | Hexadecimal digits
For instance, the following three statements will produce
the same result.
Can you write Python code for it?

%%sql -a
WITH SSNS(SSN) AS (
   VALUES
      '123-34-1322',
      'ABC-34-9999',
      'X123-44-0001',
      '123X-Y44-Z0001',
      '111-222-111'
  )
SELECT 'Original', SSN,
  CASE
    WHEN REGEXP_LIKE(SSN,'^[0-9]{3}-[0-9]{2}-[0-9]{4}$') THEN 'Valid'
    ELSE 'Invalid'
  END
FROM SSNS
UNION ALL
SELECT 'Posix', SSN,
  CASE
    WHEN REGEXP_LIKE(SSN,'^[:digit:]{3}-[:digit:]{2}-[:digit:]{4}$') THEN 'Valid'
    ELSE 'Invalid'
  END
FROM SSNS
UNION ALL
SELECT 'Escape', SSN,
  CASE
    WHEN REGEXP_LIKE(SSN,'^\d{3}-\d{2}-\d{4}$') THEN 'Valid'
    ELSE 'Invalid'
  END
FROM SSNS


I am working on the file "Db2 Regular Expressions.ipynb".
The first task is:
Back to our SSN question. Can a regular expression pattern
be used to determine whether or not the string is in the
correct format? The data will be in the format XXX-XX-XXXX
so the regular expression needs to find the three numeric
values separated by dashes.
The number pattern can be represented with the bracket
expression [0-9]. To specify the number of characters that
need to be found, we use the braces {} to specify the exact
number required.
For the three numbers in the pattern we can use [0-9]{3},
[0-9]{2}, and [0-9]{4}. Adding in the dashes gives us the
final pattern. The SQL below checks to see if a SSN is
correct.
Can you write Python code for it?


I am working on the file "Db2 Regular Expressions.ipynb".
The first task is:
Brackets can be used to make it clear what the pattern is
that you are searching for and avoid the problem of having
blanks in the expression. Brackets do have a specific usage
in regular expressions, but here we are using it only to
separate the two search strings.
Can you write Python code for it?

%%sql
SELECT STATION FROM CENTRAL_LINE
  WHERE REGEXP_LIKE(STATION,'(way)|(ing)')


I am working on the file "Db2 Regular Expressions.ipynb".
The first task is:
Every system will have different performance
characteristics, but the odds are that the LIKE statement
was faster than the regular expression. The performance is
probably ranked this way (with tx/sec being the measurement):
<pre>
LIKE > REGEXP(Non-capturing) > REGEXP(Capturing) > REGEXP(String)
</pre>

However, your system will probably get different results. So
while Regular expressions are very powerful, there is a
performance penalty when using them. In addition, you may
need to consider the use of indexes from a performance
perspective.
The following SQL will add an index to the STATION name.
Can you write Python code for it?

%%sql -q
DROP INDEX TEMP_STATION_INDEX;
CREATE INDEX TEMP_STATION_INDEX ON TEMP_LINE(STATION);


I am working on the file "Db2 Regular Expressions.ipynb".
The first task is:
Finding at least one occurrence of a pattern requires the
use of the + operator, or the bracket operators. This
example locates at least one occurrence of the "an" string
in station names.
Can you write Python code for it?

%%sql
SELECT STATION FROM CENTRAL_LINE
  WHERE REGEXP_LIKE(STATION,'(an)+')


I am working on the file "Db2 Regular Expressions.ipynb".
The first task is:
If we want to find an exact number of occurrences, we need
to use the {} notation to tell the regular expression
matcher how many we want to find. The syntax of the {}
match is:

{m} - Match exactly m times
{m,} - Match as least a minimum of m times
{m,n} - Match at least a minimum of m times and no more than n times

So the "+" symbol is equivalent to the following regular
expression using the {} syntax.
Can you write Python code for it?

%%sql
SELECT STATION FROM CENTRAL_LINE
  WHERE REGEXP_LIKE(STATION,'(an){1,}')


I am working on the file "Db2 Regular Expressions.ipynb".
The first task is:
If you didn't place the % at the beginning of the LIKE
string, only the stations that start with Ruislip would be
found.
Can you write Python code for it?

%%sql
SELECT STATION FROM CENTRAL_LINE
  WHERE STATION LIKE 'Ruislip%'


I am working on the file "Db2 Regular Expressions.ipynb".
The first task is:
If you want to match Ruislip with upper or lower case being
ignored, you would add the 'i' flag as part of the
REGEXP_LIKE (or any REGEXP function).
Can you write Python code for it?

%%sql
SELECT STATION FROM CENTRAL_LINE
  WHERE REGEXP_LIKE(STATION,'RUISLIP','i')


I am working on the file "Db2 Regular Expressions.ipynb".
The first task is:
If you wanted to include all stations that have the letter
P-R or p-e, you could add the condition within the brackets.
Can you write Python code for it?

%%sql
SELECT STATION FROM CENTRAL_LINE
   WHERE REGEXP_LIKE(STATION,'[p-rP-R]')


I am working on the file "Db2 Regular Expressions.ipynb".
The first task is:
In many cases it may be easier to find the patterns that
match and then negate the REGEXP statement! The (?...)
syntax is used for a variety of purposes in regular
expressions:
| Pattern   | Result
|:----------| :-----------
| (?: ... ) | Non-capturing parentheses. Groups the included pattern, but does not provide capturing of matching text. More efficient than capturing parentheses.
| (?> ... ) | Atomic-match parentheses. First match of the parenthesized subexpression is the only one tried. If it does not lead to an overall pattern match, back up the search for a match to a position before the "(?>"
| (?# ... ) | Free-format comment (?# comment )
| (?= ... ) | Look-ahead assertion. True if the parenthesized pattern matches at the current input position, but does not advance the input position.
| (?! ... ) | Negative look-ahead assertion. True if the parenthesized pattern does not match at the current input position. Does not advance the input position.
| (?<= ... )| Look-behind assertion. True if the parenthesized pattern matches text that precedes the current input position. The last character of the match is the input character just before the current position. Does not alter the input position. The length of possible strings that is matched by the look-behind pattern must not be unbounded (no * or + operators.)
| (?<!... ) | Negative Look-behind assertion. True if the parenthesized pattern does not match text that precedes preceding the current input position. The last character of the match is the input character just before the current position. Does not alter the input position. The length of possible strings that is matched by the look-behind pattern must not be unbounded (no * or + operators.)
For efficiency in matching, the best approach is to place
strings that you are searching for in non-capturing
parentheses (?:...) rather than the generic () parenthesis.
The following example finds all stations with "West" in the
name.
Can you write Python code for it?

%%sql
SELECT STATION FROM CENTRAL_LINE
  WHERE REGEXP_LIKE(STATION,'(West)')


I am working on the file "Db2 Regular Expressions.ipynb".
The first task is:
Sadly, we get no results! This would appear to be the wrong
result, but it's because we got lucky with our first search!
The best way to figure out what we matched in the original
query is to use the REGEXP_EXTRACT and REGEXP_INSTR
functions.

REGEXP_EXTRACT - Returns one occurrence of a substring of a string that matches the regular expression pattern.
REGEXP_INSTR - Returns the starting or ending position of the matched substring, depending on the value of the return option argument.

The following SQL gives us a clue to what was found with the
(an) pattern.
Can you write Python code for it?

%%sql
SELECT STATION,
   REGEXP_INSTR(STATION,'(an)') AS LOCATION,
   REGEXP_EXTRACT(STATION,'(an)') AS EXTRACT
FROM CENTRAL_LINE
   WHERE REGEXP_LIKE(STATION,'(an)')


I am working on the file "Db2 Regular Expressions.ipynb".
The first task is:
Searching for Katrina at the beginning and end of string
doesn't work.
Can you write Python code for it?

%%sql
SELECT COUNT(*) FROM LONGLINE
  WHERE REGEXP_LIKE(NAME,'^Katrina$')


I am working on the file "Db2 Regular Expressions.ipynb".
The first task is:
Some things to be aware of when creating the search pattern.
Spaces in the patterns themselves are significant. If the
previous search pattern had a space in one of the words, it
would not find it (unless of course there was a space in the
station name).
Can you write Python code for it?

%%sql
SELECT STATION FROM CENTRAL_LINE
  WHERE REGEXP_LIKE(STATION,'way| ing')


I am working on the file "Db2 Regular Expressions.ipynb".
The first task is:
The SSN is valid in the example above. Here are some other
examples to show whether or not the regular expression picks
up all of the errors.
Can you write Python code for it?


I am working on the file "Db2 Regular Expressions.ipynb".
The first task is:
The pattern 'Ruislip' will look for a match of Ruislip
within the STATION column. Note that this pattern will also
match 'West Ruislip' or 'Ruislip Gardens' since we placed no
restriction on where the pattern can be found in the string.
The match will also be exact (case matters). This type of
search would be equivalent to using the SQL LIKE statement:
Can you write Python code for it?

%%sql
SELECT STATION FROM CENTRAL_LINE
  WHERE STATION LIKE '%Ruislip%'


I am working on the file "Db2 Regular Expressions.ipynb".
The first task is:
The results are clearer when we plot them all on one graph!
Can you write Python code for it?


I am working on the file "Db2 Regular Expressions.ipynb".
The first task is:
To force an exact match with a string you would use both the
beginning and end anchors.
Can you write Python code for it?

%%sql
  SELECT STATION FROM CENTRAL_LINE
    WHERE REGEXP_LIKE(STATION,'^Leyton$')


I am working on the file "Db2 Regular Expressions.ipynb".
The first task is:
To match a pattern at the end of the string, the dollar sign
$ can be used.
Can you write Python code for it?

%%sql
  SELECT STATION FROM CENTRAL_LINE
    WHERE REGEXP_LIKE(STATION,'Ruislip$')


I am working on the file "Db2 Regular Expressions.ipynb".
The first task is:
We can override the regular expression search by telling it
to treat each NL/CRLF as the end of a string within a
string.
Can you write Python code for it?

%%sql
SELECT COUNT(*) FROM LONGLINE
  WHERE REGEXP_LIKE(NAME,'^Katrina$','m')


I am working on the file "Db2 Regular Expressions.ipynb".
The first task is:
We will run four queries in the following SQL.

Search for West using the LIKE statement
Search for West using a regular expression with no matching ()
Search for West using a regular expression with matching ()
Search for West using a regular expression with non-capturing parenthesis (?:)

The queries are run as many times as possible per second to get consistent results. Odds are that there will be variations in the results on your system!
Can you write Python code for it?

results_like         = %sql -t SELECT COUNT(*) FROM TEMP_LINE WHERE STATION LIKE '%West%'
results_string       = %sql -t SELECT COUNT(*) FROM TEMP_LINE WHERE REGEXP_LIKE(STATION,'West')
results_capturing    = %sql -t SELECT COUNT(*) FROM TEMP_LINE WHERE REGEXP_LIKE(STATION,'(West)')
results_noncapturing = %sql -t SELECT COUNT(*) FROM TEMP_LINE WHERE REGEXP_LIKE(STATION,'(?:West)')


I am working on the file "Db2 Regular Expressions.ipynb".
The first task is:
We will try the 4 queries again and plot the result.
Can you write Python code for it?

results_like         = %sql -t SELECT COUNT(*) FROM TEMP_LINE WHERE STATION LIKE '%West%'
results_string       = %sql -t SELECT COUNT(*) FROM TEMP_LINE WHERE REGEXP_LIKE(STATION,'West')
results_capturing    = %sql -t SELECT COUNT(*) FROM TEMP_LINE WHERE REGEXP_LIKE(STATION,'(West)')
results_noncapturing = %sql -t SELECT COUNT(*) FROM TEMP_LINE WHERE REGEXP_LIKE(STATION,'(?:West)')
%sql -q DROP TABLE RESULTS
%sql -q CREATE TABLE RESULTS(TYPE VARCHAR(16), RESULT DEC(9,2))
%sql -q INSERT INTO RESULTS VALUES ('LIKE', {results_like} ), \
  ('REGX STRING', {results_string} ), \
  ('REGX CAPTURE', {results_capturing} ), \
  ('REGX NONCAPTURE', {results_noncapturing} )
%sql SELECT * FROM RESULTS
%sql -pb SELECT * FROM RESULTS


I am working on the file "Db2 Regular Expressions.ipynb".
The first task is:
You can also negate some of the searches in a pattern by
using the [^...] syntax where the ^ tells the regular
expression not to match the following characters. The
expression [^0-9] would mean match any characters which are
not numeric.
However, regular expressions have something called negative
lookarounds which basically mean find the pattern which does
not match. You create this pattern by adding the (?!..) at
the beginning of the string. The same query (finding
stations that don't start with West) would be written with
this lookaround logic found in the SQL below.
Can you write Python code for it?

%%sql
SELECT STATION FROM CENTRAL_LINE
  WHERE REGEXP_LIKE(STATION,'^(?!West)')


I am working on the file "Db2 Regular Expressions.ipynb".
The first task is:
You should find that two stations match the pattern. The
following SQL shows which pattern is matched first in the
STATIONS names.
Can you write Python code for it?

%%sql
SELECT STATION,
   REGEXP_INSTR(STATION,'((an).*){2}') AS LOCATION,
   REGEXP_EXTRACT(STATION,'((an).*){2}') AS EXTRACT
FROM CENTRAL_LINE
   WHERE REGEXP_LIKE(STATION,'((an).*){2}')


I am working on the file "Db2 Row and Column Access Control.ipynb".
The first task is:
A DBA (Fred) who is not part of any of these groups will not be able to see any of the records, even though they may performance maintenance on the table itself.
Can you write Python code for it?

%%sql
SET HC.SESSION_USER = 'FRED';

SELECT * FROM HC.PATIENTS;


I am working on the file "Db2 Row and Column Access Control.ipynb".
The first task is:
Catalog Views
The information about the ROW and COLUMN permissions can be found in the SYSCAT.CONTROLS table. One line is found in this table for each column permission and one for the row permission on a table.
Can you write Python code for it?


I am working on the file "Db2 Row and Column Access Control.ipynb".
The first task is:
Changing the current user to one of the accounting, research, or fund raising users will result in all records being displayed.
Can you write Python code for it?

%%sql
SET HC.SESSION_USER = 'JOHN';

SELECT * FROM HC.PATIENTS;


I am working on the file "Db2 Row and Column Access Control.ipynb".
The first task is:
Column Masks
Column access control is implemented in the form of a mask, or lack thereof, on the data.
Using our Health-care scenario as the base, we will implement column access control/rules in two forms:

We MASK the account balance column. Only the ACCOUNTING team can see the account balance in the table and all others see a balance of zero.

We MASK the SIN column (Social Insurance Number column). Only the PATIENT themselves can see the full Social Insurance number and all others see only the last three digits of the number.

This first MASK is on account balance and sets it to zero for everyone except for those people in accounting.
Can you write Python code for it?

%%sql
CREATE OR REPLACE MASK HC.ACCT_BALANCE_MASK ON HC.PATIENTS FOR
  COLUMN ACCT_BALANCE RETURN
     CASE
        WHEN HC.VERIFY_ROLE_FOR_USER(HC.SESSION_USER,'ACCOUNTING') = 1
           THEN ACCT_BALANCE
        ELSE 0.00
     END
  ENABLE;


I am working on the file "Db2 Row and Column Access Control.ipynb".
The first task is:
Create some ROLES for people in the Healthcare scenario.
Can you write Python code for it?

%%sql
INSERT INTO HC.ROLES
  VALUES
    ('LEE','PCP'),('JAMES','PCP'),
    ('MAX','PATIENT'),('MIKE','PATIENT'),('SAM','PATIENT'),
    ('DOUG','PATIENT'),('BOB','PATIENT'),
    ('JOHN','ACCOUNTING'),
    ('TOM','MEMBERSHIP'),
    ('JANE','RESEARCH'),
    ('FRED','DBA');


I am working on the file "Db2 Row and Column Access Control.ipynb".
The first task is:
Dr. Lee decides to give Sam some codeine for his pain. The update is successful and we can see the results.
Can you write Python code for it?

%%sql
UPDATE HC.PATIENTS SET PHARMACY = 'Codeine' WHERE NAME = 'Sam';

SELECT * FROM HC.PATIENTS WHERE NAME = 'Sam'


I am working on the file "Db2 Row and Column Access Control.ipynb".
The first task is:
Finally, the patients will be able to see their own SIN field, but the account balance will show as zero (presumably so they don't get sick over the amount!).
Can you write Python code for it?

%%sql
SET HC.SESSION_USER = 'BOB';

SELECT * FROM HC.PATIENTS;


I am working on the file "Db2 Row and Column Access Control.ipynb".
The first task is:
Health Care Scenario
We will use a sample scenario based on the Health-care industry.
Data in a hospital needs to be provided to a number of individuals,including other organizations like insurance companies, clearing house etc. Each of these companies have their own employees, and there are government regulations in place dictating who can see what data or parts of patient data.
As a result security policies must be in place to ensure the rules are being met.
Across the scenario there will be different groups of individuals that access the database and there must be security in place to ensure each group or individual only sees the data they are allowed to see.
This SQL will create the base PATIENT table and populate it with information.
Can you write Python code for it?

%%sql -q
DROP TABLE HC.PATIENTS;

CREATE TABLE HC.PATIENTS
  (
  SIN      VARCHAR(11),
  USERID   VARCHAR(8),
  NAME     VARCHAR(8),
  ADDRESS  VARCHAR(12),
  PHARMACY VARCHAR(12),
  ACCT_BALANCE DEC(9,2),
  PCP_ID   VARCHAR(8)
  );

INSERT INTO HC.PATIENTS VALUES
  ('123 551 234','MAX','Max','First St.','hypertension',89.7,'LEE'),
  ('123 589 812','MIKE','Mike','Long St.','diabetics',8.3,'JAMES'),
  ('123 119 856','SAM','Sam','Big St.','aspirin',12.5,'LEE'),
  ('123 191 454','DOUG','Doug','Good St.','influenza',7.68,'JAMES'),
  ('123 456 789','BOB','Bob','123 Some St.','hypertension',9,'LEE');

SELECT * FROM HC.PATIENTS;


I am working on the file "Db2 Row and Column Access Control.ipynb".
The first task is:
Selecting Rows from a Table
The rule with Row and Column access control is that if you can read (select) the data then you can potentially update it. Here is a select statement issued by Dr. Lee and you can see that only patients assigned to him are displayed.
Can you write Python code for it?

%%sql
SET HC.SESSION_USER = 'LEE';

SELECT * FROM HC.PATIENTS;


I am working on the file "Db2 Row and Column Access Control.ipynb".
The first task is:
The HC.VERIFY_ROLE_FOR_USER function will mimic what the VERIFY_ROLE_FOR_USER
function does by returning a value of 1 if the user has that particular ROLE.
Can you write Python code for it?

%%sql -d
CREATE OR REPLACE FUNCTION
  HC.VERIFY_ROLE_FOR_USER(UID VARCHAR(8), IN_ROLE VARCHAR(10))
    SECURED NO EXTERNAL ACTION DETERMINISTIC
    RETURNS INT
    BEGIN ATOMIC
      RETURN
       SELECT COUNT(*) FROM HC.ROLES H
         WHERE H.USERID = UID AND H.ROLE = IN_ROLE;
    END@


I am working on the file "Db2 Row and Column Access Control.ipynb".
The first task is:
The rules now need to be activated in order for them to be enforced.
Can you write Python code for it?

%sql ALTER TABLE HC.PATIENTS ACTIVATE ROW ACCESS CONTROL;


I am working on the file "Db2 Row and Column Access Control.ipynb".
The first task is:
The second mask will return the entire SIN number for the PATIENT, but only the last three digits of the SIN for all others.
Can you write Python code for it?

%%sql
CREATE OR REPLACE MASK HC.SIN_MASK ON HC.PATIENTS FOR
  COLUMN SIN RETURN
     CASE
        WHEN HC.VERIFY_ROLE_FOR_USER(HC.SESSION_USER,'PATIENT') = 1
          THEN SIN
        ELSE
         'XXX XXX ' || SUBSTR(SIN,9,3)
     END
  ENABLE;


I am working on the file "Db2 Row and Column Access Control.ipynb".
The first task is:
Update Failure
If a doctor is not assigned to a patient, they will not be able to update their
record. The rule is that if you can't read (select) the record then you can't
update it.
Can you write Python code for it?

%%sql
SET HC.SESSION_USER = 'LEE';

UPDATE HC.PATIENTS SET PHARMACY = 'Codeine' WHERE NAME = 'Doug';


I am working on the file "Db2 Row and Column Access Control.ipynb".
The first task is:
Updating a Patient Record
Dr. Lee needs to update the medication that his patient Sam is taking.
First we examine the original record in the table.
Can you write Python code for it?

%%sql
SET HC.SESSION_USER = 'LEE';

SELECT * FROM HC.PATIENTS WHERE NAME = 'Sam';


I am working on the file "Db2 Row and Column Access Control.ipynb".
The first task is:
We also need to create a SESSION_USER global variable that can be used to identify the "current" user. Normally you would just use the SESSION_USER variable in the rules, but since we don't want to use SECADM we need to fake the userid.
Can you write Python code for it?

%sql CREATE OR REPLACE VARIABLE HC.SESSION_USER VARCHAR(8);


I am working on the file "Db2 Row and Column Access Control.ipynb".
The first task is:
We populate the database with the EMPLOYEE and DEPARTMENT tables so that we can run the various examples.
Can you write Python code for it?


I am working on the file "Db2 Row and Column Access Control.ipynb".
The first task is:
When a researcher looks at the data, they will also only see the last three digits of the SIN field, but they will get a zero balance in the accounting field.
Can you write Python code for it?

%%sql
SET HC.SESSION_USER = 'JANE';

SELECT * FROM HC.PATIENTS;


I am working on the file "Db2 Row and Column Access Control.ipynb".
The first task is:
When someone from accounting now views the records, they will only see the last three digits of the SIN field but they will see all of the accounting data.
Can you write Python code for it?


I am working on the file "Db2 Statistical Functions.ipynb".
The first task is:

PERCENTILE_CONT
This is a function that gives you a continuous percentile calculation.
Can you write Python code for it?

%%sql
SELECT PERCENTILE_CONT(0.75) WITHIN GROUP (ORDER BY SALARY)
  FROM EMPLOYEE
WHERE WORKDEPT = 'E21'


I am working on the file "Db2 Statistical Functions.ipynb".
The first task is:

CUME_DIST
The CUME_DIST column function returns the cumulative distribution of a row that is hypothetically inserted into
a group of rows.
Can you write Python code for it?

%%sql
SELECT CUME_DIST(47000) WITHIN GROUP (ORDER BY SALARY)
  FROM EMPLOYEE
WHERE WORKDEPT = 'A00'


I am working on the file "Db2 Statistical Functions.ipynb".
The first task is:

PERCENTILE_DISC
The PERCENTILE_DISC/CONT returns the value that corresponds to the specified percentile
given a sort specification by using discrete (DISC) or continuous (CONT) distribution.
Can you write Python code for it?

%%sql
SELECT PERCENTILE_DISC(0.75) WITHIN GROUP (ORDER BY SALARY)
  FROM EMPLOYEE
WHERE WORKDEPT = 'E21'


I am working on the file "Db2 Statistical Functions.ipynb".
The first task is:

PERCENT_RANK
The PERCENT_RANK column function returns the relative percentile rank of a
row that is hypothetically inserted into a group of rows.
Can you write Python code for it?

%%sql
SELECT PERCENT_RANK(47000) WITHIN GROUP (ORDER BY SALARY)
  FROM EMPLOYEE
WHERE WORKDEPT = 'A00'


I am working on the file "Db2 Statistical Functions.ipynb".
The first task is:
And here is a plot of the data to make sense of the histogram.
Can you write Python code for it?

%%sql -pb
WITH BUCKETS(EMPNO, SALARY, BNO) AS
  (
  SELECT EMPNO, SALARY,
    WIDTH_BUCKET(SALARY, 35000, 100000, 9) AS BUCKET
  FROM EMPLOYEE ORDER BY EMPNO
  )
SELECT BNO, COUNT(*) AS COUNT FROM BUCKETS
GROUP BY BNO
ORDER BY BNO ASC


I am working on the file "Db2 Statistical Functions.ipynb".
The first task is:
We can plot this information by adding some more details to the bucket output.
Can you write Python code for it?

%%sql -a
WITH BUCKETS(EMPNO, SALARY, BNO) AS
  (
  SELECT EMPNO, SALARY,
    WIDTH_BUCKET(SALARY, 35000, 100000, 9) AS BUCKET
  FROM EMPLOYEE ORDER BY EMPNO
  )
SELECT BNO, COUNT(*) AS COUNT FROM BUCKETS
GROUP BY BNO
ORDER BY BNO ASC


I am working on the file "Db2 Statistical Functions.ipynb".
The first task is:
We populate the database with the EMPLOYEE and DEPARTMENT tables so that we can run the various examples.
Can you write Python code for it?


I am working on the file "Db2 Time and Date Functions.ipynb".
The first task is:
At this point in time(!) we can start inserting records into our table. We have already set the timezone
to be Toronto, so the next insert statement will take the current time (NOW) and insert it into the table.
For reference, here is the current time.
Can you write Python code for it?


I am working on the file "Db2 Time and Date Functions.ipynb".
The first task is:
Back to Top

Extracting Weeks, Months, Quarters, and Years from a Date
There are four functions that extract different values from a date. These functions include:

THIS_QUARTER - returns the first day of the quarter
THIS_WEEK - returns the first day of the week (Sunday is considered the first day of that week)
THIS_MONTH - returns the first day of the month
THIS_YEAR - returns the first day of the year
Can you write Python code for it?

%%sql
WITH DATES(FUNCTION, RESULT) AS
  (
  VALUES
    ('CURRENT DATE   ',NOW),
    ('THIS_WEEK      ',THIS_WEEK(NOW)),
    ('THIS_MONTH     ',THIS_MONTH(NOW)),
    ('THIS_QUARTER   ',THIS_QUARTER(NOW)),
    ('THIS_YEAR      ',THIS_YEAR(NOW))
  )
SELECT * FROM DATES


I am working on the file "Db2 Time and Date Functions.ipynb".
The first task is:
Back to Top

MONTHS_BETWEEN Function
You may have noticed that the MONTHS_BETWEEN function was not in the previous list of functions. The
reason for this is that the value returned for MONTHS_BETWEEN is different from the other functions. The MONTHS_BETWEEN
function returns a DECIMAL value rather than an integer value. The reason for this is that the duration of a
month is not as precise as a day, week or year. The following example will show how the duration is
a decimal value rather than an integer. You could always truncate the value if you want an integer.
Can you write Python code for it?

%%sql
WITH DATES(FUNCTION, RESULT) AS (
  VALUES
    ('0 MONTH        ',MONTHS_BETWEEN(NOW, NOW)),
    ('1 MONTH        ',MONTHS_BETWEEN(NOW + 1 MONTH, NOW)),
    ('1 MONTH + 1 DAY',MONTHS_BETWEEN(NOW + 1 MONTH + 1 DAY, NOW)),
    ('LEAP YEAR      ',MONTHS_BETWEEN('2016-02-01','2016-03-01')),
    ('NON-LEAP YEAR  ',MONTHS_BETWEEN('2015-02-01','2015-03-01'))
  )
SELECT * FROM DATES


I am working on the file "Db2 Time and Date Functions.ipynb".
The first task is:
Back to Top

Extracting Specfic Days from a Month
There are three functions that retrieve day information from a date. These functions include:

DAYOFMONTH - returns an integer between 1 and 31 that represents the day of the argument
FIRST_DAY - returns a date or timestamp that represents the first day of the month of the argument
DAYS_TO_END_OF_MONTH - returns the number of days to the end of the month

This is the current date so that you know what all of the calculations are based on.
Can you write Python code for it?


I am working on the file "Db2 Time and Date Functions.ipynb".
The first task is:
Back to Top

Next Day Function
The previous set of functions returned a date value for the current week, month, quarter, or year (or the next one
if you used the NEXT function). The NEXT_DAY function returns the next day (after the date you supply)
based on the string representation of the day. The date string will be dependent on the codepage that you are using for the database.
The date (from an English perspective) can be:
|Day       |Short form
|:-------- |:---------
|Monday    |MON
|Tuesday   |TUE
|Wednesday |WED
|Thursday  |THU
|Friday    |FRI
|Saturday  |SAT
|Sunday    |SUN
The following SQL will show you the "day" after the current date that is Monday through Sunday.
Can you write Python code for it?

%%sql
WITH DATES(FUNCTION, RESULT) AS
  (
  VALUES
    ('CURRENT DATE   ',NOW),
    ('Monday         ',NEXT_DAY(NOW,'Monday')),
    ('Tuesday        ',NEXT_DAY(NOW,'TUE')),
    ('Wednesday      ',NEXT_DAY(NOW,'Wednesday')),
    ('Thursday       ',NEXT_DAY(NOW,'Thursday')),
    ('Friday         ',NEXT_DAY(NOW,'FRI')),
    ('Saturday       ',NEXT_DAY(NOW,'Saturday')),
    ('Sunday         ',NEXT_DAY(NOW,'Sunday'))
  )
SELECT * FROM DATES


I am working on the file "Db2 Time and Date Functions.ipynb".
The first task is:
Back to Top

DATE_PART Function
DATE_PART is similar to the EXTRACT function but it uses the more familiar syntax:
Python
DATE_PART(element, expression)
In the case of the function, the element must be placed in quotes, rather than as a keyword in the EXTRACT function. in addition, the DATE_PART always returns a BIGINT, while the EXTRACT function will return a different data type depending on the element being returned. For instance, compare the SECONDs option for both functions. In the case of EXTRACT you get a DECIMAL result while for the DATE_PART you get a truncated BIGINT.
Can you write Python code for it?

%%sql -a
WITH DATES(FUNCTION, RESULT) AS (
  VALUES
    ('EPOCH',              DATE_PART('EPOCH'          ,NOW )),
    ('MILLENNIUM(S)',      DATE_PART('MILLENNIUM'     ,NOW )),
    ('CENTURY(CENTURIES)', DATE_PART('CENTURY'        ,NOW )),
    ('DECADE(S)',          DATE_PART('DECADE'         ,NOW )),
    ('YEAR(S)',            DATE_PART('YEAR'           ,NOW )),
    ('QUARTER',            DATE_PART('QUARTER'        ,NOW )),
    ('MONTH',              DATE_PART('MONTH'          ,NOW )),
    ('WEEK',               DATE_PART('WEEK'           ,NOW )),
    ('DAY(S)',             DATE_PART('DAY'            ,NOW )),
    ('DOW',                DATE_PART('DOW'            ,NOW )),
    ('DOY',                DATE_PART('DOY'            ,NOW )),
    ('HOUR(S)',            DATE_PART('HOURS'          ,NOW )),
    ('MINUTE(S)',          DATE_PART('MINUTES'        ,NOW )),
    ('SECOND(S)',          DATE_PART('SECONDS'        ,NOW )),
    ('MILLISECOND(S)',     DATE_PART('MILLISECONDS'   ,NOW )),
    ('MICROSECOND(S)',     DATE_PART('MICROSECONDS'   ,NOW ))
  )
SELECT FUNCTION, CAST(RESULT AS BIGINT) FROM DATES;


I am working on the file "Db2 Time and Date Functions.ipynb".
The first task is:
Back to Top

DATE_TRUNC Function
DATE_TRUNC computes the same results as the DATE_PART function but then truncates the value down. Note that not all values can be truncated. The function syntax is:
Python
DATE_TRUNC(element, expression)
The element must be placed in quotes, rather than as a keyword in the EXTRACT function.
Note that DATE_TRUNC always returns a BIGINT.
The elements that can be truncated are:
|Element Name       |Description
|:----------------  |:------------------------------------------------------------------------------
|MILLENNIUM(S)      |The millennium is to be returned.
|CENTURY(CENTURIES) |The number of full 100-year periods represented by the year.
|DECADE(S)          |The number of full 10-year periods represented by the year.
|YEAR(S)            |The year portion is to be returned.
|QUARTER            |The quarter of the year (1 - 4) is to be returned.
|MONTH              |The month portion is to be returned.
|WEEK               |The number of the week of the year (1 - 53) that the specified day is to be returned.
|DAY(S)             |The day portion is to be returned.
|HOUR(S)            |The hour portion is to be returned.
|MINUTE(S)          |The minute portion is to be returned.
|SECOND(S)          |The second portion is to be returned.
|MILLISECOND(S)     |The second of the minute, including fractional parts to one thousandth of a second
|MICROSECOND(S)     |The second of the minute, including fractional parts to one millionth of a secondry data types.
Can you write Python code for it?

%%sql -a
WITH DATES(FUNCTION, RESULT) AS (
  VALUES
    ('MILLENNIUM(S)',      DATE_TRUNC('MILLENNIUM'     ,NOW )),
    ('CENTURY(CENTURIES)', DATE_TRUNC('CENTURY'        ,NOW )),
    ('DECADE(S)',          DATE_TRUNC('DECADE'         ,NOW )),
    ('YEAR(S)',            DATE_TRUNC('YEAR'           ,NOW )),
    ('QUARTER',            DATE_TRUNC('QUARTER'        ,NOW )),
    ('MONTH',              DATE_TRUNC('MONTH'          ,NOW )),
    ('WEEK',               DATE_TRUNC('WEEK'           ,NOW )),
    ('DAY(S)',             DATE_TRUNC('DAY'            ,NOW )),
    ('HOUR(S)',            DATE_TRUNC('HOURS'          ,NOW )),
    ('MINUTE(S)',          DATE_TRUNC('MINUTES'        ,NOW )),
    ('SECOND(S)',          DATE_TRUNC('SECONDS'        ,NOW )),
    ('MILLISECOND(S)',     DATE_TRUNC('MILLISECONDS'   ,NOW )),
    ('MICROSECOND(S)',     DATE_TRUNC('MICROSECONDS'   ,NOW ))
  )
SELECT FUNCTION, RESULT FROM DATES


I am working on the file "Db2 Time and Date Functions.ipynb".
The first task is:
Convert the Coordinated Universal Time timestamp '2014-11-02 06:55:00' to the 'America/Toronto' time zone.
The following returns a TIMESTAMP with the value '2014-11-02 01:55:00'.
Can you write Python code for it?

%%sql
VALUES FROM_UTC_TIMESTAMP(TIMESTAMP'2014-11-02 06:55:00', 'America/Toronto');


I am working on the file "Db2 Time and Date Functions.ipynb".
The first task is:
Convert the Coordinated Universal Time timestamp '2015-03-02 06:05:00' to the 'America/Toronto'
time zone. The following returns a TIMESTAMP with the value '2015-03-02 01:05:00'.
Can you write Python code for it?

%%sql
VALUES FROM_UTC_TIMESTAMP(TIMESTAMP'2015-03-02 06:05:00', 'America/Toronto');


I am working on the file "Db2 Time and Date Functions.ipynb".
The first task is:
Convert the timestamp '1970-01-01 00:00:00' to the Coordinated Universal Time timezone from the 'America/Denver'
timezone. The following returns a TIMESTAMP with the value '1970-01-01 07:00:00'.
Can you write Python code for it?

%%sql
VALUES TO_UTC_TIMESTAMP(TIMESTAMP'1970-01-01 00:00:00', 'America/Denver');


I am working on the file "Db2 Time and Date Functions.ipynb".
The first task is:
FIRST_DAY will return the first day of the month. You could probably compute this with standard SQL date functions, but it is a lot easier just to use this builtin function.
Can you write Python code for it?


I am working on the file "Db2 Time and Date Functions.ipynb".
The first task is:
Finally, DAYS_TO_END_OF_MOTNH will return the number of days to the end of the month. A Zero would be returned if you are on the last day of the month.
Can you write Python code for it?


I am working on the file "Db2 Time and Date Functions.ipynb".
The first task is:
If the first date range is extended by one day then the range will overlap.
Can you write Python code for it?

%%sql
VALUES
  CASE
    WHEN
      (NOW, NOW + 2 DAYS) OVERLAPS (NOW + 1 DAY, NOW + 2 DAYS) THEN 'Overlaps'
    ELSE
      'No Overlap'
  END


I am working on the file "Db2 Time and Date Functions.ipynb".
The first task is:
In order to retrieve the value of the current timezone, we take advantage of a simple user-defined function
called GET_TIMEZONE. It just retrieves the contents of the current TIME_ZONE variable that we set up.
Can you write Python code for it?

%%sql
CREATE OR REPLACE FUNCTION GET_TIMEZONE()
  RETURNS VARCHAR(255)
LANGUAGE SQL CONTAINS SQL
  RETURN (TIME_ZONE)


I am working on the file "Db2 Time and Date Functions.ipynb".
The first task is:
Note that the timsstamp appears to be the same as what we insert (plus or minus a few seconds). What actually
sits in the base table is the UTC time.
Can you write Python code for it?


I am working on the file "Db2 Time and Date Functions.ipynb".
The first task is:
So what happens if we insert a record into the table now that we are in Vancouver?
Can you write Python code for it?

%%sql
INSERT INTO TXS VALUES(2,2,NOW);
SELECT * FROM TXS;


I am working on the file "Db2 Time and Date Functions.ipynb".
The first task is:
Table of Contents

Extract Function
DATE_PART Function
DATE_TRUNC Function
Extracting Specific Days from a Month
Date Addition
Extracting Weeks, Months, Quarters, and Years
Next Day Function
Between Date/Time Functions
Months Between
Date Duration
Overlaps Predicate
UTC Time Conversions

Back to Top

Extract Function
The EXTRACT function extracts and element from a date/time value. The syntax of the EXTRACT command is:
Python
EXTRACT( element FROM expression )
This is a slightly different format from most functions that you see in the DB2. Element must be one of the following values:
|Element Name      | Description
|:---------------- | :-----------------------------------------------------------------------------------------
|EPOCH             | Number of seconds since 1970-01-01 00:00:00.00. The value can be positive or negative.
|MILLENNIUM(S)     | The millennium is to be returned.
|CENTURY(CENTURIES)| The number of full 100-year periods represented by the year.
|DECADE(S)         | The number of full 10-year periods represented by the year.
|YEAR(S)           | The year portion is to be returned.
|QUARTER           | The quarter of the year (1 - 4) is to be returned.
|MONTH             | The month portion is to be returned.
|WEEK              | The number of the week of the year (1 - 53) that the specified day is to be returned.
|DAY(S)            | The day portion is to be returned.
|DOW               | The day of the week that is to be returned. Note that "1" represents Sunday.
|DOY               | The day (1 - 366) of the year that is to be returned.
|HOUR(S)           | The hour portion is to be returned.
|MINUTE(S)         | The minute portion is to be returned.
|SECOND(S)         | The second portion is to be returned.
|MILLISECOND(S)    | The second of the minute, including fractional parts to one thousandth of a second
|MICROSECOND(S)    | The second of the minute, including fractional parts to one millionth of a second
The synonym NOW is going to be used in the next...
Can you write Python code for it?


I am working on the file "Db2 Time and Date Functions.ipynb".
The first task is:
The SET Command can be used to update the TIME_ZONE to the current location we are in.
Can you write Python code for it?


I am working on the file "Db2 Time and Date Functions.ipynb".
The first task is:
The TXS view is used by all SQL statements rather than the TXS_BASE table. The reason for this is to
take advantage of INSTEAD OF triggers that can manipulate the UTC without modifying the original SQL.
Note that when the data is returned from the view that the TXTIME field is converted from UTC to the current
TIMEZONE that we are in.
Can you write Python code for it?

%%sql
CREATE OR REPLACE VIEW TXS AS
  (
  SELECT
     ID,
     CUSTID,
     FROM_UTC_TIMESTAMP(TXTIME_UTC,GET_TIMEZONE()) AS TXTIME
  FROM
     TXS_BASE
  )


I am working on the file "Db2 Time and Date Functions.ipynb".
The first task is:
The YMD_BETWEEN function is similar to the AGE function except that it takes two date arguments. We can
simulate the AGE function by supplying the NOW function to the YMD_BETWEEN function.
Can you write Python code for it?

%%sql
WITH DATES(FUNCTION, RESULT) AS (
  VALUES
    ('1 DAY                   ',YMD_BETWEEN(NOW,NOW - 1 DAY)),
    ('1 MONTH                 ',YMD_BETWEEN(NOW,NOW - 1 MONTH)),
    ('1 YEAR                  ',YMD_BETWEEN(NOW,NOW - 1 YEAR)),
    ('1 DAY + 1 MONTH         ',YMD_BETWEEN(NOW,NOW - 1 DAY - 1 MONTH)),
    ('1 DAY + 1 YEAR          ',YMD_BETWEEN(NOW,NOW - 1 DAY - 1 YEAR)),
    ('1 DAY + 1 MONTH + 1 YEAR',YMD_BETWEEN(NOW,NOW - 1 DAY - 1 MONTH - 1 YEAR))
  )
SELECT * FROM DATES


I am working on the file "Db2 Time and Date Functions.ipynb".
The first task is:
The data retrieved reflects the fact that we are now in Vancouver from an application perspective. Looking at the
base table and you will see that everything has been converted to UTC time.
Can you write Python code for it?


I am working on the file "Db2 Time and Date Functions.ipynb".
The first task is:
This expression (DAYOFMONTH) returns the day of the month.
Can you write Python code for it?


I am working on the file "Db2 Time and Date Functions.ipynb".
The first task is:
We can modify the time that is returned to us by changing our local timezone. The statement will make
the system think we are in Vancouver.
Can you write Python code for it?

%sql SET TIME_ZONE = 'America/Vancouver'


I am working on the file "Db2 Using Prepared Statements.ipynb".
The first task is:
Autocommit and Commit Scope
By default, any SQL statements executed with the %sql magic command are immediately commited. This means that the log file has the transaction details and the results are committed to disk. In other words, you can't change your mind after the statement finishes execution.
This behavior is often referred to as AUTOCOMMIT and adds a level of overhead to statement execution because at the end of every statement the results must be "hardened". On the other hand, autocommit means you don't have to worry about explicitly committing work or causing potential locking issues because you are holding up resources. When a record is updated, no other user will be able to view it (unless using "dirty read") until you commit. Holding the resource in a lock means that other workloads may come to a halt while they wait for you to commit your work.
Here is a classic example of wanting a commit scope that is based on a series of statements:
withdrawal = 100
%sql update checking set balance = balance - withdrawal
%sql update savings set balance = balance + withdrawal
If autocommit is ON, you could have a problem with the transaction if the system failed after the first update statement. You would have taken money out of the checking account, but have not updated the savings account. To make sure that this transaction is run successfully:
%sql autocommit off
withdrawal = 100
%sql update checking set balance = balance - withdrawal
%sql update savings set balance = balance + withdrawal
%sql commit work
If the transaction fails before the COMMIT WORK, all changes to the database will be rolled back to its original state, thus protecting the integrity of the two tables.
AUTOCOMMIT
Autocommit can be turned on or off using the following syntax:
%sql AUTOCOMMIT ON | OFF
If you turn AUTOCOMMIT OFF then you need to make sure that you COMMIT work at the end of your code. If you don't there is possible you lose your work if the connection is lost to Db2.
COMMIT, ROLLBACK
To COMMIT all changes to the database...
Can you write Python code for it?

%sql -sampledata
employees = %sql -r select * from employee


I am working on the file "Db2 Using Prepared Statements.ipynb".
The first task is:
Loop with INSERT Statements
One could always use SQL to insert into this table, but we will use a loop to execute insert statements. The loop will be timed so we can get a sense of the cost of running this code. In order to make the loop run a longer the insert block is run 50 times.
Can you write Python code for it?

%sql -q DELETE FROM EMPLOYEE2
print("Starting Insert")
start_time = time.time()
i = 0
for k in range(0,50):
    for record in employees[1:]:
        i += 1
        empno,firstnme,midinit,lastname,workdept,phoneno,hiredate,job,edlevel,sex,birthdate,salary,bonus,comm = record
        %sql -q INSERT INTO EMPLOYEE2 VALUES ( \
            :empno,:firstnme,:midinit, \
            :lastname,:workdept,:phoneno, \
            :hiredate,:job,:edlevel, \
            :sex,:birthdate,:salary, \
            :bonus,:comm)
end_time = time.time()
print('Total load time for {:d} records is {:.2f} seconds'.format(i,end_time-start_time))
time_insert = end_time-start_time


I am working on the file "Db2 Using Prepared Statements.ipynb".
The first task is:
Loop with PREPARE Statement, Arrays and AUTOCOMMIT OFF
Finally, we can turn AUTOCOMMIT off and then commit the work at the end of the block to improve the total time required to insert the data. Note the use of the parameter shortform ?*14 in the code.
Can you write Python code for it?

%sql -q DELETE FROM EMPLOYEE2
%sql autocommit off
print("Starting Insert")
start_time = time.time()
i = 0
prep = %sql prepare INSERT INTO EMPLOYEE2 VALUES (?*14)
for k in range(0,50):
    for record in employees[1:]:
        i += 1
        %sql execute :prep using :record
%sql commit work
end_time = time.time()
print('Total load time for {:d} records is {:.2f} seconds'.format(i,end_time-start_time))
%sql autocommit on
time_commit = end_time-start_time


I am working on the file "Db2 Using Prepared Statements.ipynb".
The first task is:
Performance Comparison
You may have noticed that the performance of the last method is substantially faster than the other examples. The primary reason for this is the COMMIT only occuring at the end of the code.
Can you write Python code for it?

%%sql -pb
WITH RESULT(RUN,ELAPSED) AS (
  VALUES
    ('INSERT',CAST(:time_insert AS DEC(5,2))),
    ('PREPARE',CAST(:time_prepare AS DEC(5,2))),
    ('ARRAY ',CAST(:time_array AS DEC(5,2))),
    ('COMMIT ',CAST(:time_commit AS DEC(5,2)))
)
SELECT RUN, ELAPSED FROM RESULT
  ORDER BY ELAPSED DESC


I am working on the file "Db2 Using Prepared Statements.ipynb".
The first task is:
The employees variable contains all of the employee data as a Python array. The next statement will retrieve the contents of the first row only (Remember that row 0 contains the name of the columns).
Can you write Python code for it?


I am working on the file "Db2 Using Prepared Statements.ipynb".
The first task is:
We now will create another EMPLOYEE table that is an exact duplicate of what we already have.
Can you write Python code for it?

%%sql -q
DROP TABLE EMPLOYEE2;
CREATE TABLE EMPLOYEE2 AS (SELECT * FROM EMPLOYEE) DEFINITION ONLY;


I am working on the file "Deboguer_du_code_Python_depuis_Notebook_Jupyter.ipynb".
The first task is:
Ca ne marche pas chez moi, ça affiche des codes couleurs ANSI non interprétés (16/02/2021).
Avec le nouveau débuggueur visuel de Jupyter Lab - new!
TODO!

https://github.com/jupyterlab/debugger
See https://blog.jupyter.org/a-visual-debugger-for-jupyter-914e61716559
And https://stackoverflow.com/a/61120586/5889533
Can you write Python code for it?

def print_endline(i: int) -> None:
    print(1.0 / i)

def main() -> None:
    for i in range(10, -10, -1):
        print_endline(i)

main()


I am working on the file "December 05, 2016/Covered in class.ipynb".
The first task is:
Combining two or more dictionaries dictionaries
Can you write Python code for it?

from pprint import pprint

basement_rooms = {'den': 300, 'living_room': 500}
ground_level_rooms = {'kitchen': 250, 'guest_bed_room_one': 300}
upstairs_rooms = {'master_bed_room': 400, 'guest_bath': 150}

upper_floors = upstairs_rooms.copy()
upper_floors.update(ground_level_rooms)

print('Upper level rooms have')
pprint(upper_floors)
print()

entire_house = upper_floors.copy()
entire_house.update(basement_rooms)

print('All rooms in house')
pprint(entire_house)

print()
for room, square_feet in entire_house.items():
    print(room, ':', square_feet)
    print()


from pprint import pprint

basement_rooms = {'den': 300, 'living_room': 500}
ground_level_rooms = {'kitchen': 250, 'guest_bed_room_one': 300}
upstairs_rooms = {'master_bed_room': 400, 'guest_bath': 150}

upper_floors = upstairs_rooms.copy()
upper_floors.update(ground_level_rooms)

## Note no change if I go and add a room to the ground_level_rooms after creating upper_floors
ground_level_rooms['bathroom'] = 200

print('Ground level rooms')
pprint(ground_level_rooms)
print()

print('Upper level rooms have')
pprint(upper_floors)

big_rooms = {'living': 500, 'kitchen': 700}
small_rooms = {'living': 200}

my_rooms = big_rooms.copy()

print(my_rooms)

my_rooms.update(small_rooms)

print(my_rooms)


I am working on the file "December 05, 2016/Covered in class.ipynb".
The first task is:
Example with Chores, People, Rooms, and cleaning
This example will use lists and dictionaries to setup who chould do which chores in a home.
Key Concepts

Lists
Dictionaries
Iteration
Nested loop avoidance strategies via dictionaires

Room setup
Can you write Python code for it?

from pprint import pprint

chores = ['sweeping', 'vacuuming', 'dusting', 'windows']
people = ['kid_one', 'kid_two', 'adult_one', 'adult_two']
rooms = [
    'kitchen', 'living_room', 'dining_room', 'bedroom_one',
    'bedroom_two', 'master_bedroom', 'bedroom_three', 'basement',
    'attic'
]

rooms_with_size = {
    'kitchen': 400,
    'living_room': 200,
    'dining_room': 250,
    'bedroom_one': 400,
    'bedroom_two': 300,
    'master_bedroom': 350,
    'bedroom_three': 200,
    'basement': 800,
    'attic': 475
}

# Assign the chores to each room
rooms_with_chores = {}
for room in rooms:
    rooms_with_chores[room] = chores.copy()

pprint(rooms_with_chores)


I am working on the file "December 05, 2016/Covered in class.ipynb".
The first task is:
Figure out how much each person cleans -- takes more CPU time
Can you write Python code for it?

from pprint import pprint
from random import choice

chores = ['sweeping', 'vacuuming', 'dusting', 'windows']
people = ['kid_one', 'kid_two', 'adult_one', 'adult_two']
rooms = [
    'kitchen', 'living_room', 'dining_room', 'bedroom_one',
    'bedroom_two', 'master_bedroom', 'bedroom_three', 'basement',
    'attic'
]

rooms_with_size = {
    'kitchen': 400,
    'living_room': 200,
    'dining_room': 250,
    'bedroom_one': 400,
    'bedroom_two': 300,
    'master_bedroom': 350,
    'bedroom_three': 200,
    'basement': 800,
    'attic': 475
}

# Assign the chores to each room
rooms_with_chores = {}
for room in rooms:
    rooms_with_chores[room] = chores.copy()

# Assign rooms to people
people_with_rooms = {}
rooms_to_clean = rooms.copy()

while len(rooms_to_clean) > 0:

    # Randomly assign a person to a chore
    person = choice(people)
    room = choice(rooms_to_clean)

    # Need to make sure we only add rooms to the list of rooms
    # if the person has already been given at least one room
    if person in people_with_rooms:
        people_with_rooms[person].append(room)
    else:
        people_with_rooms[person] = [room]

    # This ensures we stop our loop at some point
    rooms_to_clean.remove(room)

# Ensure everyone has something to clean, even if it's no rooms at all
for person in people:
    if person not in people_with_rooms:
        people_with_rooms[person] = []

# Less efficient way to
amount_cleaned_by_person = {}
for person, rooms in people_with_rooms.items():

    amount_cleaned_by_person[person] = 0

    for room in rooms:
        amount_cleaned_by_person[person] += rooms_with_size[room]

pprint(amount_cleaned_by_person)


I am working on the file "December 05, 2016/Covered in class.ipynb".
The first task is:
Number Fruits in a dictionary -- Deep Copy
Can you write Python code for it?

from copy import deepcopy
from pprint import pprint

number_of_fruits = {
    'apples': 10,
    'pears': 15,
    'plums': 4
}

number_of_vegetables = {
    'carrots': 10,
    'celery': 3,
    'sprouts': 15
}

food_box_one = {
    'fruit': number_of_fruits.copy(),
    'vegetables': number_of_vegetables.copy()
}

food_box_two = deepcopy(food_box_one)

# Updating the fruits inside the food_box_one updates it inside of food_box_two
food_box_one['fruit']['apples'] = 25

print('Food box one')
pprint(food_box_one)

print()
print('Food box two')
pprint(food_box_two)


I am working on the file "December 05, 2016/Covered in class.ipynb".
The first task is:
Room and chore setup
Can you write Python code for it?

from pprint import pprint
from random import choice

chores = ['sweeping', 'vacuuming', 'dusting', 'windows']
people = ['kid_one', 'kid_two', 'adult_one', 'adult_two']
rooms = [
    'kitchen', 'living_room', 'dining_room', 'bedroom_one',
    'bedroom_two', 'master_bedroom', 'bedroom_three', 'basement',
    'attic'
]

rooms_with_size = {
    'kitchen': 400,
    'living_room': 200,
    'dining_room': 250,
    'bedroom_one': 400,
    'bedroom_two': 300,
    'master_bedroom': 350,
    'bedroom_three': 200,
    'basement': 800,
    'attic': 475
}

# Assign the chores to each room
rooms_with_chores = {}
for room in rooms:
    rooms_with_chores[room] = chores.copy()

# Assign rooms to people
people_with_rooms = {}
rooms_to_clean = rooms.copy()

while len(rooms_to_clean) > 0:

    # Randomly assign a person to a chore
    person = choice(people)
    room = choice(rooms_to_clean)

    # Need to make sure we only add rooms to the list of rooms
    # if the person has already been given at least one room
    if person in people_with_rooms:
        people_with_rooms[person].append(room)
    else:
        people_with_rooms[person] = [room]

    # This ensures we stop our loop at some point
    rooms_to_clean.remove(room)

# Ensure everyone has something to clean, even if it's no rooms at all
for person in people:
    if person not in people_with_rooms:
        people_with_rooms[person] = []

# Show which rooms are assigned to which people
pprint(people_with_rooms)


I am working on the file "December 12, 2016/Covered in class.ipynb".
The first task is:
Input using a file
Can you write Python code for it?

class FileReadStream(ReadStream):
    """
    Used to read data from a URL
    """

    def read(self):
        return open(self.source, 'r').read()

# This works because there is a file within this directory called `smimple_text.txt`
file_read_stream_instance = FileReadStream('simple_text.txt')

file_read_stream_instance.print_read()


I am working on the file "December 12, 2016/Covered in class.ipynb".
The first task is:
Input with a URL
Can you write Python code for it?

import requests # This is a library used to make HTTP web requests

class UrlReadStream(ReadStream):
    """
    Used to read data from a URL
    """

    def read(self):
        return requests.get(self.source).text

url_read_stream_instance = UrlReadStream('https://goo.gl/r7BDx9')

url_read_stream_instance.print_read()


I am working on the file "December 12, 2016/Covered in class.ipynb".
The first task is:
Using Read Streams
Can you write Python code for it?

class ReadStream:
    """
    Base class for reading data.
    """
    def __init__(self, source=None):
        self.source = source

    def read(self):
        raise NotImplementedError('The sub class must implement the `read` method')

    def print_read(self):
        print(self.read())

read_stream_instance = ReadStream('some data')

read_stream_instance.print_read()


I am working on the file "December 12, 2016/Covered in class.ipynb".
The first task is:
Using Write Streams
Can you write Python code for it?

class WriteStream:
    """
    Base class for writing data.
    """
    def __init__(self, source_data, destination=None):
        self.source_data = source_data
        self.destination = destination

    def write(self):
        raise NotImplementedError('Subclass must implement `write` method')

write_stream_instance = WriteStream('some_source_data', 'sample_output_file.txt')

write_stream_instance.write()


I am working on the file "December 12, 2016/Covered in class.ipynb".
The first task is:
Using standard input as the read
Can you write Python code for it?

class InputReadStream(ReadStream):
    """
    Used to read data from standard input
    """
    def read(self):
        return input(self.source)

input_read_stream_instance = InputReadStream('How many dogs do you want? ')

input_read_stream_instance.print_read()


I am working on the file "December 12, 2016/Covered in class.ipynb".
The first task is:
Writing to a file
Can you write Python code for it?

class FileWriteStream(WriteStream):
    """
    Used to write data to standard out
    """
    def write(self):
        file = open(self.destination, 'w')
        file.write(self.source_data)
        print('Wrote to file: {}'.format(self.destination))

output_write_stream_instance = FileWriteStream('some other data that I wrote', 'sample_output_file.txt')

output_write_stream_instance.write()

## Read the sample output file

file_name = 'sample_output_file.txt'

sample_output_file = FileReadStream(file_name)

sample_output_file.print_read()


I am working on the file "December 15, 2016/Covered in class.ipynb".
The first task is:
Explicit is better than implicit
Can you write Python code for it?

fruits = ['apple', 'pear', 'cranberry']

# Good
for fruit in fruits:
    print(fruit)

# Bad
for i in fruits:
    print(i)

# okay
for index in range(10):
    print(index)

# Questionable
for i in range(10):
    for j in range(11):
        for k in range(11):
            pass


I am working on the file "December 15, 2016/Covered in class.ipynb".
The first task is:
Simple is better than complex
Can you write Python code for it?

class Person:

    def __init__(self, name):
        self.name = name
        self.is_hungry = True
        self.is_tired = False

    def __str__(self):
        return '{name} {id}'.format(
            name=self.name,
            id=id(self)
        )

    def fed(self, big_meal=False):
        self.is_hungry = False

        if big_meal:
            self.is_tired = True

    def sleep(self):
        self.is_hungry = True
        self.is_tired = False

bill = Person('Bill')
sammy = Person('Sammy')
jim = Person('Jim')
sue = Person('Sue')

persons = [bill, sammy, jim, sue]

jim.fed(True)
sue.fed(True)

# Bad check -- because it's complex and hides logic
if persons[0].is_hungry and persons[1].is_hungry and persons[2].is_tired and persons[3].is_tired:
    for person in persons:
        print('I have a person', person)

print()
print()
# Better check -- it doesn't hide meaning, but is verbose
def all_are_hungry(persons):
    for person in persons:
        if not person.is_hungry:
            return False
    return True

def all_are_tired(persons):
    for person in persons:
        if not person.is_tired:
            return False
    return True

hungry_people_group = persons[:2]
tired_people = persons[2:]

if all_are_hungry(hungry_people_group) and all_are_tired(tired_people):
    for person in persons:
        print('I have a person', person)

print()
print()

# Good/Best check -- doesn't hide meaning, and isn't verbose
has_hungry = all([person.is_hungry for person in persons[:2]])
has_tired = all([person.is_tired for person in persons[2:]])

if has_hungry and has_tired:
    for person in persons:
        print('I have a person', person)

print(range(10))


I am working on the file "Decision Trees.ipynb".
The first task is:
Decision Tree Mini Project
Can you write Python code for it?

from email_preprocess import preprocess

### features_train and features_test are the features for the training
### and testing datasets, respectively
### labels_train and labels_test are the corresponding item labels
features_train, features_test, labels_train, labels_test = preprocess()


I am working on the file "Decision Trees.ipynb".
The first task is:
Part 1: Get the Decision Tree Running
Get the decision tree up and running as a classifier, setting min_samples_split=40.   It will probably take a while to train.  What’s the accuracy?
Answer
Accuracy of the classifier is 97.90%
Can you write Python code for it?

from sklearn import tree

def submitAcc():
    return clf.score(features_test, labels_test)

clf = tree.DecisionTreeClassifier(min_samples_split=40)
t0 = time()
clf.fit(features_train, labels_train)
print("done in %0.3fs" % (time() - t0))

pred = clf.predict(features_test)
print "Classifier with accurancy %.2f%%" % (submitAcc() * 100)


I am working on the file "Decision Trees.ipynb".
The first task is:
This decision tree seems to be overfitting the data. Lets plot the training points instead of the test points to explore what is going on.
Can you write Python code for it?


I am working on the file "Decision Trees.ipynb".
The first task is:
Tweak parameters
We will tweak min_sample_split parameter so we can adjust our classifier to not overfit. This gives us the minimum number of samples that must be in a population inorder to split it.
Can you write Python code for it?

clf = tree.DecisionTreeClassifier(min_samples_split=50)
t0 = time()
clf.fit(features_train, labels_train)
print("done in %0.3fs" % (time() - t0))

pred = clf.predict(features_test)
print "Classifier with accurancy %.2f%%" % (submitAcc() * 100)
prettyPicture(clf, features_test, labels_test)


I am working on the file "Decision Trees.ipynb".
The first task is:
What do you think SelectPercentile is doing?  Would a large value for percentile lead to a more complex or less complex decision tree, all other things being equal?
Answer

I think a large value for percentile would lead to a more complex decision tree.

Note the difference in training time depending on the number of features.
Answer

training time went down significantly with less features.

What’s the accuracy when percentile = 1?
Answer

Accuracy of the prediction with percentile = 1 is 96.70%
Can you write Python code for it?


I am working on the file "Decision-Tree_Learning.ipynb".
The first task is:
Decision-Tree Learning
Decision tree learning uses a decision tree as a predictive model observations about an item (represented in the branches) to conclusions about the item's target value (represented in the leaves). It is one of the predictive modelling approaches used in statistics, data mining and machine learning. Tree models where the target variable can take a finite set of values are called classification trees; in these tree structures, leaves represent class labels and branches represent conjunctions of features that lead to those class labels. Decision trees where the target variable can take continuous values (typically real numbers) are called regression trees.
Source
Entropy
$$
E(S) = - \sum\limits_{i=1}^{n} p_1 \cdot \log_2(p_i)
$$
Can you write Python code for it?

def entropy(S):

    outcomes = pd.unique(S[final])
    ent = lambda p: 0 if p == 0 else p * log2(p)

    return -sum([  ent( S[S[final] == o].size / S.size ) for o in outcomes ])


I am working on the file "Decision-Tree_Learning.ipynb".
The first task is:
Gini-Index
$$
    \mbox{Gini}(S) = 1- \sum\limits_i p_i^2
$$
$$
    \mbox{Gini}(S, A)  = \sum\limits_i \frac{|s_i|}{|S|} \cdot \mbox{Gini}(S)
$$
Can you write Python code for it?

def gini(S, A=None):

    if A == None:
        return 1-sum(
            [(S[S[final] == o].size / S.size)**2 for o in  pd.unique(S[final])]
        )

    return sum( [ ( S[A][S[A] == p].size / S[A].size ) * gini(S[S[A] == p]) for p in pd.unique(S[A])]  )

_exec = lambda f : {col: f(data, col) for col in data.columns if not col == final}

data = pd.read_csv('playgolf.txt')
final='Play Golf?'

out = pd.DataFrame( dict(
    gain = _exec(gain),
    information = _exec(information),
    gain_ratio = _exec(gain_ratio),
    gini = _exec(gini)
) )
out

_ = out.plot(kind='bar')

data = pd.read_csv('lens24.dat')
final = 'class'

out = pd.DataFrame( dict(
    gain = _exec(gain),
    information = _exec(information),
    gain_ratio = _exec(gain_ratio),
    gini = _exec(gini)
) )
out

_ = out.plot(kind='bar')


I am working on the file "Decision-Tree_Learning.ipynb".
The first task is:
Information Gain
$$
\mbox{Gain}(S, A) = E(S) - I(S,A)
$$
Can you write Python code for it?

def gain(S, A):
    return entropy(S) - information(S, A)


I am working on the file "Decision-Tree_Learning.ipynb".
The first task is:
Instrinsic Information
$$
\mbox{IntI}(S,A) = - \sum\limits_i \frac{|S_i|}{|S|} \cdot \log_2(\frac{|S_i|}{|S|})
$$
Can you write Python code for it?

def intrinsic_information(S, A):
    partitions = pd.unique(S[A])

    return -sum([( S[A][S[A] == p].size / S[A].size ) *
                log2( S[A][S[A] == p].size / S[A].size )  for p in partitions])


I am working on the file "Decision-Trees-For-Knowledge-Discovery-With-Scikit-Learn.ipynb".
The first task is:
Conclusion
To sum up, we managed to get good classification results and to be able to explain those results visually and automatically. Note that it would have been possible to solve a regression problem with the same algorithm, such as predicting a price rather than a category.
Such a technique can be useful in reverse engineering an existing system, such as an old one that has been coded in a peculiar programming language and for which the employees who coded it have left. This technique can also be used for data mining, gaining business intelligence, and insights from data.
In the case that your data does not represent a pure function like we have here, such as if for two of your input examples it is possible to have two possible different predictions, then a tree cannot model the data set with 100% accuracy. Hopefully, if you are in that situation where the logic behind the data is not perfect, it is possible to repeat the experiment by using XGBoost, which can help by incrementally training many trees to reduce the error and training them in an optimized fashion. The only disadvantage of that is that those boosted forests would be harder to explain due to the fact that you would have many trees.
Can you write Python code for it?

# Let's convert this notebook to a README for the GitHub project's title page:
!jupyter nbconvert --to markdown Decision-Trees-For-Knowledge-Discovery-With-Scikit-Learn.ipynb
!mv Decision-Trees-For-Knowledge-Discovery-With-Scikit-Learn.md README.md


I am working on the file "Decision-Trees-For-Knowledge-Discovery-With-Scikit-Learn.ipynb".
The first task is:
Define the features and preprocess the car evaluation data set
We'll preprocess the attributes into redundant features, such as using an integer index (linear) to represent a value for an attribute, as well as also using a one-hot encoding for each attribute's possible values as new features. Despite the fact that this is redundant, this will help to make the tree smaller since it has more choice on how to split data on each branch.
Can you write Python code for it?

# The integer values for features will take
# a range from 0 to n-1 in the lists of possible values:
input_labels = [
    ["buying", ["vhigh", "high", "med", "low"]],
    ["maint", ["vhigh", "high", "med", "low"]],
    ["doors", ["2", "3", "4", "5more"]],  # Here indexes are not real values
    ["persons", ["2", "4", "more"]],
    ["lug_boot", ["small", "med", "big"]],
    ["safety", ["low", "med", "high"]],
]

class_names = ["unacc", "acc", "good", "vgood"]

# Load data set
data = np.genfromtxt(os.path.join('data', 'car.data'), delimiter=',', dtype="U")
data_inputs = data[:, :-1]
data_outputs = data[:, -1]

def str_data_to_one_hot(data, input_labels):
    """Convert each feature's string to a flattened one-hot array. """
    X_int = LabelEncoder().fit_transform(data.ravel()).reshape(*data.shape)
    X_bin = OneHotEncoder().fit_transform(X_int).toarray()

    feature_names = []
    for a in input_labels:
        key = a[0]
        for b in a[1]:
            value = b
            feature_names.append("{}_is_{}".format(key, value))

    return X_bin, feature_names

def str_data_to_linear(data, input_labels):
    """Convert each feature's string to an integer index"""
    X_lin = np.array([[
        input_labels[a][1].index(j) for a, j in enumerate(i)
    ] for i in data])

    # Integer feature indexes will range
    # from 0 to n-1 from indexes in the label list:
    feature_names = [i[0] + "_index" for i in input_labels]

    return X_lin, feature_names

# Take both one-hot and linear versions of input features:
X_one_hot, feature_names_one_hot = str_data_to_one_hot(data_inputs, input_labels)
X_linear_int, feature_names_linear_int = str_data_to_linear(data_inputs, input_labels)

# Put that together:
X = np.concatenate([X_one_hot, X_linear_int], axis=-1)
feature_names = feature_names_one_hot + feature_names_linear_int

# Outputs use indexes, this is not one-hot:
integer_y = np.array([class_names.index(i) for i in data_outputs])

print("Data set's shape,")
print("X.shape, integer_y.shape,...


I am working on the file "Decision-Trees-For-Knowledge-Discovery-With-Scikit-Learn.ipynb".
The first task is:
Note that the tree below can also be viewed here online.

Plot the importance of each input features of the simple tree:
Note here that it is the feature importance according to our simple, shallow tree. A fully complex trees would surely include more of the features/attributes, and with different proportions.
Can you write Python code for it?

def feature_importance_chart(clf, classifier_name, feature_names):
    sorted_feature_importances, sorted_feature_names = (
        zip(*sorted(zip(clf.feature_importances_, feature_names)))
    )
    plt.figure(figsize=(16, 9))
    plt.barh(range(len(sorted_feature_importances)), sorted_feature_importances)
    plt.yticks(
        range(len(sorted_feature_importances)),
        ["{}: {:.3}".format(a, b) for a, b in zip(sorted_feature_names, sorted_feature_importances)]
    )
    plt.title("The Gini feature importance for the {} \n"
              "(total decrease in node impurity, weighted by the "
              "probability of reaching that node)".format(classifier_name))
    plt.show()

feature_importance_chart(clf, "simple tree", feature_names)


I am working on the file "Decision-Trees-For-Knowledge-Discovery-With-Scikit-Learn.ipynb".
The first task is:
Note that the tree below can also be viewed here online. It would also be possible to extract the tree as true code and create a function.

Finally, the full feature importance:
Can you write Python code for it?

feature_importance_chart(clf, "complex tree", feature_names)


I am working on the file "Decision-Trees-For-Knowledge-Discovery-With-Scikit-Learn.ipynb".
The first task is:
Train a simple decision tree to fit the data set:
First, let's define some hyperparameters, such as the depth of the tree.
Can you write Python code for it?

max_depth = 6
clf = tree.DecisionTreeClassifier(max_depth=max_depth)
clf = clf.fit(X, integer_y)

print("Decision tree trained!")
accuracy = clf.score(X, integer_y)
print("Errors:", 100 - accuracy * 100, "%")
print("Accuracy:", accuracy * 100, "%")


I am working on the file "Decision-Trees-For-Knowledge-Discovery-With-XGBoost.ipynb".
The first task is:
In our case, note that it is possible to have an error of 0 (thus an accuracy of 100%) since we have a dataset that represents a function, which is mathematically deterministic and could be interpreted as programmatically pure in the case it would be implemented. But wait... we just implemented and recreated the function that was used to model the dataset with our trees! We don't need cross validation nor a test set, because our training data already covers the full feature space (attribute space).
Finally, the full attributes/features importance:
Can you write Python code for it?

# Some plot options from the doc:
# importance_type : str, default "weight"
#     How the importance is calculated: either "weight", "gain", or "cover"
#     "weight" is the number of times a feature appears in a tree
#     "gain" is the average gain of splits which use the feature
#     "cover" is the average coverage of splits which use the feature
#         where coverage is defined as the number of samples affected by the split

importance_types = ["weight", "gain", "cover"]
for i in importance_types:
    print("Importance type:", i)
    fig, ax = plt.subplots(figsize=(12, 7))
    xgb.plot_importance(bst, importance_type=i, ax=ax)
    plt.show()


I am working on the file "Decision-Trees-For-Knowledge-Discovery-With-XGBoost.ipynb".
The first task is:
Let's now generate slightly more complex trees to aid inspection

Let's go deeper and build deeper trees. However, those trees are not maximally complex since XGBoost is rather built to boost over forests of small trees than a big one.
Can you write Python code for it?

num_rounds = 1  # Do not use boosting for now, we want only 1 decision tree per class.
num_classes = len(output_labels)
num_trees = num_rounds * num_classes

# Let's use a max_depth of 4 for the sole goal of simplifying the visual representation produced
# (ideally, a tree would be deeper to classify perfectly on that dataset)
param = {
    'max_depth': 9,
    'objective': 'multi:softprob',
    'num_class': num_classes
}

bst = xgb.train(param, dtrain, num_boost_round=num_rounds)
print("Decision trees trained!")
print("Mean Error Rate:", bst.eval(dtrain))
print("Accuracy:", (bst.predict(dtrain).argmax(axis=-1) == integer_y).mean()*100, "%")

# Plot our complex trees:
plot_first_trees(bst, output_labels, trees_name="complex_tree")

# And their feature importance:
print("Now, our feature importance chart considers more features, but it is still not complete.")
fig, ax = plt.subplots(figsize=(12, 7))
xgb.plot_importance(bst, ax=ax)
plt.show()


I am working on the file "Decision-Trees-For-Knowledge-Discovery-With-XGBoost.ipynb".
The first task is:
Note that the above trees can be viewed here online:
https://github.com/Vooban/Decision-Trees-For-Knowledge-Discovery/tree/master/exported_xgboost_trees
Finding a perfect classifier rather than an easily explainable one
We'll now use boosting. The resulting trees can't be explained as easily as the previous ones, since one classifier will now have incrementally many trees for each class to reduce error, each new trees based on the errors of the previous ones. And those trees will each be weighted.
Can you write Python code for it?

num_rounds = 10  # 10 rounds of boosting, thus 10 trees per class.
num_classes = len(output_labels)
num_trees = num_rounds * num_classes

param = {
    'max_depth': 20,
    'eta': 1.43,
    'objective': 'multi:softprob',
    'num_class': num_classes,
}

bst = xgb.train(param, dtrain, early_stopping_rounds=1, num_boost_round=num_rounds, evals=[(dtrain, "dtrain")])
print("Boosted decision trees trained!")
print("Mean Error Rate:", bst.eval(dtrain))
print("Accuracy:", (bst.predict(dtrain).argmax(axis=-1) == integer_y).mean()*100, "%")


I am working on the file "Decision-Trees-For-Knowledge-Discovery-With-XGBoost.ipynb".
The first task is:
Plot and save the trees (one for each class):
The 4 trees of the classifer (one tree per class) will each output a number that represents how much it is probable that the thing to classify belongs to that class, and it is by comparing the output at the end of all the trees for a given example that we could get the maximal output as associating the example to that class.  Indeed, the binary situation where we have only one tree that outputs a positive and else negative number would be simpler to interpret rather than classifying for 4 binary classes at once.
Can you write Python code for it?

def plot_first_trees(bst, output_labels, trees_name):
    """
    Plot and save the first trees for multiclass classification
    before any boosting was performed.
    """
    for tree_idx in range(len(output_labels)):
        class_name = output_labels[tree_idx]
        graph_save_path = os.path.join(
            "exported_xgboost_trees",
            "{}_{}_for_{}".format(trees_name, tree_idx, class_name)
        )

        graph = xgb.to_graphviz(bst, num_trees=tree_idx)
        graph.render(graph_save_path)

        # from IPython.display import display
        # display(graph)
        ### Inline display in the notebook would be too huge and would require much side scrolling.
        ### So we rather plot it anew with matplotlib and a fixed size for inline quick view purposes:
        fig, ax = plt.subplots(figsize=(16, 16))
        plot_tree(bst, num_trees=tree_idx, rankdir='LR', ax=ax)
        plt.title("Saved a high-resolution graph for the class '{}' to: {}.pdf".format(class_name, graph_save_path))
        plt.show()

# Plot our simple trees:
plot_first_trees(bst, output_labels, trees_name="simple_tree")


I am working on the file "Decision-Trees-For-Knowledge-Discovery-With-XGBoost.ipynb".
The first task is:
Train simple decision trees (here using XGBoost) to fit the data set:
First, let's define some hyperparameters, such as the depth of the tree.
Can you write Python code for it?

num_rounds = 1  # Do not use boosting for now, we want only 1 decision tree per class.
num_classes = len(output_labels)
num_trees = num_rounds * num_classes

# Let's use a max_depth of 4 for the sole goal of simplifying the visual representation produced
# (ideally, a tree would be deeper to classify perfectly on that dataset)
param = {
    'max_depth': 4,
    'objective': 'multi:softprob',
    'num_class': num_classes
}

bst = xgb.train(param, dtrain, num_boost_round=num_rounds)
print("Decision trees trained!")
print("Mean Error Rate:", bst.eval(dtrain))
print("Accuracy:", (bst.predict(dtrain).argmax(axis=-1) == integer_y).mean()*100, "%")


I am working on the file "DecisionTree_Math_Fruits.ipynb".
The first task is:
2. Leaf node - Gives prediction
Can you write Python code for it?

"""
A Leaf node classifies data.
This holds a dictionary of class (e.g., "Apple") -> number of time it
appears in the rows from the training data that reach this leaf.
"""

class Leaf:
    def __init__(self, rows):
        self.predictions = class_counts(rows)


I am working on the file "DecisionTree_Math_Fruits.ipynb".
The first task is:
All Work Done !!! Now It's time to Build a Model from given Training data
Can you write Python code for it?

my_tree = build_tree(training_data)

print_tree(my_tree)


I am working on the file "DecisionTree_Math_Fruits.ipynb".
The first task is:
Build a Tree
Can you write Python code for it?

def build_tree(rows):

    # Try partitioing the dataset on each of the unique attribute,
    # calculate the information gain,
    # and return the question that produces the highest gain.
    gain, question = find_best_split(rows)

    # Base case: no further info gain
    # Since we can ask no further questions,
    # we'll return a leaf.
    if gain == 0:
        return Leaf(rows)

    # If we reach here, we have found a useful feature / value
    # to partition on.
    true_rows, false_rows = partition(rows, question)

    # Recursively build the true branch.
    true_branch = build_tree(true_rows)

    # Recursively build the false branch.
    false_branch = build_tree(false_rows)

    # Return a Question node.
    # This records the best feature / value to ask at this point,
    # as well as the branches to follow
    # dependingo on the answer.
    return Decision_Node(question, true_branch, false_branch)


I am working on the file "DecisionTree_Math_Fruits.ipynb".
The first task is:
Calculate the Information gain for a question given uncertainity at present node and incertainities at left and right child nodes
Can you write Python code for it?

def info_gain(left, right, current_uncertainty):
    #we need to calculate weighted avg of impurities at both child nodes
    p = float(len(left)) / (len(left) + len(right))
    return current_uncertainty - p * gini(left) - (1 - p) * gini(right)


I am working on the file "DecisionTree_Math_Fruits.ipynb".
The first task is:
Check for example
Can you write Python code for it?


I am working on the file "DecisionTree_Math_Fruits.ipynb".
The first task is:
Define a function which partitions the dataset on given question in True and False rows/examples
Can you write Python code for it?

"""For each row in the dataset, check if it satisfies the question. If
    so, add it to 'true rows', otherwise, add it to 'false rows'.
    """
def partition(rows, question):
    true_rows, false_rows = [], []
    for row in rows:
        if question.match(row):
            true_rows.append(row)
        else:
            false_rows.append(row)
    return true_rows, false_rows


I am working on the file "DecisionTree_Math_Fruits.ipynb".
The first task is:
Define nodes in tree
1. Decision Node - Node with Question to ask
Can you write Python code for it?

"""
A Decision Node asks a question.
This holds a reference to the question, and to the two child nodes.
"""
class Decision_Node:
    def __init__(self,question,true_branch,false_branch):
        self.question = question
        self.true_branch = true_branch
        self.false_branch = false_branch


I am working on the file "DecisionTree_Math_Fruits.ipynb".
The first task is:
Evaluate
Can you write Python code for it?

for row in testing_data:
    print ("Actual: %s. Predicted: %s" %
           (row[-1], print_leaf(classify(row, my_tree))))


I am working on the file "DecisionTree_Math_Fruits.ipynb".
The first task is:
Let's write a class for a question which can be asked to partition the data
Each object of a question class holds a column_no and a col_value
 Eg. column_no = 0 denotes color and so col_value can be Green, Yellow or Red
 We can write a method which would compare the feature value of example with the feature value of Question
Can you write Python code for it?

class Question:
    def __init__(self,col, val):
        self.col = col
        self.val = val

    def match(self,example):
        # Compare the feature value in an example to the
        # feature value in this question.
        value = example[self.col]
        if is_numeric(value):
            return value >= self.val
        else:
            return value == self.val

    def __repr__(self):
        # method to print the question in a readable format.
        condition = "=="
        if is_numeric(self.val):
            condition = ">="
        return "Is %s %s %s?" % (
            header[self.col], condition, str(self.val))


I am working on the file "DecisionTree_Math_Fruits.ipynb".
The first task is:
Now calculate a Gini Impurity for a node with given input rows of training dataset
Can you write Python code for it?

 """Calculate the Gini Impurity for a list of rows."""
def gini(rows):
    counts = class_counts(rows)
    impurity = 1
    for lbl in counts:
        prob_of_lbl = counts[lbl] / float(len(rows))
        impurity -= prob_of_lbl**2
    return impurity


I am working on the file "DecisionTree_Math_Fruits.ipynb".
The first task is:
Print Prediction at Leaf Node
Can you write Python code for it?

 """A nicer way to print the predictions at a leaf."""
def print_leaf(counts):
    total = sum(counts.values()) * 1.0
    probs = {}
    for lbl in counts.keys():
        probs[lbl] = str(int(counts[lbl] / total * 100)) + "%"
    return probs