I am working on the file "Day_2_Software_engineering_best_practices/solutions/04_modules/04_modules.ipynb".
The first task is:
Plot helper functions
We can create two functions: (i) to plot all spectra and (ii) plot the mean spectra with the std intervals.
We will make a "private" function which will be used by both plot types.
Can you write Python code for it?

fig, ax = plot_spectra(frequency, spectra, 'All training spectra')

fig, ax = plot_spectra_by_type(frequency, spectra, molecule)
ax.set_title('Mean spectra in function of the molecules')

fig, ax = plot_spectra_by_type(frequency, spectra, concentration,
                               'Mean spectra in function of the concentrations')


I am working on the file "Day_2_Software_engineering_best_practices/solutions/04_modules/04_modules.ipynb".
The first task is:
Reusability for new data:
Can you write Python code for it?

spectra_test, concentration_test, molecule_test = read_spectra('data/spectra_4.csv')

plot_spectra(frequency, spectra_test,
             'All training spectra')
plot_spectra_by_type(frequency, spectra_test, molecule_test,
                     'Mean spectra in function of the molecules')
plot_spectra_by_type(frequency, spectra_test, concentration_test,
                     'Mean spectra in function of the concentrations');


I am working on the file "Day_2_Software_engineering_best_practices/solutions/04_modules/04_modules.ipynb".
The first task is:
Training and testing a machine learning model for regression
Can you write Python code for it?

regression_experiment(spectra, spectra_test,
                      concentration, concentration_test)

# compute the statistics on the training data
med, var = fit_params(spectra)
# transform the training and testing data
spectra_scaled = transform(spectra, med, var)
spectra_test_scaled = transform(spectra_test, med, var)

regression_experiment(spectra_scaled, spectra_test_scaled,
                      concentration, concentration_test)


I am working on the file "Day_2_Software_engineering_best_practices/solutions/06_07_08_full_package/04_modules.ipynb".
The first task is:
IO: Reading and preprocess the data
We can define a function which will read the data and process them.
Can you write Python code for it?

# read the frequency and get a pandas serie
frequency = pd.read_csv('data/freq.csv')['freqs']

# read all data for training
filenames = ['data/spectra_{}.csv'.format(i)
             for i in range(4)]

spectra, concentration, molecule = [], [], []
for filename in filenames:
    spectra_file, concentration_file, molecule_file = read_spectra(filename)
    spectra.append(spectra_file)
    concentration.append(concentration_file)
    molecule.append(molecule_file)

# Concatenate in single DataFrame and Serie
spectra = pd.concat(spectra)
concentration = pd.concat(concentration)
molecule = pd.concat(molecule)


I am working on the file "Day_2_Software_engineering_best_practices/solutions/06_07_08_full_package/04_modules.ipynb".
The first task is:
Plot helper functions
We can create two functions: (i) to plot all spectra and (ii) plot the mean spectra with the std intervals.
We will make a "private" function which will be used by both plot types.
Can you write Python code for it?

plot_spectra(frequency, spectra, 'All training spectra')

plot_spectra_by_type(frequency, spectra, molecule,
                     'Mean spectra in function of the molecules')

plot_spectra_by_type(frequency, spectra, concentration,
                     'Mean spectra in function of the concentrations')


I am working on the file "Day_2_Software_engineering_best_practices/solutions/06_07_08_full_package/04_modules.ipynb".
The first task is:
Reusability for new data:
Can you write Python code for it?

spectra_test, concentration_test, molecule_test = read_spectra('data/spectra_4.csv')

plot_spectra(frequency, spectra_test,
             'All training spectra')
plot_spectra_by_type(frequency, spectra_test, molecule_test,
                     'Mean spectra in function of the molecules')
plot_spectra_by_type(frequency, spectra_test, concentration_test,
                     'Mean spectra in function of the concentrations')


I am working on the file "Day_2_Software_engineering_best_practices/solutions/06_07_08_full_package/04_modules.ipynb".
The first task is:
Training and testing a machine learning model for classification
Can you write Python code for it?

for clf in [RandomForestClassifier(random_state=0),
            LinearSVC(random_state=0)]:

    pipeline = make_pipeline(StandardScaler(),
                             PCA(n_components=100, random_state=0),
                             clf)
    y_pred = pipeline.fit(spectra, molecule).predict(spectra_test)
    plot_cm(confusion_matrix(molecule_test, y_pred),
            pipeline.classes_,
            'Confusion matrix using {}'.format(clf.__class__.__name__))
    print('Accuracy score: {0:.2f}'.format(pipeline.score(spectra_test,
                                                          molecule_test)))


I am working on the file "Day_2_Software_engineering_best_practices/solutions/06_07_08_full_package/04_modules.ipynb".
The first task is:
Training and testing a machine learning model for regression
Can you write Python code for it?


I am working on the file "Day_3/18-Basemap.ipynb".
The first task is:
Can you create your own map and zoom into Norfolk, label Norwich, Lowestoft and Sherringham.
Can you write Python code for it?


I am working on the file "Day_3/18-Basemap.ipynb".
The first task is:
Having some gridlines always helps too! But let's drop the bathymetry for now.
Can you write Python code for it?

my_map = Basemap(projection='ortho', lat_0=50, lon_0=-20,
              resolution='l', area_thresh=1000.0)

my_map.drawcoastlines()
my_map.drawcountries()
my_map.fillcontinents(color='coral')

my_map.drawmeridians(np.arange(0, 360, 30))
my_map.drawparallels(np.arange(-90, 90, 30))


I am working on the file "Day_3/18-Basemap.ipynb".
The first task is:
How about Portsmouth and Southampton too?
Can you write Python code for it?

my_map = Basemap(projection='lcc', lat_0=50, lon_0=-20,
    resolution = 'i', area_thresh = 0.1, # i = intermediate resoltion, f = full resolution (might take a while!)
    llcrnrlon=-2, llcrnrlat=50.5,
    urcrnrlon=0, urcrnrlat=50.8)

my_map.drawcoastlines()
my_map.drawcountries()
my_map.fillcontinents(color='coral')
my_map.drawmapboundary()

my_map.drawmeridians(np.arange(0, 360, 30))
my_map.drawparallels(np.arange(-90, 90, 30))

lons = [-1.3126, -1.1010, -1.4043]
lats = [50.7507, 50.8429, 50.9096]
x,y = my_map(lons, lats)
my_map.plot(x, y, 'bo', markersize=10)


I am working on the file "Day_3/18-Basemap.ipynb".
The first task is:
Let's try and correct those squares by increasing the resolution.
Can you write Python code for it?

my_map = Basemap(projection='lcc', lat_0=50, lon_0=-20,
    resolution = 'i', area_thresh = 0.1, # i = intermediate resoltion, f = full resolution (might take a while!)
    llcrnrlon=-2, llcrnrlat=50.5,
    urcrnrlon=0, urcrnrlat=50.8)

my_map.drawcoastlines()
my_map.drawcountries()
my_map.fillcontinents(color='coral')
my_map.drawmapboundary()

my_map.drawmeridians(np.arange(0, 360, 30))
my_map.drawparallels(np.arange(-90, 90, 30))

print("That's a bit better!")


I am working on the file "Day_3/18-Basemap.ipynb".
The first task is:
Now to add a little bit of detail. Draw country boundaries, fill in the continents with a colour, and add bathymetry.
Can you write Python code for it?

my_map = Basemap(projection='ortho', lat_0=50, lon_0=-20,
              resolution='l', area_thresh=1000.0)

my_map.drawcoastlines()
my_map.drawcountries()
my_map.fillcontinents(color='coral')
my_map.etopo()  # bathymetry


I am working on the file "Day_3/18-Basemap.ipynb".
The first task is:
Now to offset the labels slightly
Can you write Python code for it?

my_map = Basemap(projection='lcc', lat_0=50, lon_0=-20,
    resolution = 'i', area_thresh = 0.1, # i = intermediate resoltion, f = full resolution (might take a while!)
    llcrnrlon=-2, llcrnrlat=50.5,
    urcrnrlon=0, urcrnrlat=50.8)

my_map.drawcoastlines()
my_map.drawcountries()
my_map.fillcontinents(color='coral')
my_map.drawmapboundary()

my_map.drawmeridians(np.arange(0, 360, 30))
my_map.drawparallels(np.arange(-90, 90, 30))

lons = [-1.3126, -1.1010, -1.4043]
lats = [50.7507, 50.8429, 50.9096]
x,y = my_map(lons, lats)
my_map.plot(x, y, 'bo', markersize=10)

labels = ['Cowes', 'Portsmouth', 'Southampton']
for label, xpt, ypt in zip(labels, x, y):
    plt.text(xpt-7000, ypt+3000, label)


I am working on the file "Day_3/18-Basemap.ipynb".
The first task is:
Now to use the Basemap function to create a map with a set projection define the centre point using lat/lon, resolution: l=low, h=high, and area_threshold: only plots coastal feastures greater than 1000km2
Can you write Python code for it?

my_map = Basemap(projection='ortho', lat_0=50, lon_0=-20,
              resolution='l', area_thresh=1000.0)

my_map.drawcoastlines()


I am working on the file "Day_3/19-Cartopy-Intro.ipynb".
The first task is:
Creating GeoAxes

Cartopy-matplotlib interface is set up via the projection keyword when constructing Axes / SubAxes
The resulting instance (cartopy.mpl.geoaxes.GeoAxesSubplot) has new methods specific to drawing cartographic data, e.g. coastlines:
Can you write Python code for it?

ax = plt.axes(projection=ccrs.PlateCarree())
ax.coastlines()
print('axes type:', type(ax))


I am working on the file "Day_3/19-Cartopy-Intro.ipynb".
The first task is:
Plotting layers directly from Web Map Service (WMS) and Web Map Tile Service (WMTS)
Can you write Python code for it?

url = 'http://map1c.vis.earthdata.nasa.gov/wmts-geo/wmts.cgi'

ax = plt.axes(projection=ccrs.PlateCarree())
ax.add_wmts(url, 'VIIRS_CityLights_2012')


I am working on the file "Day_3/19-Cartopy-Intro.ipynb".
The first task is:
Then let's import the cartopy
Can you write Python code for it?


I am working on the file "Day_3/19-Cartopy-Intro.ipynb".
The first task is:
Unfortunately, gridline labels work only in PlateCarree and Mercator projections.
We can control the specific tick values by using matplotlib's locator object, and the formatting can be controlled with matplotlib formatters:
Can you write Python code for it?

import matplotlib.ticker as mticker
from cartopy.mpl.gridliner import LATITUDE_FORMATTER

ax = plt.axes(projection=ccrs.PlateCarree())
ax.coastlines()
gl = ax.gridlines(draw_labels=True)

gl.xlocator = mticker.FixedLocator([-180, -45, 0, 45, 180])
gl.yformatter = LATITUDE_FORMATTER

fig = plt.figure(figsize=(9, 6))
ax = fig.add_subplot(111, projection=ccrs.PlateCarree())
ax.set_global()

lons = -75, 77.2, 151.2, -75
lats = 43, 28.6, -33.9, 43

ax.plot(lons, lats,
        color='green', linewidth=2, marker='o', ms=10,
        transform=ccrs.Geodetic())

# feature = cartopy.feature.LAND
feature = cartopy.feature.NaturalEarthFeature(name='land', category='physical',
                                              scale='110m',
                                              edgecolor='red', facecolor='black')
ax.add_feature(feature)
_ = ax.add_feature(cartopy.feature.LAKES, facecolor='b')
states = cartopy.feature.NaturalEarthFeature(category='cultural', scale='50m', facecolor='none',
                                             name='admin_1_states_provinces_lines')
_ = ax.add_feature(states, edgecolor='gray')


I am working on the file "Day_3/20-NetCDF-Intro.ipynb".
The first task is:
...or import local dataset
Can you write Python code for it?

GETM = xr.open_dataset('../data/cefas_GETM_nwes.nc4')
GETM

GETM.dims

print(type(GETM.coords['latc']))
GETM.coords['latc'].shape

# List name of dataset attributes
GETM.attrs.keys()

# List variable names
GETM.data_vars.keys()


I am working on the file "Day_3/20-NetCDF-Intro.ipynb".
The first task is:
Accessing data values
Can you write Python code for it?


I am working on the file "Day_3/20-NetCDF-Intro.ipynb".
The first task is:
Arithmetic operations
Can you write Python code for it?

top=GETM['temp'].isel(time=0,level=4)
bottom=GETM['temp'].isel(time=0,level=0)

diff=top-bottom

diff.plot()


I am working on the file "Day_3/20-NetCDF-Intro.ipynb".
The first task is:
Define selection using nearest value
Can you write Python code for it?

#GETM.sel(level=1)['temp']
GETM['temp'].sel(level=1,lonc=-5.0,latc=-50.0, method='nearest')

try:
    GETM['temp'].sel(level=1,lonc=-5.0,latc=-50.0, method='nearest',tolerance=0.5)
except KeyError:
    print('ERROR: outside tolerance of '+str(0.5))


I am working on the file "Day_3/20-NetCDF-Intro.ipynb".
The first task is:
Exercise

Extract the bathymetry
Extract the seabed temperature    isel(level=0)
Produce a scatter plot of depth vs. seabed temperature
Can you write Python code for it?

# bathy = GETM

# bedtemp=GETM

# plt.scatter(  , ,marker='.')


I am working on the file "Day_3/20-NetCDF-Intro.ipynb".
The first task is:
Indexing and selecting data
From http://xarray.pydata.org/

Can you write Python code for it?

#positional by integer
print( temp[0,2,:,:].shape )

# positional by label
print( temp.loc['1996-02-02T01:00:00',:,:,:].shape )

# by name and integer
print( temp.isel(level=1,latc=90,lonc=100).shape )

# by name and label
print( temp.sel(time='1996-02-02T01:00:00').shape )
#temp.loc


I am working on the file "Day_3/20-NetCDF-Intro.ipynb".
The first task is:
The main advantages of using xarray versus plain netCDF4 are:

intelligent selection along labelled dimensions (and also indexes)
groupby operations
data alignment
IO (netcdf)
conversion from and to Pandas.DataFrames

Lets peek inside our example file using ncdump
$ ncdump data/cefas_GETM_nwes.nc | more
netcdf cefas_GETM_nwes {
dimensions:
        latc = 360 ;
        lonc = 396 ;
        time = UNLIMITED ; // (6 currently)
        level = 5 ;
variables:
        double bathymetry(latc, lonc) ;
                bathymetry:units = "m" ;
                bathymetry:long_name = "bathymetry" ;
                bathymetry:valid_range = -5., 4000. ;
                bathymetry:_FillValue = -10. ;
                bathymetry:missing_value = -10. ;
        float h(time, level, latc, lonc) ;
                h:units = "m" ;
                h:long_name = "layer thickness" ;
                h:_FillValue = -9999.f ;
                h:missing_value = -9999.f ;
        double latc(latc) ;
                latc:units = "degrees_north" ;
        double level(level) ;
                level:units = "level" ;
        double lonc(lonc) ;
                lonc:units = "degrees_east" ;
        float temp(time, level, latc, lonc) ;
                temp:units = "degC" ;
                temp:long_name = "temperature" ;
                temp:valid_range = -2.f, 40.f ;
                temp:_FillValue = -9999.f ;
                temp:missing_value = -9999.f ;
        double time(time) ;
                time:long_name = "time" ;
                time:units = "seconds since 1996-01-01 00:00:00" ;
Now let's go back to python and use xarray.
Import remote dataset
xarray supports OpenDAP. This means that a dataset can be accessed remotely and subsetted as needed. Only the selected parts are downloaded.
Can you write Python code for it?

# Naughty datasets might require decode_cf=False
# Here it just needed decode_times=False

naughty_data = xr.open_dataset(
      'http://iridl.ldeo.columbia.edu/SOURCES/.OSU/.PRISM/.monthly/dods',
      decode_times=False)
naughty_data


I am working on the file "Day_3/22-Final-Project.ipynb".
The first task is:
World Ocean Atlas
Inspired by this blog post: https://ocefpaf.github.io/python4oceanographers/blog/2015/05/04/woa13/

NOAA's World Ocean Atlas provides open-access gridded data of temperature, salinity and other ocean parameters in netCDF format.
It is a set of objectively analyzed (1$^\circ$ grid) climatological fields at standard depth levels for annual, seasonal, and monthly compositing periods. It also includes associated statistical fields of observed oceanographic profile data.

If you choose to analyse these data, we recommend that you start by:

downloading 5-degree data of temperature and oxygen
plotting the data on the global map (do not use jet/rainbow colormap!)
calculating an average depth profile and plotting it beside the map

Arctic Sea Ice
Data

In this project you are offered to use NOAA/NSIDC Climate Data Record of Passive Microwave Sea Ice Concentration.
In the ../data/ directory, there are 2 netCDF files seaice_conc_monthly* that correspond to September 1991 (original FTP link) and September 2012 (original FTP link).
If you want to download data for other months, visit the NSIDC's data portal.

Ideas for the project

Plot one of the time slices on a map with North Polar Stereographic projection
Create a figure with 3 subplots
Plot the 1991 sea ice concentration in the 1st subplot, 2012 sea ice in the 2nd, and the difference in the 3rd.

Getting started
For this project, we recommend that you:
* use xarray for opening and reading the netCDF files
* may use xarray.open_mf_dataset() to load both files at once
* use cartopy for creating a plot with a correct map projection
* use appropriate colormaps for the sea ice concentration and difference
To get started, copy the following cell into your Project notebook.
Can you write Python code for it?

# import cartopy.crs as ccrs
# import matplotlib.pyplot as plt
# import xarray as xr
# %matplotlib inline

# ds = xr.open_mfdataset('../data/seaice_conc_monthly_*.nc')
## or
# ds1 = xr.open_dataset('../data/seaice_conc_monthly_nh_f08_199109_v02r00.nc')
# ds2 = xr.open_dataset('../data/seaice_conc_monthly_nh_f17_201209_v02r00.nc')

## Extract longitude and latitude values, then the sea ice concentration itself

## Code for creating a map
# fig = plt.figure()
# ax = fig.add_subplot(111, projection=ccrs.???(central_longitude=0))
# ax.coastlines(resolution='110m', linewidth=0.5)

# ax.gridlines()
# ax.set_extent([-180, 180, 40, 90], crs=ccrs.PlateCarree())


I am working on the file "Day_3/solutions/19-Cartopy-Intro-Solution.ipynb".
The first task is:
Creating GeoAxes

Cartopy-matplotlib interface is set up via the projection keyword when constructing Axes / SubAxes
The resulting instance (cartopy.mpl.geoaxes.GeoAxesSubplot) has new methods specific to drawing cartographic data, e.g. coastlines:
Can you write Python code for it?


I am working on the file "Day_3/solutions/19-Cartopy-Intro-Solution.ipynb".
The first task is:
Exercise
Can you write Python code for it?

import numpy as np

x = np.linspace(310, 390, 25)
y = np.linspace(-24, 25, 35)
x2d, y2d = np.meshgrid(x, y)

data = np.cos(np.deg2rad(y2d) * 4) + np.sin(np.deg2rad(x2d) * 4)


I am working on the file "Day_3/solutions/19-Cartopy-Intro-Solution.ipynb".
The first task is:
Idea 1

Use data in a rotated pole coordinate system
Create a proper CRS, plot it
Plot the data on a different map with a different projection (e.g., Plate Carree)
Can you write Python code for it?

rot_crs = ccrs.RotatedPole(177.5, 37.5)

ax = plt.axes(projection=rot_crs)
ax.coastlines()

fig = plt.figure()
ax = fig.add_subplot(111, projection=ccrs.PlateCarree())
ax.contourf(x2d, y2d, data, transform=rot_crs)
ax.coastlines()


I am working on the file "Day_3/solutions/19-Cartopy-Intro-Solution.ipynb".
The first task is:
Notice that unless we specify a map extent (we did so via the set_global method in this case) the map will zoom into the range of the plotted data.
Decorating the map
We can add grid lines and tick labels to the map using the gridlines() method:
Can you write Python code for it?

ax = plt.axes(projection=ccrs.Mercator())
ax.coastlines()
gl = ax.gridlines(draw_labels=True)


I am working on the file "Day_3/solutions/19-Cartopy-Intro-Solution.ipynb".
The first task is:
Then let's import the cartopy
Can you write Python code for it?


I am working on the file "Day_3/solutions/19-Cartopy-Intro-Solution.ipynb".
The first task is:
Unfortunately, gridline labels work only in PlateCarree and Mercator projections.
We can control the specific tick values by using matplotlib's locator object, and the formatting can be controlled with matplotlib formatters:
Can you write Python code for it?


I am working on the file "Db2 Calling Stored Procedures.ipynb".
The first task is:
A normal call to this stored procedure (no flags) will return the result set. Note we must supply the second parameter here but do not supply a value since it is an output value. The null keyword must be used.
Can you write Python code for it?


I am working on the file "Db2 Calling Stored Procedures.ipynb".
The first task is:
Adding the -a (all output) flag to a stored procedure call will display all results. This is the same behavior when running regular SQL statements. If you want to assigned the result to a variable, you only need to add an assignment statement to the SQL.
Can you write Python code for it?

allemps = %sql CALL SHOWEMP
allemps.head(5)


I am working on the file "Db2 Calling Stored Procedures.ipynb".
The first task is:
Calling the stored procedure is done using the standard SQL CALL procname(arguments) format. There are specific rules for supplying arguments to a stored procedure:
* The CALL command is supported in a %sql statement only. It cannot be used as part of a %%sql block.
* Null arguments must use the null keyword, rather than the Python None equivalent.
* Brackets () are not required for stored procedures that have no arguments.
The next statement will execute the stored procedure and display the results by default.
Can you write Python code for it?


I am working on the file "Db2 Calling Stored Procedures.ipynb".
The first task is:
If you use the standard %sql command without the -r parameter, only the answer set will be returned or displayed. An assignment statement with -r is slightly more complex:
* The first value will be the array of results
* Subsequent values will be the input/output parameters of the stored procedure
The next SQL statement will assign the results to a single variable.
Can you write Python code for it?


I am working on the file "Db2 Calling Stored Procedures.ipynb".
The first task is:
Remember that there is no result set with the stored procedure so the parameters are returned.
Can you write Python code for it?


I am working on the file "Db2 Calling Stored Procedures.ipynb".
The first task is:
Stored Procedure Parameters
Stored procedures can have variables passed as parameters. To pass a parameter to a stored procedure, place a colon (:) in front of the variable name:
%sql CALL SHOWDEPT(:deptno,null)
The python variable deptno will be substituted into the CALL statement when it is executed.
Can you write Python code for it?

%%sql -d
create or replace procedure deptcount(in indeptno char(3), out rowcount int)
begin
    set rowcount = (SELECT COUNT(*) FROM EMPLOYEE WHERE WORKDEPT=indeptno);
end@


I am working on the file "Db2 Calling Stored Procedures.ipynb".
The first task is:
Stored procedures without answer sets will always return the parameters or None if nothing was supplied. This simple stored procedure increments the number that was sent to it.
Can you write Python code for it?

%%sql -d
create or replace procedure oneparm (inout r INT)
begin
    SET r = r + 1;
end@


I am working on the file "Db2 Calling Stored Procedures.ipynb".
The first task is:
System Stored Procedures
The Db2 System Stored procedures work using this syntax except for procedures that return binary XML output. At this point in time there is a limitation in retrieving this data using the Python Db2 API calls that are available. An example of a working procedure call is the REORGCHK procedure.
Can you write Python code for it?


I am working on the file "Db2 Calling Stored Procedures.ipynb".
The first task is:
The first value in the oneemp array (oneemp[0]) is the answer set array, while the subsequent values are the parameters that are passed (or returned) by the stored procedure. oneemp[1] should be equal to the employee number we were searching for.
Can you write Python code for it?


I am working on the file "Db2 Calling Stored Procedures.ipynb".
The first task is:
The use of the -r (raw output) flag will force the output to be converted into a two-dimensional array. The first line of th array will contain the column names, while the remainder will have the results in it. This format is useful if you want to write an application in Python to manipulate the data.
Can you write Python code for it?


I am working on the file "Db2 Calling Stored Procedures.ipynb".
The first task is:
The use of the -r flag becomes mandatory when you are retrieving an answer set from a stored procedure, and also need to access the return results of the arguments to the stored procedure. SQL stored procedures can have input, output, and input/output values. These values are returned back via the %sql command but can only be accessed when you use the -r flag.
The following stored procedure will return the employees in a department and also a count of the records found.
Can you write Python code for it?

%%sql -d
create or replace procedure showdept(in indeptno char(3), out rowcount int)
DYNAMIC RESULT SETS 1
begin
    DECLARE c1 CURSOR WITH RETURN TO CALLER FOR
       SELECT * FROM EMPLOYEE WHERE WORKDEPT=indeptno;
    set rowcount = (SELECT COUNT(*) FROM EMPLOYEE WHERE WORKDEPT=indeptno);
    OPEN c1;
end@


I am working on the file "Db2 Calling Stored Procedures.ipynb".
The first task is:
To access the value of the rowcount, the -r flag must be used and the results assigned to a variable.
Can you write Python code for it?

alldept = %sql -r CALL SHOWDEPT('E11',null)


I am working on the file "Db2 Calling Stored Procedures.ipynb".
The first task is:
When calling this procedure we must supply the employee number as a 6 character field. The behaviour will be similar to the first example with no parameters.
Can you write Python code for it?


I am working on the file "Db2 Compatibility Features.ipynb".
The first task is:
Adding the FETCH FIRST clause only reduces the rows returned, not the rows that
are used to compute the GROUPing result.
Can you write Python code for it?

%%sql
SELECT WORKDEPT, COUNT(*) FROM EMPLOYEE
  GROUP BY WORKDEPT
  ORDER BY WORKDEPT
  FETCH FIRST 5 ROWS ONLY


I am working on the file "Db2 Compatibility Features.ipynb".
The first task is:
And here is the list of employees that make more than the average salary of the 3rd highest department in the
company.
Can you write Python code for it?

%%sql
SELECT LASTNAME, SALARY FROM EMPLOYEE
  WHERE
    SALARY > (
       SELECT AVG(SALARY) FROM EMPLOYEE
         GROUP BY WORKDEPT
         ORDER BY AVG(SALARY) DESC
         OFFSET 2 ROWS FETCH FIRST 1 ROW ONLY
       )
ORDER BY SALARY


I am working on the file "Db2 Compatibility Features.ipynb".
The first task is:
Back to Top

Binary Data Types
Db2 11 introduces two new binary data types: BINARY and VARBINARY. These two data types can contain any combination
of characters or binary values and are not affected by the codepage of the server that the values are stored on.
A BINARY data type is fixed and can have a maximum length of 255 bytes, while a VARBINARY column can contain up to
32672 bytes. Each of these data types is compatible with columns created with the FOR BIT DATA keyword.
The BINARY data type will reduce the amount of conversion required from other data bases. Although binary data was supported with the FOR BIT DATA clause on a character column, it required manual DDL changes when migrating a table definition.
This example shows the creation of the three types of binary data types.
Can you write Python code for it?

%%sql -q
DROP TABLE HEXEY;

CREATE TABLE HEXEY
  (
  AUDIO_SHORT BINARY(255),
  AUDIO_LONG  VARBINARY(1024),
  AUDIO_CHAR  VARCHAR(255) FOR BIT DATA
  );


I am working on the file "Db2 Compatibility Features.ipynb".
The first task is:
Back to Top

Boolean Data Type
The boolean data type (true/false) has been available in SQLPL and PL/SQL scripts for some time. However,
the boolean data type could not be used in a table definition. Db2 11 FP1 now allows you to use this
data type in a table definition and use TRUE/FALSE clauses to compare values.
This simple table will be used to demonstrate how BOOLEAN types can be used.
Can you write Python code for it?

%%sql -q
DROP TABLE TRUEFALSE;

CREATE TABLE TRUEFALSE (
    EXAMPLE INT,
    STATE   BOOLEAN
);


I am working on the file "Db2 Compatibility Features.ipynb".
The first task is:
Back to Top

CHAR Datatype Size Increase
The CHAR datatype was limited to 254 characters in prior releases of Db2. In Db2 11, the limit has been increased
to 255 characters to bring it in line with other SQL implementations.
First we drop the table if it already exists.
Can you write Python code for it?

%%sql -q
DROP TABLE LONGER_CHAR;

CREATE TABLE LONGER_CHAR
  (
  NAME CHAR(255)
  );


I am working on the file "Db2 Compatibility Features.ipynb".
The first task is:
Back to Top

SELECT Enhancements
Db2 has the ability to limit the amount of data retrieved on a SELECT statement
through the use of the FETCH FIRST n ROWS ONLY clause. In Db2 11, the ability to offset
the rows before fetching was added to the FETCH FIRST clause.
Simple SQL with Fetch First Clause
The FETCH first clause can be used in a variety of locations in a SELECT clause. This
first example fetches only 10 rows from the EMPLOYEE table.
Can you write Python code for it?

%%sql
SELECT LASTNAME FROM EMPLOYEE
  FETCH FIRST 5 ROWS ONLY


I am working on the file "Db2 Compatibility Features.ipynb".
The first task is:
Back to Top

Synonym Data types
Db2 has the standard data types that most developers are familiar with, like CHAR, INTEGER, and DECIMAL. There are other SQL implementations that use different names for these data types, so Db2 11 now allows these data types as syonomys for the base types.
These data types are:
|Type   |Db2 Equivalent
|:----- |:-------------
|INT2   |SMALLINT
|INT4   |INTEGER
|INT8   |BIGINT
|FLOAT4 |REAL
|FLOAT8 |FLOAT
The following SQL will create a table with all of these data types.
Can you write Python code for it?

%%sql -q
DROP TABLE SYNONYM_EMPLOYEE;

CREATE TABLE SYNONYM_EMPLOYEE
  (
  NAME VARCHAR(20),
  SALARY     INT4,
  BONUS      INT2,
  COMMISSION INT8,
  COMMISSION_RATE FLOAT4,
  BONUS_RATE FLOAT8
  );


I am working on the file "Db2 Compatibility Features.ipynb".
The first task is:
GROUP BY Ordinal Location
The GROUP BY command behavior also changes in NPS mode. The following SQL statement groups results
using the default Db2 syntax:
Can you write Python code for it?

%%sql
SET SQL_COMPAT='DB2';

SELECT WORKDEPT,INT(AVG(SALARY))
  FROM EMPLOYEE
GROUP BY WORKDEPT;


I am working on the file "Db2 Compatibility Features.ipynb".
The first task is:
Handling binary data with a FOR BIT DATA column was sometimes tedious, so the BINARY columns will make coding a little simpler. You can compare and assign values between any of these types of columns. The next SQL statement will update the AUDIO_CHAR column with the contents of the AUDIO_SHORT column. Then the SQL will test to make sure they are the same value.
Can you write Python code for it?

%%sql
UPDATE HEXEY
  SET AUDIO_CHAR = AUDIO_SHORT


I am working on the file "Db2 Compatibility Features.ipynb".
The first task is:
Here is an example with the GROUP BY statement. This first SQL statement gives us the total
answer set - the count of employees by WORKDEPT.
Can you write Python code for it?

%%sql
SELECT WORKDEPT, COUNT(*) FROM EMPLOYEE
  GROUP BY WORKDEPT
  ORDER BY WORKDEPT


I am working on the file "Db2 Compatibility Features.ipynb".
The first task is:
Here is the list of employees that make more than the average salary of the 3rd highest department in the
company. Note that the LIMIT clause specifies only the offset (LIMIT x) or the offset and limit (LIMIT y,x) when you do not use the LIMIT keyword. One would think that LIMIT x OFFSET y would translate into LIMIT x,y but that is not the case. Don't try to figure out the SQL standards reasoning behind the syntax!
Can you write Python code for it?

%%sql
SELECT LASTNAME, SALARY FROM EMPLOYEE
  WHERE
    SALARY > (
       SELECT AVG(SALARY) FROM EMPLOYEE
         GROUP BY WORKDEPT
         ORDER BY AVG(SALARY) DESC
         LIMIT 2,1
       )
ORDER BY SALARY


I am working on the file "Db2 Compatibility Features.ipynb".
The first task is:
ISNULL, NOTNULL is an alias for IS NULL, IS NOT NULL
Can you write Python code for it?

%%sql
WITH EMP(LASTNAME, WORKDEPT) AS
  (
  VALUES ('George','A01'),
         ('Fred',NULL),
         ('Katrina','B01'),
         ('Bob',NULL)
  )
SELECT * FROM EMP WHERE
   WORKDEPT ISNULL


I am working on the file "Db2 Compatibility Features.ipynb".
The first task is:
If NPS compatibility is turned on then then you use the GROUP BY clause with an ordinal location.
Can you write Python code for it?

%%sql
SET SQL_COMPAT='NPS';
SELECT WORKDEPT, INT(AVG(SALARY))
  FROM EMPLOYEE
GROUP BY 1;


I am working on the file "Db2 Compatibility Features.ipynb".
The first task is:
If we turn on NPS compatibility, you see a couple of special characters change behavior. Specifically the
^ operator becomes a "power" operator, and the # becomes an XOR operator.
Can you write Python code for it?


I am working on the file "Db2 Compatibility Features.ipynb".
The first task is:
If you try using the ordinal location (similar to an ORDER BY clause), you will
get an error message.
Can you write Python code for it?

%%sql
SELECT WORKDEPT, INT(AVG(SALARY))
  FROM EMPLOYEE
GROUP BY 1;


I am working on the file "Db2 Compatibility Features.ipynb".
The first task is:
In this example, the letter 'o' will be replaced with an '1'.
Can you write Python code for it?


I am working on the file "Db2 Compatibility Features.ipynb".
The first task is:
LOG is an alias for LN
Can you write Python code for it?

%%sql
VALUES ('LOG',LOG(10))
UNION ALL
VALUES ('LN', LN(10))


I am working on the file "Db2 Compatibility Features.ipynb".
The first task is:
Now we can check to see what has been inserted into the table.
Can you write Python code for it?


I am working on the file "Db2 Compatibility Features.ipynb".
The first task is:
Now we can move the answer set ahead by 5 rows and get the remaining
5 rows in the answer set.
Can you write Python code for it?

%%sql
SELECT LASTNAME FROM EMPLOYEE
  OFFSET 5 ROWS
  FETCH FIRST 5 ROWS ONLY


I am working on the file "Db2 Compatibility Features.ipynb".
The first task is:
RAWTOHEX Function
The RAWTOHEX function returns a hexadecimal representation of a value as a character string. The
result is a character string itself.
Can you write Python code for it?


I am working on the file "Db2 Compatibility Features.ipynb".
The first task is:
Retrieving the data in a SELECT statement will return an integer value for display purposes.
1 is true and 0 is false (binary 1 and 0).
Comparison operators with BOOLEAN data types will use TRUE, FALSE, 1 or 0 or any of the supported binary values. You have the choice of using the equal (=) operator or the IS or IS NOT syntax as shown in the following SQL.
Can you write Python code for it?

%%sql
SELECT * FROM TRUEFALSE
  WHERE STATE = TRUE OR STATE = 1 OR STATE = 'on' OR STATE IS TRUE


I am working on the file "Db2 Compatibility Features.ipynb".
The first task is:
STRRIGHT is an alias for RIGHT
Can you write Python code for it?

%%sql
VALUES ('RIGHT',RIGHT('Hello There',5))
UNION ALL
VALUES ('STRRIGHT',STRRIGHT('Hello There',5))


I am working on the file "Db2 Compatibility Features.ipynb".
The first task is:
TO_HEX Function
The TO_HEX function converts a numeric expression into a character hexadecimal representation. For example, the
numeric value 255 represents x'FF'. The value returned from this function is a VARCHAR value and its
length depends on the size of the number you supply.
Can you write Python code for it?


I am working on the file "Db2 Compatibility Features.ipynb".
The first task is:
The DEFINITION ONLY clause will create the table but not load any data into it. Adding the WITH DATA
clause will do an INSERT of rows into the newly created table. If you have a large amount of data
to load into the table you may be better off creating the table with DEFINITION ONLY and then using LOAD
or other methods to load the data into the table.
Can you write Python code for it?

%sql -q DROP TABLE AS_EMP
%sql CREATE TABLE AS_EMP AS (SELECT EMPNO, SALARY+BONUS AS PAY FROM EMPLOYEE) WITH DATA;


I am working on the file "Db2 Compatibility Features.ipynb".
The first task is:
This example will show the four functions used against SMALLINT (INT2) data types.
Can you write Python code for it?

%%sql
WITH LOGIC(EXAMPLE, X, Y, RESULT) AS
  (
  VALUES
     ('INT2AND(X,Y)',XINT2,YINT2,INT2AND(XINT2,YINT2)),
     ('INT2OR(X,Y) ',XINT2,YINT2,INT2OR(XINT2,YINT2)),
     ('INT2XOR(X,Y)',XINT2,YINT2,INT2XOR(XINT2,YINT2)),
     ('INT2NOT(X)  ',XINT2,YINT2,INT2NOT(XINT2))
  )
SELECT * FROM LOGIC


I am working on the file "Db2 Compatibility Features.ipynb".
The first task is:
This example will use the 4 byte (INT4) data type.
Can you write Python code for it?

%%sql
WITH LOGIC(EXAMPLE, X, Y, RESULT) AS
  (
  VALUES
     ('INT4AND(X,Y)',XINT4,YINT4,INT4AND(XINT4,YINT4)),
     ('INT4OR(X,Y) ',XINT4,YINT4,INT4OR(XINT4,YINT4)),
     ('INT4XOR(X,Y)',XINT4,YINT4,INT4XOR(XINT4,YINT4)),
     ('INT4NOT(X)  ',XINT4,YINT4,INT4NOT(XINT4))
  )
SELECT * FROM LOGIC


I am working on the file "Db2 Compatibility Features.ipynb".
The first task is:
This example works in the same manner as the last one, but uses
the "+" sign syntax. The format is a lot simpler to remember than OUTER JOIN
syntax, but it is not part of the SQL standard.
Can you write Python code for it?

%%sql
SELECT DEPTNAME, LASTNAME FROM
  DEPARTMENT D, EMPLOYEE E
WHERE D.DEPTNO = E.WORKDEPT (+)


I am working on the file "Db2 Compatibility Features.ipynb".
The first task is:
Translate will also remove a character if it is not in the "to" list.
Can you write Python code for it?


I am working on the file "Db2 Compatibility Features.ipynb".
The first task is:
VAR_POP is an alias for VARIANCE
Can you write Python code for it?

%%sql
SELECT 'STDDEV_POP', STDDEV_POP(X) FROM XYCOORDS
UNION ALL
SELECT 'STDDEV', STDDEV(X) FROM XYCOORDS


I am working on the file "Db2 Compatibility Features.ipynb".
The first task is:
VAR_SAMP is an alias for VARIANCE_SAMP
Can you write Python code for it?

%%sql
SELECT 'VAR_SAMP', VAR_SAMP(X) FROM XYCOORDS
UNION ALL
SELECT 'VARIANCE_SAMP', VARIANCE_SAMP(X) FROM XYCOORDS


I am working on the file "Db2 Compatibility Features.ipynb".
The first task is:
We populate the database with the EMPLOYEE and DEPARTMENT tables so that we can run the various examples.
Can you write Python code for it?


I am working on the file "Db2 Compatibility Features.ipynb".
The first task is:
We should have one record that is equal.
Can you write Python code for it?

%%sql
SELECT COUNT(*) FROM HEXEY WHERE
  AUDIO_SHORT = AUDIO_CHAR


I am working on the file "Db2 Compatibility Features.ipynb".
The first task is:
When you create a table with these other data types, Db2 does not use these "types" in the catalog. What Db2 will do is use the Db2 type instead of these synonym types. What this means is that if you describe the contents of a table,
you will see the Db2 types displayed, not these synonym types.
Can you write Python code for it?

%%sql
SELECT DISTINCT(NAME), COLTYPE, LENGTH FROM SYSIBM.SYSCOLUMNS
  WHERE TBNAME='SYNONYM_EMPLOYEE' AND TBCREATOR=CURRENT USER


I am working on the file "Db2 Compatibility Features.ipynb".
The first task is:
You can also add ORDER BY and GROUP BY clauses in the SELECT statement. Note that
Db2 still needs to process all of the records and do the ORDER/GROUP BY work
before limiting the answer set. So you are not getting the first 5 rows "sorted". You
are actually getting the entire answer set sorted before retrieving just 5 rows.
Can you write Python code for it?

%%sql
SELECT LASTNAME FROM EMPLOYEE
  ORDER BY LASTNAME
  FETCH FIRST 5 ROWS ONLY


I am working on the file "Db2 Compatibility Features.ipynb".
The first task is:
You can also use the OFFSET clause as part of the FETCH FIRST ONLY to get chunks of data from the
original table.
Can you write Python code for it?

%%sql -q
DROP TABLE AS_EMP;
CREATE TABLE AS_EMP(DEPARTMENT, LASTNAME) AS
  (SELECT WORKDEPT, LASTNAME FROM EMPLOYEE
     OFFSET 5 ROWS
     FETCH FIRST 10 ROWS ONLY
  ) WITH DATA;
SELECT * FROM AS_EMP;


I am working on the file "Db2 Compatibility Features.ipynb".
The first task is:
You can specify a zero offset to begin from the beginning.
Can you write Python code for it?

%%sql
SELECT LASTNAME FROM EMPLOYEE
  OFFSET 0 ROWS
  FETCH FIRST 10 ROWS ONLY


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
Additional DEPARTMENT Table
In addition to the JSON_EMP table, the following SQL will generate a traditional table called JSON_DEPT
that can be used to determine the name of the department an individual works in.
Can you write Python code for it?

%%sql -q
DROP TABLE JSON_DEPT;

CREATE TABLE JSON_DEPT
  (
  SEQ INT NOT NULL GENERATED ALWAYS AS IDENTITY,
  DEPT_DATA BLOB(4000) INLINE LENGTH 4000
  );

INSERT INTO JSON_DEPT(DEPT_DATA) VALUES
  JSON2BSON('{"deptno":"A00", "mgrno":"000010", "admrdept":"A00", "deptname":"SPIFFY COMPUTER SERVICE DIV."}'),
  JSON2BSON('{"deptno":"B01", "mgrno":"000020", "admrdept":"A00", "deptname":"PLANNING"                    }'),
  JSON2BSON('{"deptno":"C01", "mgrno":"000030", "admrdept":"A00", "deptname":"INFORMATION CENTER"          }'),
  JSON2BSON('{"deptno":"D01",                   "admrdept":"A00", "deptname":"DEVELOPMENT CENTER"          }'),
  JSON2BSON('{"deptno":"D11", "mgrno":"000060", "admrdept":"D01", "deptname":"MANUFACTURING SYSTEMS"       }'),
  JSON2BSON('{"deptno":"D21", "mgrno":"000070", "admrdept":"D01", "deptname":"ADMINISTRATION SYSTEMS"      }'),
  JSON2BSON('{"deptno":"E01", "mgrno":"000050", "admrdept":"A00", "deptname":"SUPPORT SERVICES"            }'),
  JSON2BSON('{"deptno":"E11", "mgrno":"000090", "admrdept":"E01", "deptname":"OPERATIONS"                  }'),
  JSON2BSON('{"deptno":"E21", "mgrno":"000100", "admrdept":"E01", "deptname":"SOFTWARE SUPPORT"            }'),
  JSON2BSON('{"deptno":"F22",                   "admrdept":"E01", "deptname":"BRANCH OFFICE F2"            }'),
  JSON2BSON('{"deptno":"G22",                   "admrdept":"E01", "deptname":"BRANCH OFFICE G2"            }'),
  JSON2BSON('{"deptno":"H22",                   "admrdept":"E01", "deptname":"BRANCH OFFICE H2"            }'),
  JSON2BSON('{"deptno":"I22",                   "admrdept":"E01", "deptname":"BRANCH OFFICE I2"            }'),
  JSON2BSON('{"deptno":"J22",                   "admrdept":"E01", "deptname":"BRANCH OFFICE J2"            }')
;


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
An easier way is to use the JSON_GET_POS_ARR_INDEX function.
This function will search array values without having to
extract the array values with the JSON_TABLE function.
The format of the JSON_GET_POS_ARR_INDEX function is:
Python
   element = JSON_GET_POS_ARR_INDEX(document, field)
The arguments are:
- document - BSON document
- field - The field we are looking for and its value
- element - The first occurrence of the value in the array
The format of the field argument is "{field:value}" and it needs to be in
BSON format. This means you needs to add the JSON2BSON
function around the field specification.
Python
    JSON2BSON( '{"field":"value"}' )
This function only tests for equivalence and the data type should match what is
already in the field. The return value is the position
within the array that the value was found, where the first
element starts at zero.
In our JSON_EMP table, each employee has one or more phone
numbers. The following SQL will retrieve all employees who
have the extension 1422:
Can you write Python code for it?

%%sql
SELECT JSON_VAL(EMP_DATA, 'lastname', 's:10') AS LASTNAME
  FROM JSON_EMP
WHERE JSON_GET_POS_ARR_INDEX(EMP_DATA,
  JSON2BSON('{"phoneno":1422}')) >= 0


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
And we can check it using our original view.
Can you write Python code for it?


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
Applications will only deal with the EMP_TXS view. Any
inserts will use the text version of the JSON and not have
to worry about using the JSON2BSON function since the
underlying INSTEAD OF trigger will take care of the
conversion.
The following insert statement only includes the JSON string
since the sequence number will be generated automatically as
part of the insert.
Can you write Python code for it?


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
At this point we can create three INSTEAD OF triggers to handle insert,
updates and deletes on the view.
On INSERT the DEFAULT keyword is used to generate the ID number, the JSON field is
set to NULL and the BSON column contains the converted value of the JSON string.
Can you write Python code for it?

%%sql -d
CREATE OR REPLACE TRIGGER I_EMP_TXS
  INSTEAD OF INSERT ON EMP_TXS
  REFERENCING NEW AS NEW_TXS
  FOR EACH ROW MODE DB2SQL
BEGIN ATOMIC
  INSERT INTO BASE_EMP_TXS VALUES (
     DEFAULT,
     NULL,
     SYSTOOLS.JSON2BSON(NEW_TXS.INFO)
     );
END
@


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
Back to Top

Retrieving Array Values
Selecting data from an array type will always give you the first value (element zero).
The employees all have extension numbers but some of them have more than one.
Some of the extensions start with a zero so since the column is being treated as an
integer you will get only 3 digits. It's probably better to define it as a character
string rather than a number! Note that
Can you write Python code for it?

%sql -a SELECT JSON_VAL(EMP_DATA, 'phoneno.0', 'i') FROM JSON_EMP


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
Back to Top

Dealing with JSON Arrays
JSON arrays require specialized handling since there is no easy way to map an array to a single column in a relational table. Instead, there are a number of functions which will convert the array into a table so that you can access the individual elements in an SQL statement.

Accessing all Elements in an Array
The following query works because we do not treat the field phoneno as an array:
Can you write Python code for it?

%sql SELECT JSON_VAL(EMP_DATA, 'phoneno', 'i') FROM JSON_EMP


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
Back to Top

JSON BOOLEAN VALUES
JSON has a data type which can be true or false (boolean). Db2 doesn't have an
equivalent data type for boolean, so we need to retrieve it as an integer or
character string (true/false).
The JSON type for boolean values is 8.
The SQL statement below inserts a true and false value into the table.
Can you write Python code for it?

%%sql
DELETE FROM SANDBOX;

INSERT INTO SANDBOX VALUES
  JSON2BSON('{"valid":true, "invalid":false}');


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
Back to Top

BSON2JSON: Retrieving a JSON Document
Note that the data that is stored in a JSON column is in a special binary format called BSON. Selecting from the field will only result in hexadecimal characters being displayed.
Can you write Python code for it?

%%sql
SELECT CAST(JSON_FIELD AS VARCHAR(60)) FROM TESTJSON


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
Back to Top

JSON DATE, TIME, and TIMESTAMPS
This first SQL statement will insert a JSON field that uses the $date modifier.
Can you write Python code for it?

%%sql
DELETE FROM SANDBOX;

INSERT INTO SANDBOX VALUES
  JSON2BSON('{"today":{"$date":"2016-07-01T12:00:00"}}');


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
Back to Top

Indexing JSON Documents
Db2 supports computed indexes, which allows for the use
of functions like JSON_VAL to be used as part of the index
definition. For instance, searching for an employee number
will result in a scan against the table if no indexes are
defined:
Can you write Python code for it?

%%sql -q
DROP INDEX IX_JSON;
SELECT JSON_VAL(EMP_DATA, 'lastname', 's:20') AS LASTNAME
  FROM JSON_EMP
WHERE  JSON_VAL(EMP_DATA, 'empno', 's:6') = '000010';


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
Back to Top

JSON INTEGERS and BIGINT
Integers within JSON documents are easily identified as numbers that don't have a
decimal places in them. There are two different types of integers supported
within Db2 and are identified by the size (number of digits) in the number itself.

Integer - A set of digits that do not include a decimal place. The number cannot exceed -2,147,483,648 to 2,147,483,647.
Bigint  - A set of digits that do not include a decimal place but exceed that of an integer. The number cannot exceed -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807.

You don't explicitly state the type of integer that you are using.
The system will detect the type based on its size.
The JSON_TYPE function will return a value of 16 for integers and 18 for a
large integer (BIGINT). To retrieve a value from an integer field you need to
use the "i" flag and "l" (lowercase L) for big integers.
This first SQL statement will create a regular integer field.
Can you write Python code for it?

%%sql
INSERT INTO SANDBOX VALUES
  JSON2BSON('{"count":9782333}')


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
Back to Top

JSON2BSON: Inserting a JSON Document
Inserting into a column requires the use of the JSON2BSON function. The JSON2BSON function (and BSON2JSON) are used to transfer data in and out of a traditional Db2 BLOB column. There is no native JSON data type in Db2. Input to the JSON2BSON function must be a properly formatted JSON document. In the event that the document does not follow proper JSON rules, you will get an error code from the function.
Can you write Python code for it?

%%sql
INSERT INTO TESTJSON VALUES ( JSON2BSON('{Name:"George"}') )


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
Back to Top

Manipulating JSON Documents
The last section described how we can insert and retrieve entire JSON documents from a column in a table. This
section will explore a number of functions that allow access to individual fields within the JSON document. These
functions are:

JSON_VAL - Extracts data from a JSON document into SQL data types
JSON_TABLE - Returns a table of values for a document that has array types in it
JSON_TYPE - Returns documents that have a field with a specific data type (like array, or date)
JSON_LEN - Returns the count of elements in an array type inside a document
JSON_GET_POS_ARR_INDEX - Retrieve the index of a value within an array type in a document

Our examples in this section will require a couple of tables to be created.
Sample JSON Table Creation
The following SQL will load the JSON_EMP table with a number of JSON objects. These records are modelled
around the SAMPLE database JSON_EMP table.
Can you write Python code for it?


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
Back to Top

JSON NUMBERS and FLOATING POINT
JSON numbers are recognized by Db2 when there is a decimal point in the value.
Floating point values are recognized using the Exx specifier after the number
which represents the power of 10 that needs to be applied to the base value.
For instance, 1.0E01 is the value 10.
The JSON type for numbers is 1, whether it is in floating point format or decimal format.
The SQL statement below inserts a salary into the table (using the
standard decimal place notation).
Can you write Python code for it?

%%sql -q
DELETE FROM SANDBOX;

INSERT INTO SANDBOX VALUES
  JSON2BSON('{"salary":92342.20}');


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
Back to Top

Path Statement Requirements
All of the Db2 JSON functions have been placed into the SYSTOOLS schema. This means that in order to execute
any of these commands, you must prefix the command with SYSTOOLS, as in SYSTOOLS.JSON2BSON. In order to
remove this requirement, you must update the CURRENT PATH value to include SYSTOOLS as part of it. The
SQL below will tell you what the current PATH is.
Can you write Python code for it?


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
Back to Top

Retrieving Data from a BSON Document
Now that we have inserted some JSON data into a table, this section will explore
the use of the JSON_VAL function to retrieve individual fields from the documents.
This built-in function will return a value from a document in a format that you specify.
The ability to dynamically change the returned data type is extremely important when we
examine index creation in another section.
The JSON_VAL function has the format:
Python
JSON_VAL(document, field, type)
JSON_VAL takes 3 arguments:
- document - BSON document
- field - The field we are looking for (search path)
- type - The return type of data being returned
The search path and type must be constants - they cannot be variables so their
use in user-defined functions is limited to using constants.
A typical JSON record will contain a variety of data types and structures as
illustrated by the following record from the JSON_EMP table.
Python
  {
    "empno":"200170",
    "firstnme":"KIYOSHI",
    "midinit":"",
    "lastname":"YAMAMOTO",
    "workdept":"D11",
    "phoneno":[2890],
    "hiredate":"09/15/2005",
    "job":"DESIGNER",
    "edlevel":16,
    "sex":"M",
    "birthdate":"01/05/1981",
    "pay": {
      "salary":64680.00,
      "bonus":500.00,
      "comm":1974.00
    }
  }
There are number of fields with different formats, including strings (firstnme),
integers (edlevel), decimal (salary), date (hiredate), a number array (phoneno),
and a structure (pay). JSON data can consist of nested objects, arrays and very
complex structures. The format of a JSON object is checked when using the JSON2BSON
function and an error message will be issued if it does not conform to the
JSON specification.
The JSON_VAL function needs to know how to return the data type back from the JSON
record, so you need to specify what the format should be. The possible formats are:
| Code  | Format                                        |
|:------|:----------------------------------------------|
| n    ...
Can you write Python code for it?

%%sql
SELECT trim(JSON_VAL(EMP_DATA,'lastname','s:40')),
       JSON_VAL(EMP_DATA,'pay.salary','f')
  FROM JSON_EMP
WHERE
  JSON_VAL(EMP_DATA,'empno','s:6') = '200170'


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
Back to Top

Simplifying JSON SQL Inserts and Retrieval
From a development perspective, you always need to convert
documents to and from JSON using the BSON2JSON and JSON2BSON
functions. There are ways to hide these functions from an
application and simplify some of the programming.
One approach to simplifying the conversion of documents
between formats is to use INSTEAD OF triggers. These
triggers can intercept transactions before they are applied
to the base tables. This approach requires that we create a
view on top of an existing table.
The first step is to create the base table with two copies
of the JSON column. One will contain the original JSON
character string while the second will contain the converted
BSON. For this example, the JSON column will be called INFO,
and the BSON column will be called BSONINFO. The use of two
columns containing JSON would appear strange at first. The
reason for the two columns is that Db2 expects the BLOB
column to contain binary data. You cannot insert a character
string (JSON) into the BSON column without converting it
first. Db2 will raise an error so the JSON column is there
to avoid an error while the conversion takes place.
From a debugging perspective, we can keep both the CLOB and
BLOB values in this table if we want. The trigger will set
the JSON column to null after the BSON column has been
populated.
Can you write Python code for it?

%%sql -q
DROP TABLE BASE_EMP_TXS;

CREATE TABLE BASE_EMP_TXS (
  SEQNO    INT NOT NULL GENERATED ALWAYS AS IDENTITY,
  INFO     VARCHAR(4000),
  BSONINFO BLOB(4000) INLINE LENGTH 4000
);


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
Back to Top

JSON Strings
For character strings, you must specify what the maximum
length is. This example will return the size of the lastname
field as 10 characters long.
Can you write Python code for it?

%sql SELECT JSON_VAL(EMP_DATA, 'lastname', 's:10') FROM JSON_EMP


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
By default, only the first number of an array is returned
when you use JSON_VAL. However, there will be situations
where you do want to return all the values in an array. This
is where the JSON_TABLE function must be used.
The format of the JSON_TABLE function is:
Python
   JSON_TABLE(document, field, type)
The arguments are:

document - BSON document
field - The field we are looking for
type - The return type of data being returned

JSON_TABLE returns two columns: Type and Value. The type
is one of a possible 18 values found in the table below. The
Value is the actual contents of the field.
ID  |   TYPE             |      ID  |   TYPE
---:| :------------------| -------: | :---------------------
 1  |   Double           |      10  |   Null
 2  |   String           |      11  |   Regular Expression
 3  |   Object           |      12  |   Future use
 4  |   Array            |      13  |   JavaScript
 5  |   Binary data      |      14  |   Symbol
 6  |   Undefined        |      15  |   Javascript (with scope)
 7  |   Object id        |      16  |   32-bit integer
 8  |   Boolean          |      17  |   Timestamp
 9  |   Date             |      18  |   64-bit integer
The TYPE field is probably something you wouldn't require
as part of your queries since you are already specifying the
return type in the function.
The format of the JSON_TABLE function is like JSON_VAL
except that it returns a table of values. You must use this
function as part of FROM clause and a table function
specification. For example, to return the contents of the
phone extension array for just one employee (000230) we can
use the following JSON_TABLE function.
Can you write Python code for it?

%%sql
SELECT PHONES.TYPE, CAST(PHONES.VALUE AS VARCHAR(10)) AS VALUE
  FROM JSON_EMP E,
       TABLE( JSON_TABLE(E.EMP_DATA,'phoneno','i') ) AS PHONES
WHERE JSON_VAL(E.EMP_DATA,'empno','s:6') = '000010'


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
Db2 can now use the index to retrieve the record and the following plot shows the increased throughput.
Can you write Python code for it?

%%sql -pb
WITH RESULTS(RUN, RESULT) AS (
    VALUES ('No Index',:noindex),('With Index',:withindex)
)
SELECT * FROM RESULTS


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
Finally, the DELETE trigger will just remove the row.
Can you write Python code for it?

%%sql -d
CREATE OR REPLACE TRIGGER D_EMP_TX
  INSTEAD OF DELETE ON EMP_TXS
  REFERENCING OLD AS OLD_TXS
  FOR EACH ROW MODE DB2SQL
BEGIN ATOMIC
  DELETE FROM BASE_EMP_TXS
     WHERE
       BASE_EMP_TXS.SEQNO = OLD_TXS.SEQNO;
END
@


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
If you need to access a specific array element in a field, you can use the "dot"
notation after the field name. The first element starts at zero. If we select
the 2nd element (.1) all the employees that have a second extension will have a
value retrieved while the ones who don't will have a null value.
Back to Top

Retrieving Structured Fields
Structured fields are retrieved using the same dot notation as arrays.
The field is specified by using the "field.subfield" format and these fields can be
an arbitrary number of levels deep.
The pay field in the employee record is made up of three additional fields.
Python
"pay": {
   "salary":64680.00,
   "bonus":500.00,
   "comm":1974.00
}
To retrieve these three fields, you need to explictly name them since
retrieving pay alone will not work.
Can you write Python code for it?

%%sql
SELECT JSON_VAL(EMP_DATA,'pay.salary','i'),
       JSON_VAL(EMP_DATA,'pay.bonus','i'),
       JSON_VAL(EMP_DATA,'pay.comm','i')
  FROM JSON_EMP
WHERE
  JSON_VAL(EMP_DATA,'empno','s:6') = '200170'


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
If you specify ":na" after the type specifier, you will get an error if the field
is an array type. Hopefully you already know the format of your JSON data and can
avoid having to check to see if arrays exist. What this statement will tell you is
that one of the records you were attempting to retrieve was an array type. In fact,
all the phone extensions are being treated as array types even though they have only
one value in many cases.
Can you write Python code for it?

%sql SELECT JSON_VAL(EMP_DATA, 'phoneno', 'i:na') FROM JSON_EMP


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
If you want to manipulate the BSON directly (say change the employee number),
you need to refer to the BASE table instead.
Can you write Python code for it?

%%sql
UPDATE BASE_EMP_TXS
  SET BSONINFO = JSON_UPDATE(BSONINFO,
    '{$set: {"empno":"111111"}}')
  WHERE SEQNO = 1


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
Numeric data can be retrieved in either number (n) formant, integer (i - note that
you will get truncation), or floating point (f).
Can you write Python code for it?

%%sql
SELECT JSON_VAL(DATA,'salary','n') AS DECIMAL,
       JSON_VAL(DATA,'salary','i') AS INTEGER,
       JSON_VAL(DATA,'salary','f') AS FLOAT
FROM SANDBOX


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
One thing that you should note is that the JSON that is retrieved has been modified slightly so that
all of the values have quotes around them to avoid any ambiguity. Note that we didn't necessarily require them
when we input the data. For instance, our original JSON document what was inserted looked like this:
Python
{Name:"George"}
What gets returned is slightly different, but still considered to be the same JSON document. You must ensure that the
naming of any fields is consistent between documents. "Name", "name", and "NAME" are all considered different fields. One option is to use lowercase field names, or to use camel-case (first letter is capitalized) in all of your field definitions. The important thing is to keep the naming consistent so you can find the fields in the document.
Python
{"Name":"George"}
Back to Top

BSON_VALIDATE: Checking the format of a JSON document
There is no validation done against the contents of a BLOB column which contains JSON data.
As long as the JSON object is under program control and you are using the JSON functions,
you are probably not going to run across problems with the data. You should probably stick to either Db2 JSON
functions to access your JSON columns or the db2nosql (MongoDB syntax).
In the event you believe that a document is corrupted for some reason, you can use the BSON_VALIDATE to make sure it
is okay (or not!). The function will return a value of 1 if the record is okay, or a zero otherwise. The one row
that we have inserted into the TESTJSON table should be okay.
Can you write Python code for it?

%%sql
SELECT BSON_VALIDATE(JSON_FIELD) FROM TESTJSON


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
Only a subset of the results is shown above, but you will
see that there are multiple lines for employees who have
more than one extension.
The results of a TABLE function must be named (AS ...) if
you need to refer to the results of the TABLE function in
the SELECT list or in other parts of the SQL.
You can use other SQL operators to sort or organize the
results. For instance, we can use the ORDER BY operator to
find out which employees have the same extension. Note how
the TABLE function is named PHONES and the VALUES column is
renamed to PHONE.
Can you write Python code for it?

%%sql
SELECT JSON_VAL(E.EMP_DATA, 'lastname', 's:10') AS LASTNAME,
       CAST (PHONES.VALUE AS VARCHAR(10)) AS PHONE
  FROM JSON_EMP E,
       TABLE( JSON_TABLE(E.EMP_DATA,'phoneno','i') ) AS PHONES
ORDER BY PHONE


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
Querying the data type of this field using JSON_VAL will return a value of 9 (date type).
Can you write Python code for it?

%sql SELECT JSON_TYPE(DATA,'today',2048) FROM SANDBOX


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
Rerunning the SQL results in the following performance:
Can you write Python code for it?

withindex = %sql -t \
SELECT JSON_VAL(EMP_DATA, 'lastname', 's:20') AS LASTNAME \
  FROM JSON_EMP \
WHERE  JSON_VAL(EMP_DATA, 'empno', 's:6') = '000010' \


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
Retrieving the document shows that the lastname field has now been added to the record.
Can you write Python code for it?

%sql -j SELECT BSON2JSON(DATA) FROM SANDBOX


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
Selecting from the EMP_TXS view will return the JSON in a readable format:
Can you write Python code for it?


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
Since we have an integer in the JSON field, we also have the option of returning
the value as a floating-point number (f) or as a decimal number (n). Either of
these options will work with integer values.
Can you write Python code for it?

%%sql
SELECT JSON_VAL(DATA,'count','n') AS DECIMAL,
       JSON_VAL(DATA,'count','f') AS FLOAT
FROM SANDBOX


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
Specifying the 'l' flag will make the data be returned properly.
Can you write Python code for it?

%sql SELECT JSON_VAL(DATA,'count','l') FROM SANDBOX


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
Table of Contents

Db2 JSON Functions and Registration
Path Requirements

* [Programming with Db2 JSON](#programming)
  * [Inserting JSON Document (JSON2BSON)](#json2bson)
  * [Retrieving a JSON Document (BSON2JSON)](#bson2json)
  * [Checking a JSON Document for Consistency](#bsonvalidate)

* [Manipulating JSON Documents](#manipulating)
  * [Retrieving Data](#retrieving)
  * [Retrieving Individual Fields](#atomic)
  * [Accessing Elements in an Array](#array)
  * [Accessing Portions of a Structure](#structures)
  * [Determining if a Field is Null](#nulls)
  * [How to Join JSON Fields](#joins)

* [JSON Data Types](#datatypes)
  * [INTEGERS and BIGINT](#integers)
  * [NUMBERS and FLOATING POINT](#numbers)
  * [BOOLEAN Values](#boolean)
  * [DATE, TIME, and TIMESTAMPS](#date)
  * [Strings](#strings)

* [Dealing with Arrays](#arrays)
  * [Retrieving Array Values as a Table](#table)
  * [Determining the Size of an Array](#length)
  * [Searching an Array for a Value](#getpos)

* [Updating JSON Documents](#updating)
* [Indexing Fields in JSON Documents](#indexing)
* [How to Simplify JSON Inserts and Retrieval](#simple)

[Back to Top](#top)

# Db2 JSON Functions

There is one built-in Db2 JSON function and a number of other functions that must be registered within
Db2 before they can be used. The names of the functions and their purpose are described below.

  - **JSON_VAL** - Extracts data from a JSON document into SQL data types
  - **JSON_TABLE** - Returns a table of values for a document that has array types in it
  - **JSON_TYPE** - Returns documents that have a field with a specific data type (like array, or date)
  - **JSON_LEN** - Returns the count of elements in an array type inside a document
  - **BSON2JSON** - Convert BSON formatted document into JSON strings
  - **JSON2BSON** - Convert JSON strings into a BSON document format
  - **JSON_GET_POS_ARR_INDEX** - Retrieve the index of a value within an array type in a document
  - **JSON_UPDATE** - Update a particular...
Can you write Python code for it?

%%sql -q
CREATE FUNCTION SYSTOOLS.JSON_TABLE(
  INJSON BLOB(16M), INELEM VARCHAR(2048), RETTYPE VARCHAR(100))
  RETURNS TABLE(TYPE INTEGER, VALUE VARCHAR(2048))
  LANGUAGE C
  PARAMETER STYLE SQL
  PARAMETER CCSID UNICODE
  NO SQL
  NOT FENCED
  DETERMINISTIC
  NO EXTERNAL ACTION
  DISALLOW PARALLEL
  SCRATCHPAD 2048
  EXTERNAL NAME 'db2json!jsonTable';

CREATE FUNCTION SYSTOOLS.JSON_TYPE(
  INJSON BLOB(16M), INELEM VARCHAR(2048), MAXLENGTH INTEGER)
  RETURNS INTEGER
  LANGUAGE C
  PARAMETER STYLE SQL
  PARAMETER CCSID UNICODE
  NO SQL
  NOT FENCED
  DETERMINISTIC
  ALLOW PARALLEL
  RETURNS NULL ON NULL INPUT
  NO EXTERNAL ACTION
  EXTERNAL NAME 'db2json!jsonType';

CREATE FUNCTION SYSTOOLS.JSON_LEN(
  INJSON BLOB(16M), INELEM VARCHAR(2048))
  RETURNS INTEGER
  LANGUAGE C
  PARAMETER STYLE SQL
  PARAMETER CCSID UNICODE
  NO SQL
  NOT FENCED
  DETERMINISTIC
  ALLOW PARALLEL
  NO EXTERNAL ACTION
  SCRATCHPAD 2048
  EXTERNAL NAME 'db2json!jsonLen';

CREATE FUNCTION SYSTOOLS.BSON2JSON(INBSON BLOB(16M)) RETURNS CLOB(16M)
  LANGUAGE C
  PARAMETER STYLE SQL
  PARAMETER CCSID UNICODE
  NO SQL
  NOT FENCED
  DETERMINISTIC
  ALLOW PARALLEL
  NO EXTERNAL ACTION
  SCRATCHPAD 2048
  EXTERNAL NAME 'db2json!jsonBsonToJson';

CREATE FUNCTION SYSTOOLS.JSON2BSON(INJSON CLOB(16M)) RETURNS BLOB(16M)
  LANGUAGE C
  PARAMETER STYLE SQL
  PARAMETER CCSID UNICODE
  NO SQL
  NOT FENCED
  DETERMINISTIC
  ALLOW PARALLEL
  NO EXTERNAL ACTION
  SCRATCHPAD 2048
  EXTERNAL NAME 'db2json!jsonToBson';

CREATE FUNCTION SYSTOOLS.JSON_GET_POS_ARR_INDEX(
  INJSON BLOB(16M), QUERY VARCHAR(32672) FOR BIT DATA)
  RETURNS INTEGER
  LANGUAGE C
  PARAMETER STYLE SQL
  PARAMETER CCSID UNICODE
  NO SQL
  NOT FENCED
  DETERMINISTIC
  ALLOW PARALLEL
  CALLED ON NULL INPUT
  NO EXTERNAL ACTION
  SCRATCHPAD 2048
  EXTERNAL NAME 'db2json!jsonGetPosArrIndex';

CREATE FUNCTION SYSTOOLS.JSON_UPDATE(
  INJSON BLOB(16M), INELEM VARCHAR(32672))
  RETURNS BLOB(16M)
  LANGUAGE C
  PARAMETER STYLE SQL
  PARAMETER CCSID UNICODE
  NO SQL
  NOT FENCED
 ...


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
The BSON_VALIDATE should return a zero for this particular row since it is not a valid BSON document.
Can you write Python code for it?


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
The JSON_TYPE function will verify that this is a floating point field (Type=1).
Can you write Python code for it?

%sql SELECT JSON_TYPE(DATA,'salary',2048) AS TYPE FROM SANDBOX


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
The JSON_TYPE function will verify that this is an integer field (Type=16).
Can you write Python code for it?

%%sql
SELECT JSON_TYPE(DATA,'count',2048) AS TYPE
  FROM SANDBOX


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
The base table only contains the BSON but the view translates the value back into a readable format.
An update statement that replaces the entire string works as expected.
Can you write Python code for it?

%%sql -j
UPDATE EMP_TXS SET INFO = '{"empno":"000010"}' WHERE SEQNO = 1;

SELECT INFO FROM EMP_TXS;


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
The floating-point value can be retrieved as a number, integer, or floating point value.
Can you write Python code for it?


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
The following SQL will generate a list of data types and field names found within this document.
Can you write Python code for it?

%%sql
SELECT 'STRING',JSON_TYPE(DATA, 'string', 2048) FROM TYPES
UNION ALL
SELECT 'INTEGER',JSON_TYPE(DATA, 'integer', 2048) FROM TYPES
UNION ALL
SELECT 'NUMBER',JSON_TYPE(DATA, 'number', 2048) FROM TYPES
UNION ALL
SELECT 'DATE',JSON_TYPE(DATA, 'date', 2048) FROM TYPES
UNION ALL
SELECT 'BOOLEAN', JSON_TYPE(DATA, 'boolean', 2048) FROM TYPES
UNION ALL
SELECT 'ARRAY', JSON_TYPE(DATA, 'array', 2048) FROM TYPES
UNION ALL
SELECT 'OBJECT', JSON_TYPE(DATA, 'object', 2048) FROM TYPES


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
The following command will time the select statement.
Can you write Python code for it?

noindex = %sql -t \
SELECT JSON_VAL(EMP_DATA, 'lastname', 's:20') AS LASTNAME \
  FROM JSON_EMP \
WHERE  JSON_VAL(EMP_DATA, 'empno', 's:6') = '000010'


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
The following sections will show how we can get atomic (non-array) types out of
a JSON document. We are not going to be specific which documents we want, other
than what field we want to retrieve.
A temporary table called SANDBOX is used throughout these examples:
Can you write Python code for it?

%%sql -q
DROP TABLE SANDBOX;

CREATE TABLE SANDBOX (DATA BLOB(4000) INLINE LENGTH 4000);


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
The next SQL statement will create a field with a BIGINT size. Note that we don't
need to specify anything other than have a very big number!
Can you write Python code for it?

%%sql
DELETE FROM SANDBOX;

INSERT INTO SANDBOX VALUES
  JSON2BSON('{"count":94123512223422}');


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
The results contain 40 employees who have a middle initial field, and two that do not.
The results can be misleading because an employee can have the midinit field defined,
but no value assigned to it:
Python
{
 "empno":"000120",
 "firstnme":"SEAN",
 "midinit":"",
 "lastname":"O''CONNELL",...
}
In this case, the employee does not have a middle name, but the field is present.
To determine whether an employee does not have a middle name, you will need to check
for a NULL value (the field does not exist) or the field exists and contains no content.
Can you write Python code for it?

%%sql
SELECT COUNT(*) FROM JSON_EMP
  WHERE JSON_VAL(EMP_DATA,'midinit','s:40') = '' OR
        JSON_VAL(EMP_DATA,'midinit','u') IS NULL


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
To force the value to return just the time portion, either
store the data as a time value (HH:MM:SS) string or store a
timestamp and use the TIME function to extract just that
portion of the timestamp.
Can you write Python code for it?

%sql SELECT TIME(JSON_VAL(DATA,'today','ts')) FROM SANDBOX


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
To retrieve the value, we can ask that it be formatted as an integer or number.
Can you write Python code for it?

%%sql
SELECT JSON_VAL(DATA,'valid','n') AS TRUE_DECIMAL,
       JSON_VAL(DATA,'valid','i') AS TRUE_INTEGER,
       JSON_VAL(DATA,'invalid','n') AS FALSE_DECIMAL,
       JSON_VAL(DATA,'invalid','i') AS FALSE_INTEGER
FROM SANDBOX


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
To use INSTEAD OF triggers, a view needs to be created on
top of the base table. Note that we explicitly use the
SYSTOOLS schema to make sure we are getting the correct
function used here.
Can you write Python code for it?

%%sql
CREATE OR REPLACE VIEW EMP_TXS AS
  (SELECT SEQNO, BSON2JSON(BSONINFO) AS INFO FROM BASE_EMP_TXS)


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
Using the 'd' specification will return the value as a date.
Can you write Python code for it?

%sql SELECT JSON_VAL(DATA,'today','d') FROM SANDBOX


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
We can check the count of records to make sure that 42 employees were added to our table.
Can you write Python code for it?


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
We can check to see that the data is stored in the document as a BIGINT by
using the JSON_TYPE function.
Can you write Python code for it?

%sql SELECT JSON_TYPE(DATA,'count',2048) FROM SANDBOX


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
We will double-check what type the field is in the JSON record.
Can you write Python code for it?

%sql SELECT JSON_TYPE(DATA,'valid',2048) AS TYPE FROM SANDBOX


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
What about timestamps? If you decide to store a timestamp into a field, you can
retrieve it in a variety of ways. This first set of SQL statements will retrieve
it as a string.
Can you write Python code for it?

%%sql
DELETE FROM SANDBOX;

INSERT INTO SANDBOX VALUES
  JSON2BSON('{"today":"' || VARCHAR(NOW()) || '"}');

SELECT JSON_VAL(DATA,'today','s:30') FROM SANDBOX;


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
You can also ask for the timestamp value by using the 'ts'
specification. Note that you can't get just the time portion
unless you use a SQL function to cast it.
Can you write Python code for it?

%sql SELECT JSON_VAL(DATA,'today','ts') FROM SANDBOX


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
You can even find out how many people are sharing
extensions! The HAVING clause tells Db2 to only return
groupings where there are more than one employee with the
same extension.
Can you write Python code for it?

%%sql
SELECT CAST(PHONES.VALUE AS VARCHAR(10)) AS PHONE, COUNT(*) AS COUNT
  FROM JSON_EMP E,
       TABLE( JSON_TABLE(E.EMP_DATA,'phoneno','i') ) AS PHONES
GROUP BY PHONES.VALUE HAVING COUNT(*) > 1
ORDER BY PHONES.VALUE


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
You must specify a length for the 's' parameter otherwise
you will get an error from the function. If the size of the
character string is too large to return, then the function
will return a null value for that field.
Can you write Python code for it?

%sql SELECT JSON_VAL(EMP_DATA, 'lastname', 's:8') FROM JSON_EMP


I am working on the file "Db2 JSON Features.ipynb".
The first task is:
You need to ensure that the data types from both JSON functions are compatible for
the join to work properly. In this case, the department number and the work department
are both returned as 3-byte character strings. If you decided to use integers
instead or a smaller string size, the join will not work as expected because
the conversion will result in truncated or NULL values.
If you plan on doing joins between JSON objects, you may want to consider creating
indexes on the documents to speed up the join process. More information on the use
of indexes is found at the end of this chapter.
Back to Top

JSON Data Types
If you are unsure of what data type a field contains, you can use the the JSON_TYPE
function to determine the type before retrieving the field.
The JSON_TYPE function has the format:
Python
ID = JSON_TYPE(document, field, 2048)
JSON_TYPE takes 3 arguments:
- document - BSON document
- field - The field we are looking for (search path)
- search path size - 2048 is the required value
The 2048 specifies the maximum length of the field parameter and should be
left at this value.
When querying the data types within a JSON document, the following values are returned.
ID  |   TYPE             |      ID  |   TYPE
---:| :------------------| -------: | :---------------------
 1  |   Double           |      10  |   Null
 2  |   String           |      11  |   Regular Expression
 3  |   Object           |      12  |   Future use
 4  |   Array            |      13  |   JavaScript
 5  |   Binary data      |      14  |   Symbol
 6  |   Undefined        |      15  |   Javascript (with scope)
 7  |   Object id        |      16  |   32-bit integer
 8  |   Boolean          |      17  |   Timestamp
 9  |   Date             |      18  |   64-bit integer
The next SQL statement will create a table with standard types within it.
Can you write Python code for it?

%%sql -q
DROP TABLE TYPES;

CREATE TABLE TYPES
  (DATA BLOB(4000) INLINE LENGTH 4000);

INSERT INTO TYPES VALUES
  JSON2BSON(
  '{
  "string"    : "string",
  "integer"   : 1,
  "number"    : 1.1,
  "date"      : {"$date": "2016-06-20T13:00:00"},
  "boolean"   : true,
  "array"     : [1,2,3],
  "object"    : {type: "main", phone: [1,2,3]}
  }');


I am working on the file "Db2 Jupyter Extensions Tutorial.ipynb".
The first task is:
A special note regarding the output from a SELECT statement. If the SQL statement is the last line of a block, the results will be displayed by default (unless you assigned the results to a variable). If the SQL is in the middle of a block of statements, the results will not be displayed. To explicitly display the results you must use the display function (or pDisplay if you have imported another library like pixiedust which overrides the pandas display function).
Can you write Python code for it?

# Set the maximum back
%sql option maxrows 10
%sql values 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15


I am working on the file "Db2 Jupyter Extensions Tutorial.ipynb".
The first task is:
Adding the -j option to the %sql (or %%sql) command will format the first column of a return set to better
display the structure of the document. Note that if your answer set has additional columns associated with it, they will not be displayed in this format.
Can you write Python code for it?


I am working on the file "Db2 Jupyter Extensions Tutorial.ipynb".
The first task is:
An assignment statement to a variable will result in an equivalent Python dictionary type being created. Note that we must use the raw -j flag to make sure we only get the data and not a data frame.
Can you write Python code for it?

x = %sql -j SELECT * FROM SHOWJSON
print("First Name is " + x[0]["firstname"] + " and the last name is " + x[0]['lastname'])


I am working on the file "Db2 Jupyter Extensions Tutorial.ipynb".
The first task is:
Db2 has builtin functions for dealing with JSON objects. There is another Jupyter notebook which goes through this in detail. Rather than using those functions, the following code will create a Db2 table with a string column that will contain the contents of this JSON record.
Can you write Python code for it?

%%sql
DROP TABLE SHOWJSON;
CREATE TABLE SHOWJSON (INJSON VARCHAR(256));


I am working on the file "Db2 Jupyter Extensions Tutorial.ipynb".
The first task is:
Finally, for a line chart, the x contains the labels and the y values are used.
Can you write Python code for it?


I am working on the file "Db2 Jupyter Extensions Tutorial.ipynb".
The first task is:
For a pie chart, the first column is used to label the slices, while the data comes from the second column.
Can you write Python code for it?


I am working on the file "Db2 Jupyter Extensions Tutorial.ipynb".
The first task is:
If you are using a single statement then there is no need to use a delimiter. However, if you are combining a number of commands then you must use the semicolon.
Can you write Python code for it?

%%sql
DROP TABLE STUFF;
CREATE TABLE STUFF (A INT);
INSERT INTO STUFF VALUES
  1,2,3;
SELECT * FROM STUFF;


I am working on the file "Db2 Jupyter Extensions Tutorial.ipynb".
The first task is:
If you know that these errors may occur you can silence them with the -q option.
Can you write Python code for it?

%%sql -q
DROP TABLE TABLE_NOT_FOUND;
DROP TABLE TABLE_SPELLED_WRONG;


I am working on the file "Db2 Jupyter Extensions Tutorial.ipynb".
The first task is:
If you retrieve two columns of information, the first column is used for the labels (X axis or pie slices) and
the second column contains the data.
Can you write Python code for it?


I am working on the file "Db2 Jupyter Extensions Tutorial.ipynb".
The first task is:
If you use the same syntax as Db2, Python will remove the quote in the string! It interprets this as two strings (O and CONNELL) being concatentated together. That probably isn't what you want! So the safest approach is to use double quotes around your string when you assign it to a variable. Then you can use the variable in the SQL statement as shown in the following example.
Can you write Python code for it?

lastname = "O'CONNELL"
%sql -e SELECT * FROM EMPLOYEE WHERE LASTNAME = :lastname


I am working on the file "Db2 Jupyter Extensions Tutorial.ipynb".
The first task is:
If you want to iterate over all of the rows and columns, you could use the following Python syntax instead of
creating a for loop that goes from 0 to 41.
Can you write Python code for it?

for row in rows:
    line = ""
    for col in row:
        line = line + str(col) + ","
    print(line)


I am working on the file "Db2 Jupyter Extensions Tutorial.ipynb".
The first task is:
If you want to retrieve the data from a column that contains JSON records, you must use the -j flag to insert the contents back into a variable.
Can you write Python code for it?


I am working on the file "Db2 Jupyter Extensions Tutorial.ipynb".
The first task is:
JSON Formatting
Db2 supports querying JSON that is stored in a column within a table. Standard output would just display the
JSON as a string. For instance, the following statement would just return a large string of output.
Can you write Python code for it?


I am working on the file "Db2 Jupyter Extensions Tutorial.ipynb".
The first task is:
JSON fields can be inserted into Db2 columns using Python dictionaries. This makes the input and output of JSON fields much simpler. For instance, the following code will create a Python dictionary which is similar to a JSON record.
Can you write Python code for it?

employee = {
    "firstname" : "John",
    "lastname" : "Williams",
    "age" : 45
}


I am working on the file "Db2 Jupyter Extensions Tutorial.ipynb".
The first task is:
Notice how the string constant was updated to contain two quotes when inserted into the SQL statement. This is done automatically by the %sql magic command, so there is no need to use the two single quotes when assigning a string to a variable. However, you must use the two single quotes when using constants in a SQL statement.
Builtin Variables
There are 5 predefined variables defined in the program:

database - The name of the database you are connected to
uid - The userid that you connected with
hostname = The IP address of the host system
port - The port number of the host system
max - The maximum number of rows to return in an answer set

Theses variables are all part of a structure called _settings. To retrieve a value, use the syntax:
python
db = _settings['database']
There are also 3 variables that contain information from the last SQL statement that was executed.

sqlcode - SQLCODE from the last statement executed
sqlstate - SQLSTATE from the last statement executed
sqlerror - Full error message returned on last statement executed

You can access these variables directly in your code. The following code segment illustrates the use of the SQLCODE variable.
Can you write Python code for it?

empnos = ['000010','999999']
for empno in empnos:
    ans1 = %sql -r SELECT SALARY FROM EMPLOYEE WHERE EMPNO = :empno
    if (sqlcode != 0):
        print("Employee "+ empno + " left the company!")
    else:
        print("Employee "+ empno + " salary is " + str(ans1[1][0]))


I am working on the file "Db2 Jupyter Extensions Tutorial.ipynb".
The first task is:
Options
There are two options that can be set with the %sql command. These options are:
- MAXROWS n - The maximum number of rows that you want to display as part of a SQL statement. Setting MAXROWS to -1 will return all output, while maxrows of 0 will suppress all output.
- RUNTIME n - When using the timer option on a SQL statement, the statement will execute for n number of seconds. The result that is returned is the number of times the SQL statement executed rather than the execution time of the statement. The default value for runtime is one second, so if the SQL is very complex you will need to increase the run time.
To set an option use the following syntax:
%sql option option_name value option_name value ....
The following example sets all three options:
%sql option maxrows 100 runtime 2
The values will be saved between Jupyter notebooks sessions.
Connections to Db2
Before any SQL commands can be issued, a connection needs to be made to the Db2 database that you will be using. The connection can be done manually (through the use of the CONNECT command), or automatically when the first %sql command is issued.
The Db2 magic command tracks whether or not a connection has occured in the past and saves this information between notebooks and sessions. When you start up a notebook and issue a command, the program will reconnect to the database using your credentials from the last session. In the event that you have not connected before, the system will prompt you for all the information it needs to connect. This information includes:

Database name (SAMPLE)
Hostname - localhost (enter an IP address if you need to connect to a remote server)
PORT - 50000 (this is the default but it could be different)
Userid - DB2INST1
Password - No password is provided so you have to enter a value
Maximum Rows - 10 lines of output are displayed when a result set is returned

There will be default values presented in the panels that you can accept, or enter your own values. All of the information will be stored in the...
Can you write Python code for it?


I am working on the file "Db2 Jupyter Extensions Tutorial.ipynb".
The first task is:
SQL Character Strings
Character strings require special handling when dealing with Db2. The single quote character ' is reserved for delimiting string constants, while the double quote " is used for naming columns that require special characters. You cannot use the double quote character to delimit strings that happen to contain the single quote character. What Db2 requires you do is placed two quotes in a row to have them interpreted as a single quote character. For instance, the next statement will select one employee from the table who has a quote in their last name: O'CONNELL.
Can you write Python code for it?

%sql SELECT * FROM EMPLOYEE WHERE LASTNAME = 'O''CONNELL'


I am working on the file "Db2 Jupyter Extensions Tutorial.ipynb".
The first task is:
SQL output will not be suppressed, so the following command will still show the results.
Can you write Python code for it?

%%sql -q
DROP TABLE TABLE_NOT_FOUND;
DROP TABLE TABLE_SPELLED_WRONG;
VALUES 1,2,3;


I am working on the file "Db2 Jupyter Extensions Tutorial.ipynb".
The first task is:
Sample Data
Many of the Db2 notebooks depend on two of the tables that are found in the SAMPLE database. Rather than
having to create the entire SAMPLE database, this option will create and populate the EMPLOYEE and
DEPARTMENT tables in your database. Note that if you already have these tables defined, they will not be dropped.
Can you write Python code for it?


I am working on the file "Db2 Jupyter Extensions Tutorial.ipynb".
The first task is:
Selecting from this table will show that the data has been inserted as a string.
Can you write Python code for it?


I am working on the file "Db2 Jupyter Extensions Tutorial.ipynb".
The first task is:
The delimiter change will only take place for the statements following the %%sql command. Subsequent cells
in the notebook will still use the semicolon. You must use the -d option for every cell that needs to use the
semicolon in the script.
Limiting Result Sets
The default number of rows displayed for any result set is 10. You have the option of changing this option when initially connecting to the database. If you want to override the number of rows display you can either update
the control variable, or use the -a option. The -a option will display all of the rows in the answer set. For instance, the following SQL will only show 10 rows even though we inserted 15 values:
Can you write Python code for it?


I am working on the file "Db2 Jupyter Extensions Tutorial.ipynb".
The first task is:
The field can be inserted into a character column (or BSON if you use the JSON functions) by doing a direct variable insert.
Can you write Python code for it?

%%sql -q
DROP TABLE SHOWJSON;
CREATE TABLE SHOWJSON(JSONIN VARCHAR(128));


I am working on the file "Db2 Jupyter Extensions Tutorial.ipynb".
The first task is:
The following SQL will plot the number of employees per department.
Can you write Python code for it?

%%sql -pb
SELECT WORKDEPT, COUNT(*)
  FROM EMPLOYEE
GROUP BY WORKDEPT


I am working on the file "Db2 Jupyter Extensions Tutorial.ipynb".
The first task is:
The number of rows in the result set can be determined by using the length function and subtracting one for the header row.
Can you write Python code for it?


I am working on the file "Db2 Jupyter Extensions Tutorial.ipynb".
The first task is:
The same data as a pie chart.
Can you write Python code for it?


I am working on the file "Db2 Jupyter Extensions Tutorial.ipynb".
The first task is:
There is an additional way of capturing the data through the use of the -r flag.
<pre>
var = %sql -r select * from employee
</pre>
Rather than returning a dataframe result set, this option will produce a list of rows. Each row is a list itself. The column names are found in row zero (0) and the data rows start at 1. To access the first column of the first row, you would use var[1][0] to access it.
Can you write Python code for it?

rows = %sql -r select * from employee fetch first 3 rows only
print(rows[1][0])


I am working on the file "Db2 Jupyter Extensions Tutorial.ipynb".
The first task is:
Timing SQL Statements
Sometimes you want to see how the execution of a statement changes with the addition of indexes or other
optimization changes. The -t option will run the statement on the LINE or one SQL statement in the CELL for
exactly one second. The results will be displayed and optionally placed into a variable. The syntax of the
command is:
<pre>
sql_time = %sql -t SELECT * FROM EMPLOYEE
</pre>
For instance, the following SQL will time the VALUES clause.
Can you write Python code for it?


I am working on the file "Db2 Jupyter Extensions Tutorial.ipynb".
The first task is:
To insert the Dictionary (JSON Record) into this Db2 table, you only need to use the variable name as one of the fields being inserted.
Can you write Python code for it?

%sql INSERT INTO SHOWJSON VALUES :customer


I am working on the file "Db2 Jupyter Extensions Tutorial.ipynb".
The first task is:
Variables in %sql Blocks
Python variables can be passed to a %sql line command, and to a %%sql block. For both forms of the %sql command you can pass variables by placing a colon in front of the variable name.
python
%sql SELECT * FROM EMPLOYEE WHERE EMPNO = :empno
The following example illustrates the use of a variable in the SQL.
Can you write Python code for it?

empno = '000010'
%sql SELECT * FROM EMPLOYEE WHERE EMPNO = :empno


I am working on the file "Db2 Jupyter Extensions Tutorial.ipynb".
The first task is:
When timing a statement, no output will be displayed. If your SQL statement takes longer than one second you
will need to modify the runtime options. You can use the %sql option runtime command to change the duration the statement runs.
Can you write Python code for it?

%sql option runtime 5
%sql -t VALUES 1,2,3,4,5,6,7,8,9
%sql option runtime 1


I am working on the file "Db2 Jupyter Extensions Tutorial.ipynb".
The first task is:
You can assign the result set directly to a variable.
Can you write Python code for it?

x = %sql select * from employee fetch first 3 rows only


I am working on the file "Db2 Jupyter Extensions Tutorial.ipynb".
The first task is:
You can doublecheck that the substitution took place by using the -e option which echos the SQL command after substitution.
Can you write Python code for it?

%sql -e SELECT * FROM EMPLOYEE WHERE EMPNO = :empno


I am working on the file "Db2 Jupyter Extensions Tutorial.ipynb".
The first task is:
You will notice that the displayed result will split the visible rows to the first 5 rows and the last 5 rows.
Using the -a option will display all values in a scrollable table.
Can you write Python code for it?


I am working on the file "Db2 Jupyter Macros.ipynb".
The first task is:
A variable can be converted to uppercase by placing the ^ beside the variable name or number.
Can you write Python code for it?

%%sql macro runit
echo The first parameter is {^1}

%sql %runit(Hello There)


I am working on the file "Db2 Jupyter Macros.ipynb".
The first task is:
An empty string also returns a null value.
Can you write Python code for it?


I am working on the file "Db2 Jupyter Macros.ipynb".
The first task is:
Calling runit will display the variable that was set in the first macro.
Can you write Python code for it?


I am working on the file "Db2 Jupyter Macros.ipynb".
The first task is:
Calling the macro will show how the variable names get assigned and used.
Can you write Python code for it?

%sql %showvar(parameter 1, another parameter,message="Hello World")


I am working on the file "Db2 Jupyter Macros.ipynb".
The first task is:
Echo Command
As you already noticed in the previous example, the echo command will display information on the screen. Any text following the command will have variables substituted and then displayed with a green box surrounding it. The following code illustates the use of the command.
Can you write Python code for it?

%%sql macro showecho
echo Here is a message
echo Two lines are shown


I am working on the file "Db2 Jupyter Macros.ipynb".
The first task is:
Exit Command
The exit command will terminate the processing within a macro and not run the generated SQL. You would use this when a condition is not met within the macro (like a missing parameter).
Can you write Python code for it?

%%sql macro showexit
echo This message gets shown
SELECT * FROM EMPLOYEE FETCH FIRST ROW ONLY
exit
echo This message does not get shown


I am working on the file "Db2 Jupyter Macros.ipynb".
The first task is:
Finally, any string that is supplied to the macro will not include the quotes in the variable. The Hello World string will not have quotes when it is displayed:
Can you write Python code for it?


I am working on the file "Db2 Jupyter Macros.ipynb".
The first task is:
Finally, issue the macro with multiple parameters.
Can you write Python code for it?


I am working on the file "Db2 Jupyter Macros.ipynb".
The first task is:
If we remove the -e option, the final SQL will not be shown.
Can you write Python code for it?


I am working on the file "Db2 Jupyter Macros.ipynb".
The first task is:
If you pass an empty value (or if a variable does not exist), a "null" value will be shown.
Can you write Python code for it?


I am working on the file "Db2 Jupyter Macros.ipynb".
The first task is:
If you want to have a message go across multiple lines use the <br> to start a new line.
Can you write Python code for it?

%%sql macro showecho
echo Here is a paragraph. <br> And a final paragraph.

%sql %showecho