Package: scidb-strm
RequirementsSciDB19.11or newer.Apache Arrow5.0.0to11.0.0.Python3.6.x,3.7.x,3.8.x,3.9.x, or3.10.xRequired Python packages:dill
pandas
pyarrowInstallationInstall latest release:pip install scidb-strmInstall development version from GitHub:pip install git+http://github.com/paradigm4/stream.git#subdirectory=py_pkgThe Python library needs to be installed on the SciDB server. The
library needs to be installed on the client as well, if Python code is
to be send from the client to the server.SciDB-Strm Python API and ExamplesOnce installed theSciDB-StrmPython library can be imported withimport scidbstrm. The library provides a high and low-level access
to the SciDBstreamoperator as well as the ability to send Python
code to the SciDB server.High-level access is provided by the functionmap:map(map_fun, finalize_fun=None)Read SciDB chunks. For each chunk, callmap_funand stream its
result back to SciDB. Iffinalize_funis provided, call it after
all the chunks have been processed.See0-iquery.txtfor a succinct example
using themapfunction.See1-map-finalize.pyfor an example
using themapfunction. The Python script has to be copied onto
the SciDB instance.Python code can be send to the SciDB server for execution using
thepack_funcandread_funcfunctions:pack_func(func)Serialize Python function for use asupload_dataininputorloadoperators.read_func()Read and de-serialize function from SciDB.See2-pack-func.pyfor an example of
using thepack_funcandread_funcfunctions.Low-level access is provided by thereadandwritefunctions:read()Read a data chunk from SciDB. Returns a Pandas DataFrame or None.write(df=None)Write a data chunk to SciDB.See3-read-write.pyfor an example using
thereadandwritefunctions. The Python script has to be
copied onto the SciDB instance.A convenience invocation of the Python interpreter is provided inpython_mapvariable and it is set to:python -uc "import scidbstrm; scidbstrm.map(scidbstrm.read_func())"Finally, see4-machine-learning.pyfor a more complex example of going through the steps of using
machine learning (preprocessing, training, and prediction).Debugging Python CodeWhen debugging Python code executed as part of thestreamoperatordo notuse theprintfunction. Thestreamoperator
communicates with the Python process usingstdout. Theprintfunction writes output tostdout. So, using theprintfunction
would interfere with the inter-process communication.Instead, use thedebugfunction provided by the library. The
function formats the arguments as strings and printed them all out
separated by space. For example:debug("Value of i is", 10)Alternatively, output can be written directly tostderrusing thewritefunction. For example:import sys

x = [1, 2, 3]
sys.stderr.write("{}\n".format(x))The output is written in thescidb-stderr.logfiles of each
instance, for example:/opt/scidb/18.1/DB-scidb/0/0/scidb-stderr.log
/opt/scidb/18.1/DB-scidb/0/1/scidb-stderr.logIf using SciDB18.1installed in the default location and
configured with one server and two instances.ImportError: No module namedWhen trying to de-serialize a Python function uploaded to SciDB usingpack_func, one might encounter:ImportError: No module named ...This error is becausedill, the Python serialization library,
links the function to the module in which it is defined. This can be
resolved in two ways:Make the named module available on all the SciDB instancesIf the module is small, the recursivedillmode can be
used. Replace:foo_pack = scidbstrm.pack_func(foo)with:foo_pack = numpy.array([dill.dumps(foo, 0, recurse=True)])