Package: arkouda
Arkouda (αρκούδα): NumPy-like arrays at massive scale backed by Chapel.NOTE: Arkouda is under the MIT license.Gitter channelsArkouda Gitter channelChapel Gitter channelTalks on ArkoudaMike Merrill's CHIUW 2019 talkBill Reus' CLSAC 2019 talk(PAW-ATM)talkandabstractAbstract:Exploratory data analysis (EDA) is a prerequisite for all data
science, as illustrated by the ubiquity of Jupyter notebooks, the
preferred interface for EDA among data scientists. The operations
involved in exploring and transforming the data are often at least as
computationally intensive as downstream applications (e.g. machine
learning algorithms), and as datasets grow, so does the need for HPC-enabled
EDA. However, the inherently interactive and open-ended nature of
EDA does not mesh well with current HPC usage models. Meanwhile, several
existing projects from outside the traditional HPC space attempt to
combine interactivity and
distributed computation using programming paradigms and tools from
cloud computing, but none of these projects have come close to meeting
our needs for high-performance EDA.To fill this gap, we have
developed a software package, called Arkouda, which allows a user to
interactively issue massively parallel computations on distributed
data using functions and syntax that mimic NumPy, the underlying
computational library used in the vast majority of Python data science
workflows. The computational heart of Arkouda is a Chapel interpreter
that
accepts a pre-defined set of commands from a client (currently
implemented in Python) and
uses Chapel's built-in machinery for multi-locale and multithreaded
execution. Arkouda has benefited greatly from Chapel's distinctive
features and has also helped guide the development of the language.In early applications, users of Arkouda have tended to iterate rapidly
between multi-node execution with Arkouda and single-node analysis in
Python, relying on Arkouda to filter a large dataset down to a smaller
collection suitable for analysis in Python, and then feeding the results
back into Arkouda computations on the full dataset. This paradigm has
already proved very fruitful for EDA. Our goal is to enable users to
progress seamlessly from EDA to specialized algorithms by making Arkouda
an integration point for HPC implementations of expensive kernels like
FFTs, sparse linear algebra, and graph traversal. With Arkouda serving
the role of a shell, a data scientist could explore, prepare, and call
optimized HPC libraries on massive datasets, all within the same
interactive session.InstallationRequirements:requires chapel 1.22.0requires zeromq version >= 4.2.5, tested with 4.2.5 and 4.3.1requires hdf5requires python 3.6 or greaterrequires numpyrequires Sphinx and sphinx-argparse to build python documentationrequires pytest and pytest-env to execute the Python test harnessMacOS Environment InstallationIt is usually very simple to get things going on a mac:brewinstallzeromq

brewinstallhdf5

brewinstallchapel# Although not required, is is highly recommended to install Anaconda to provide a# Python 3 environment and manage Python dependencies:wgethttps://repo.anaconda.com/archive/Anaconda3-2020.02-Linux-x86_64.sh
shAnaconda3-2020.02-Linux-x86_64.shsource~/.bashrc# Otherwise, Python 3 can be installed with brewbrewinstallpython3# !!! the standard way of installing through pip3 installs an old version of arkouda# !!! the arkouda python client is available via pip# !!! pip will automatically install python dependencies (zmq and numpy)# !!! however, pip will not build the arkouda server (see below)# !!!pip3 install arkouda## install the version of the arkouda python package which came with the arkouda_server# if you plan on editing the arkouda python package use the -e flag# from the local arkouda repo/directory run...pip3install-e.## these packages are nice but not a requirementpip3installpandas
pip3installjupyterIf it is preferred to build Chapel instead of using the brew install, the process is as follows:# on my mac build chapel in my home directory with these settings...exportCHPL_HOME=~/chapel/chapel-1.22.0source$CHPL_HOME/util/setchplenv.bashexportCHPL_COMM=gasnetexportCHPL_COMM_SUBSTRATE=smpexportCHPL_TARGET_CPU=nativeexportGASNET_QUIET=YexportCHPL_RT_OVERSUBSCRIBED=yescd$CHPL_HOMEmake# Build chpldoc to enable generation of Arkouda docsmakechpldoc# Add the Chapel executable (chpl) to PATH either in ~/.bashrc (single user)# or /etc/environment (all users):exportPATH=$CHPL_HOME/bin/linux64-x86_64/:$PATHLinux Environment InstallationThere is no Linux Chapel install, so the first two steps in the Linux Arkouda install are
to install the Chapel dependencies followed by downloading and building Chapel:# Update Linux kernel and install Chapel dependencies
sudo apt-get update
sudo apt-get install gcc g++ m4 perl python python-dev python-setuptools bash make mawk git pkg-config

# Download latest Chapel release, explode archive, and navigate to source root directory
wget https://github.com/chapel-lang/chapel/releases/download/1.22.0/chapel-1.22.0.tar.gz
tar xvf chapel-1.22.0.tar.gz
cd chapel-1.22.0/

# Set CHPL_HOME
export CHPL_HOME=$PWD

# Add chpl to PATH
source $CHPL_HOME/util/setchplenv.bash

# Set remaining env variables and execute make
export CHPL_COMM=gasnet
export CHPL_COMM_SUBSTRATE=smp
export CHPL_TARGET_CPU=native
export GASNET_QUIET=Y
export CHPL_RT_OVERSUBSCRIBED=yes
cd $CHPL_HOME
make

# Build chpldoc to enable generation of Arkouda docs
make chpldoc

# Optionally add the Chapel executable (chpl) to the PATH for all users: /etc/environment
export PATH=$CHPL_HOME/bin/linux64-x86_64/:$PATHAs is the case with the MacOS install, it is highly recommended to install Anaconda to provide a Python environment
and manage Python dependencies:wget https://repo.anaconda.com/archive/Anaconda3-2020.02-Linux-x86_64.sh
 sh Anaconda3-2020.02-Linux-x86_64.sh
 source ~/.bashrcBuilding ArkoudaDownload, clone, or fork thearkouda repo. Further instructions assume that
the current directory is the top-level directory of the repo.If your environment requires non-system paths to find dependencies (e.g.,
if using the ZMQ and HDF5 bundled with [Anaconda]), append each path to a new fileMakefile.pathslike so:# Makefile.paths# Custom Anaconda environment for Arkouda$(eval$(calladd-path,/home/user/anaconda3/envs/arkouda))#                      ^ Note: No space after comma.Thechplcompiler will be executed with-I,-Land an-rpathto each path.# If zmq and hdf5 have not been installed previously, execute make install-deps
make install-deps

# Run make to build the arkouda_server executable
makeNow that the arkouda_server is built and tested, install the Python libraryInstalling the Arkouda Python Librarypip3 install -e .Testing ArkoudaThere are two unit test suites for Arkouda, one for Python and one for Chapel. As mentioned above, the Arkouda Python
test harness leverages thepytestandpytest-envlibraries, whereas the Chapel test harness does not require any external librares.For more details regarding Arkouda testing, please consult the Python testREADMEand Chapel testREADME, respectively.Building the Arkouda documentationMake sure you've installed the Sphinx and sphinx-argparse packages (e.g.pip3 install -U Sphinx sphinx-argparse).Important: if you've built Chapel, you must execute make chpldoc as detailed above.Runmake docto build both the Arkouda python documentation and the Chapel server documentationThe output is currently in subdirectories of thearkouda/docarkouda/doc/python # python frontend documentation
arkouda/doc/server # chapel backend server documentationTo view the documentation for the Arkouda python client, point your browser tofile:///path/to/arkouda/doc/python/index.html,
substituting the appropriate path for your configuration.Running arkouda_serverThe command-line invocation depends on whether you built a single-locale version (withCHPL_COMM=none) or
multi-locale version (withCHPL_COMMset).Single-locale startup:./arkouda_serverMulti-locale startup (user selects the number of locales):./arkouda_server-nl2Also can run server with memory checking turned on using./arkouda_server--memTrack=trueBy default, the server listens on port5555and prints verbose output. These options can be changed with command-line
flags--ServerPort=1234and--v=falseMemory checking is turned off by default and turned on by using--memTrack=trueLogging messages are turned on by default and turned off by using--logging=falseVerbose messages are turned on by default and turned off by using--v=falseOther command line options are available, view them by using--helpTesting arkouda_serverTo sanity check the arkouda server, you can runmakecheckThis will start the server, run a few computations, and shut the server down. In addition, the check script can be executed
against a running server by running the following Python command:python3tests/check.pylocalhost5555Contributing to ArkoudaIf you'd like to contribute, please seeCONTRIBUTING.md.