Package: daskcheck
Project daskcheckTools for a more simple use of thedask. Dask scheduler is defined
in~/.dask_ serverfile, the content is just an ip address.Work is in progress...The ideadefine properly thecore function(xcorefuncin the example)
withTHE proper returndaskcheckthen will take care about:sending the task to the scheduler (but this is natural)uploads (python) single-file module to workers (via .upload)sending the parameterscollecting results and saving them tolocal json filein futureabout parsing the localjsonfilein futureabout sending local (bash) scripts to workers... (?)in futureabout managing (worker's) folders with data output
(if output is too large)...Files in the repoIt is becoming a bit messy, so for the reference:batchforworkerX BATCH TEST in remote ~/sand
bindaskcheck.py                     X nothing will be main script
config.py                             X module configs
convjson2spectra.py                 exodaskoutput conversion to spectra
daskcheck                             FOLDER
daskcheck.py                          module OPERATIONS
daskresultslog20230510142235.json   log file
docextr.py                            Xattemt to autocreate
exodask.py                          ???work on exogam
f.py                                  ??resulting autogenerate
OldLogs                               Previous files
README.md                             MD
README.org                            this file
runschedulerRUN SCHEDULER
runsyncversions??KEEP dasksched-workers in sync
runworkerRUN WORKER
setup.py                              setup python
singlemod.py                          MODULE for import
singlexec.py                          XOBSOLETE - remote exec BASH
unitname.py                           generic module
version.py                            version is hereWAITING Instalation of daskcheckpip install daskcheckInstallation of dask 2023Seehttps://docs.dask.org/en/stable/install.htmlpip install "dask[complete]"Launching dask scheduler/workersPay attention to correct/compatible libraries on different workersopen port 8786 and 8787 on scheduler and longrangeof ports on
workers...Testing basics on localhost - the most simple in-terminal wayrun_scheduleryou see it runningdask worker 127.0.0.1:8786 --nworkers 1 --nthreads 1at this stage do not runrun_worker, it looks for serverStarting established connection to tcp://127.0.0.1:8786./daskcheck.py locsee local function output, should work in any case./daskcheck.py test -s 127.0.0.1overide the server to 127.0.0.1 to see if everything worksSet your server on worker(s)nano ~/.dask_serverand put 10.10.0.2 or whatever address your server
hasTest basics on networkgo to server and runrun_schedulergo to worker and runrun_workerCHECK LIBRARIES and upgrade what is needed e.g.:pip3 install tornado --upgradeTEST with./daskcheck.py testfrom workerLaunching worker from cmdline withlocalschedulerdask     worker 127.0.0.1:8786 --nworkers 5 --nthreads 1Testing daskJust local run of getcpuinfo./daskcheck.py locThis runs (scheduler and workers should be running) 40x getcpuinfoin
cluster./daskcheck.py netDOING Run single-file - (python) moduleOR (bash) batch./daskcheck.py file-fileparameter tells a filename and parameters
is comming...# python function with main - example
./daskcheck.py file singlemod.py  11..33How is that done?importlib is used to get the module's MAIN,UPLOADsinglemod.pyto the scheduler,chdir to/dask_sendboxandruntell()andmain()function of the singlemod.py:singlemod.py - writes a file to (worker's)~/dask_sendboxdeprec* singlexec.py - launches./runme- if not present in
(worker's)~/dask_sendbox, it crashesbatchforworker- bash script, writes file to (worker's)~/dask_sendboxPreviously,batch_for_workerLOAD and RUN was hardwired
in thesignlexec.py.singlemod.py- whatever returns, goes to json and cvsMonitoring daskHave open port on scheduler:xdg-open http://localhost:8787[TODO]{.todo .TODO} Recollection the data from json {#recollection-the-data-from-json}to recover...[TODO]{.todo .TODO} Python native check ?what? {#python-native-check-what}It must be checked that python works too - as beforeDEVELOPMENTrestart scheduler after updatingdaskcheckcheck open port when runworker[ok]Appendixrunworker- environment needed and command#!/bin/bash

# PYTHON bin exports
export PATH=$PATH:$HOME/.local/bin

# ROOT exports
export PYTHONPATH=$HOME/root/lib/
export ROOTSYS=$HOME/root
export PATH=$ROOTSYS/bin:~/bin:$PATH
export LD_LIBRARY_PATH=$ROOTSYS/lib:$ROOTSYS/lib/root:$LD_LIBRARY_PATH

source $HOME/root/bin/thisroot.sh

# CONFIG FOR WORKER
export DISPLAY=:0
export DS=$HOME/.dask_server  # get IP of the scheduler
export DSER=`cat $DS`
export HOST=`hostname`
export workers=2 # how many cores to run
export PORT=8786
cd /tmp

if [ -f  "$DS" ]; then
    echo ... OK $DS exists
else
    echo ... NO $DS exists
    sleep 5
    echo ...
    exit 1
fi


echo ... I am on $HOST and trying to connect to /$DSER/ one thread per worker
dask worker ${DSER}:${PORT}      --nworkers $workers --nthreads 1runscheduler- script#!/bin/bash


#dask scheduler --port 8786
export PATH=$PATH:$HOME/.local/bin

export PORT=8786
export HOST=`hostname`

cd /tmp

if [ "$HOST" = "core6a" ]; then
    echo ... starting scheduler
    dask scheduler   --port ${PORT}   #  --bokeh-port 8787
fi
echo ... scheduler eneded or not launched at all
sleep 5
exit 0Run a (python) function from python codeThis must be updated...exo_dask.pyContains a working (in the past) example, usingroot.This is (or should be) a python code that usesdaskcheckfor sending a
function.It is evidently crippled for the moment...from daskcheck import daskcheck

from fire import Fire
import time
import platform
import datetime as dt
import json

def main( parlist ):
    """
    Initiated by Fire. If one parameter, runs locally with local xcorefunc
    """
    parameters = daskcheck.prepare_params( parlist )

    if type(parameters)==list:
        print("i... viable for DASK ....")
        daskcheck.submit( daskcheck.get_cpu_info , parameters)
    elif type(parameters)==tuple:
        print("i... viable for DASK ....")
        daskcheck.submit( daskcheck.get_cpu_info , parameters)
    else:
        print("i... running only locally")
        my_results = xcorefunc( 1 , parameters ) # order = 1, just arbitrary number
        # Write LOG file.
        now = dt.datetime.now()
        stamp = now.strftime("%Y%m%d_%H%M%S")
        with open(f"dask_results_log_{stamp}.json", "w") as fp:
            json.dump( my_results , fp, sort_keys=True, indent='\t', separators=(',', ': '))
    return

def xcorefunc( order, param):
    """
    CORE function to be sent to dask schedule.

    :param order: order number of the call
    :param param: parameter to be sent
    """
    import ROOT # *TRICK* I need to import here to avoid breaking pickle
    start_time = time.perf_counter()

    return order, [platform.node(),  f"{time.perf_counter() - start_time:.1f} s" ]


if __name__=="__main__":
    Fire(main)