Package: ncapi-client
ncapi-client-pythonThis repository contains a python client for the NCAPI, and a
command-line utilityncsthat allows using this client.All the methods for non-interactive tasks (e.g. dataset management,
training, batch prediciton) are contained in thencapi_client.Clientclass, and all the methods for the interactive ones are contained in
thencapi_client.SessionClient.InstallationYou can install the client by running:pipinstall.For development purposes, you probably want to install a "editable" version,
which creates symlinks instead of actually copying stuff in the directories:pipinstall-e.General descriptionThere are several sets of commands and utils for managing your data
and models, which are exposed in all of the clients. Those are
described in more detail below.  For cli interface,ncs, you can see
the overall list of in the help doc:Commands:authAuthenticatewithyourusernameandpassword.datasetDatasetmanagement.evaluateBatchevaluation.modelModelmanagement.predictBatchprediction.sessionInteractivesessionsmanagement.trainTrainingjobsmanagement.AuthenticationIn order to connect to the API you should first authenticate with your
username and password, which will save your settings in~/.config/ncs.
The easiest way to do this is to runncs auth:ncsauth
Pleaseenteryourusername:ncadmin
Pleaseenteryourpassword:
PleaseenterAPIurl[http://0.0.0.0:5000]:You can then use the python client and ncs, which will load and update
your settings when necessary.For python, you can also create an instance ofncapi_client.Clientand pass
it credentials and API url (optional if you have custom installation):NCAPI_URL='https://cloud.neuralconcept.com:5000'NCAPI_USERNAME='ncuser'NCAPI_PASSWORD='ncpassword'c=Client(NCAPI_URL,NCAPI_USERNAME,NCAPI_PASSWORD)Managing datasetsYou can add (upload) datasets in one of the supported formats (formats
are described in the corresponding docs), and then use your datasets for training and testing# adding a datasetd=c.dataset_add(name="name-of-the-dataset",path="/path/to/the/data")# getting a datasetd=c.dataset_get("name-of-some-dataset")# deleting the datasetc.dataset_delete(d.uuid)# listing the datasetsfordinc.datasets:print(d.name,d.uuid,d.status)# listing the files in the datasetf=d.dataset_files_list()# getting a sample from the datasets=d.dataset_samples_get(d.uuid,"<sample-id>")For CLIncs dataset COMMANDaddAddanewNAMEdatasetlocatedatlocalPATH.deleteDeletesthedataset.samfilesDatasetfilemanagement.getGetverboseinfoaboutthedataset.listListallthedatasetsManaging modelsFor models, you can choose from a rich set of pre-defined model
templates, which you can easily customise by simply editing a
yaml-based config file. A typical config would look as follows:# training parameterstrain:batch_size:1tag:alphanum_steps:1500000optimizer:init_lr:1.0e-4min_lr:1.0e-6decay_rate:0.7decay_every:30000loss:loss_fn_name:null# model-specific parametersmodel:name:name-of-your-model# this is one of the standard model typesclass_name:ncs.models.NormalizedRegressorcharacteristic_len:0.4num_blocks:8block_width:64There are two main blocks:train- which contains customizable
training parameters such as the step size schedule,model- which
contains the definition of the model, i.e. it's name, the class name
(type of the model), and model-specific settings, such as number of
blocks or number of parameters per block (those will really depend on
a particular model type).For python:# listing available modelsforminc.models:print(m.uuid,m.name)# adding a modelm=c.model_add('/path-to-your-model-config.yml'')# deleting a modelc.model_delete(m.uuid)# getting verbose info about the modelprint(c.model_get(m.uuid))For CLIncs model COMMAND:Commands:addAddamodelwithyamlconfigatPATH.deleteDeleteamodelbyUUID.getGetverboseinfoaboutthemodel.listListavailablemodels.Managing training jobsWith NCAPI you can run large-scale training jobs on a cluster (cloud
based or custom), without a need to worry about managing computational
resources: our backend does it automatically.For python:# list the trainingsfortinsorted(c.trainings,key=lambdav:v.uuid):print(t.uuid,t.status)# stop one of the trainingsc.training_stop(c.trainings[0].uuid)# restart one of the previously stopped trainingsc.training_restart(c.trainings[0].uuid)# submit a new trainingjob=c.training_submit(c.models[0].uuid,c.datasets[0].uuid)# you can also provide additional user config for the training# which overrides the model config, e.g. if you want to train# with different training step size schedule or try different loss# functionjob=c.training_submit(model,dataset,"/path-to-custom-config"")For CLI:Commands:deleteDeleteatraining.getGetverboseinformationaboutatraining.listListavailabletrainings.restartRestartatraining.stopStopatraining.submitSubmitatrainingtothequeue.Managing prediction jobsTODO: finish the python docsFor CLI, runncs predict COMMAND:Commands:deleteDeleteaprediction.getGetresultsofthepredictionwithUUIDandsavetheminDST.stopStopapredictionjob.submitSubmitapredictionjobforagivenMODELandDATASET.Managing evaluation jobsTODO: finish the python docsFor CLI, runncs evaluate COMMAND:Commands:deleteDeleteanevaluationjob.getGetsummarymetricsoftheevaluation.restartRestartanevaluationjob.stopStopanevaluationjob.submitSubmitanevaluationjobforagivenMODELandDATASET.Managing interactive sessionsIn python:# list running sessionsforsinc.sessions:print(s)session=c.session_start(model.id,dataset.id,sample.id)session=c.session_get(session.id)print("waiting while the session starts...")whilesession.status!="RUNNING":session=c.session_get(session.id)time.sleep(1.0)print("it has started!")# start a session client and interactively play with your modelsc=SessionClient(session,client=c)# get the current mesh (if supported by the model)verts,faces=sc.mesh# get the predictions (if supported by the model)preds=kc.predict(input_scalars=dict(angle=0.8,speed=0.5))# do some post-processing on field valuesfields=np.clip(preds['fields'],0,10)# choosing some random control points and applying a random deformationcontrol_points=verts[np.random.choice(verts.shape[0],size=2)]cp_deformations=np.random.normal(scale=0.1,size=control_points.shape)sc.apply_deformation(control_points=control_points,cp_deformations=cp_deformations,)# get updated mesh and predictionsverts,faces=sc.meshpreds=sc.predict(input_scalars=dict(angle=0.8,speed=0.5))In CLI, you can run some of the high-level commands viancs session COMMAND:Commands:deleteDeleteanexistingsession.listListofinteractivesessions.startStartaninteractivesessionwithagivenmodelanddatasetDocumentationTo update the html docs, you will need sphinx >= 2.0. You can then go in ncapi-python-client root directory and runsphinx-build -b html docs/ docs/_build/html/Corresponding docs will be created in the docs/_build/html folder.
If you added new modules, you might want to create new *.rst files in the docs/ folder for these modules.