Package: faculty-xval
faculty-xvalCross validation of machine-learning models on Faculty platform. At present, the
package mostly offers a way to cross validate models in parallel by means of
Faculty jobs. To access the functionality one makes use of the class:faculty_xval.validation.JobsCrossValidatorAdditional information is found in the example notebooks provided. Please have a
look at the sectionTry out the examplesbelow.The package supportskerasandsklearnmodels. Whilst one can write custom
models that are compatible withfaculty-xval, no guarantee is given that the
package handles these situations correctly, in particular because of issues
concerning the randomisation of weights.Two sets of installation instructions are provided below:If you would like to simply usefaculty-xval, please follow theUser installation instructions.If you would like to developfaculty-xvalfurther, please follow theDeveloper installation instructions.User installation instructionsCreate an environmentIn your project on Faculty platform, create an environment namedfaculty_xval.
In thePYTHONsection, selectPython 3andpipfrom the dropdown menus.
Then, typefaculty-xvalin the text box, and click on theADDbutton.The environment installs the packagefaculty-xval, and should be applied on
every server that you create; this includes both interactive servers and job
servers, as explained next.Create a job definitionCreate a new job definition namedcross_validation. In theCOMMANDsection,
paste the following:faculty_xval_jobs_xval $IN_PATHSThen, add aPARAMETERwith the nameIN_PATHS, and ensure that theMake field mandatorybox is checked.Finally, underSERVER SETTINGS, addfaculty_xvalto theENVIRONMENTSsection.For cross-validation jobs that are computationally intensive, we recommend using
dedicated servers as opposed to running on shared infrastructure. To achieve
this, click onLarge and GPU serversunderSERVER RESOURCES, and select an
appropriate server type from the dropdown menu.Remember to clickSAVEwhen you are finished.Developer installation instructionsSelect a usernameBefore beginning the installation process, pick an appropriate username, such asfoo. This does not necessarily need to match your Faculty platform username.
In the following instructions, your selected username will be referred to as<USER_NAME>.Clone the repositoryCreate the folder/project/<USER_NAME>. Then, run the commands:cd/project/<USER_NAME>
gitclonehttps://github.com/facultyai/faculty-xval.gitCreate an environmentNext, create an environment in your project namedfaculty_xval_<USER_NAME>.In this environment, underSCRIPTS, paste in the following code to theBASHsection, remembering to change theUSER_NAMEdefinition on the second line to
your selected<USER_NAME>:# Remember to change username!USER_NAME=<USER_NAME># Install faculty-xval from local repository.pipinstall/project/$USER_NAME/faculty-xval/# Turn USER_NAME into an environment variable.echo"export USER_NAME=$USER_NAME">/etc/faculty_environment.d/app.shif[[-d/etc/service/jupyter]];thensudosvrestartjupyterfiThis environment should be applied on every server that you create; this
includes both 'normal' interactive servers and job servers, as explained next.Create a job definitionNext, create a new job definition namedcross_validation_<USER_NAME>. In theCOMMANDsection, paste the following:faculty_xval_jobs_xval $in_pathsThen, add aPARAMETERwith the namein_paths, and ensure that theMake field mandatorybox is checked.Finally, underSERVER SETTINGS, addfaculty_xval_<USER_NAME>to theENVIRONMENTSsection.For cross-validation jobs that are computationally intensive, we recommend using
dedicated servers as opposed to running in the cluster. To achieve this, click
onLarge and GPU serversunderSERVER RESOURCES, and select an appropriate
server type from the dropdown menu.Remember to clickSAVEwhen you are finished.Try out the examplesPlease clone this repository. Examples of cross validation withfaculty-xvalfor the different types of model are provided in the directoriesexamples/kerasandexamples/sklearn. Usage instructions are then divided in
two notebooks:jobs_cross_validator_run.ipynbloads the data, instantiates the model, and
starts a Faculty job that carries out the cross validation.jobs_cross_validator_analyse.ipynbgathers the results from the cross
validation, reloads the target data, and calculates the model accuracy over
multiple train-test splits.Note that the example notebooks must be run in the order just defined.