Package: lolopy
Python Wrapper for Lolololopyimplements a Python interface to theLolo machine learning library.Lolo is a Scala library that contains a variety of machine learning algorithms, with a particular focus on algorithms that provide robust uncertainty estimates.lolopygives access to these algorithms as scikit-learn compatible interfaces and automatically manages the interface between Python and the JVM (i.e., you can uselolopywithout knowing that it is running on the JVM)Installationlolopyis available on PyPi. Install it by calling:pip install lolopyTo uselolopy, you will also need to install Java JRE >= 1.8 on your system.
ThelolopyPyPi package contains the compiledlololibrary, so it is ready to use after installation.DevelopmentLolopy requires Python >= 3.8, Java JDK >= 1.8, and sbt to be installed on your system when developing lolopy.Before developinglolopy, compileloloon your system using sbt.
We have provided aMakefilethat contains the needed operations.
To build and installlolopycallmakein this directory.UseTheRandomForestRegressorclass most clearly demonstrates the use oflolopy.
This class is based on theRandom Forest with Jackknife-based uncertainty estimates of Wagner et al,
which - in effect - uses the variance between different trees in the forest to produce estimates of the uncertainty of each prediction.
Using this algorithm is as simple as using theRandomForestRegressor from scikit-learn:fromlolopy.learnersimportRandomForestRegressorrf=RandomForestRegressor()rf.fit(X,y)y_pred,y_std=rf.predict(X,return_std=True)The results of this code is to produce the predicted values (y_pred) and their uncertainties (y_std).See theexamplesfolder for more examples and details.You may need to increase the amount of memory available tololopywhen using it on larger dataset sizes.
Setting the maximum memory footprint for the JVM running the machine learning calculations can be
achieved by setting theLOLOPY_JVM_MEMORYenvironment variable.
The value forLOLOPY_JVM_MEMORYis used to set the maximum heap size for the JVM
(seeOracle's documentation for details).
For example, "4g" allowsloloto use 4GB of memory.Implementation and Performancelolopyis built using thePy4Jlibrary to interface with the Lolo scala library.
Py4J provides the ability to easily managing a JVM server, create Java objects in that JVM, and call Java methods from Python.
However, Py4Jhas slow performance in transfering large arrays.
To transfer arrays of features (e.g., training data) to the JVM before model training or evaluation, we transform the data to/from Byte arrays on the Java and Python sides.
Transfering data as byte arrays does allow for quickly moving data between the JVM and Python but requires holding 3 copies of the data in memory at once (Python, Java Byte array, and Java numerical array).
We could reduce memory usage by passing the byte array in chunks, but this is currently not implemented.Our performance for model training is comparable to scikit-learn, as shown in the figure below.
The blue-shaded region in the figure represents the time required to pass training data to the JVM.
We note that training times are equivalent between using the Scala interface to Lolo andlolopyfor training set sizes above 100.Lolopy and lolo are currently slower than scikit-learn for model evaluation, as shown in the figure below.
The model timings are evaluated on a dataset size of 1000 with 145 features.
The decrease in model performance with training set size is an effect of the number of trees in the forest being equal to the training set size.
Lolopy and lolo have similar performance for models with training set sizes of above 100.
Below a training set size of 100, the cost of sending data limits the performance oflolopy.For more details, see thebenchmarking notebook.