Limited-Memory Broyden-Fletcher-Goldfarb-Shanno Algorithm in ML.NET | by Robert Krzaczyński | Towards Data Science

Some time ago I published an article about the implementation of Naive Bayes using ML.NET. Continuing this series today I would like to introduce you to the Limited-Memory Broyden-Fletcher-Goldfarb-Shanno method. I will start with a theory and an explanation of what this method is about and what it is used for.
Some facts
The creator of the algorithm is Jorge Nocedal. It was created at the Optimization Center, a joint venture of Argonne National Laboratory and Northwestern University. The original source code was written in FORTRAN. It is referred to as software for large-scale unconstrained optimization. As you might expect from the name, this method is similar to BFGS, but it uses less memory. Therefore, it is perfect for large datasets.
L-BFGS
This is an algorithm from the Quasi-Newton family of methods. These are algorithms for finding local extrema of functions, which are based on Newton’s method of finding stationary points of functions.
Time for some math
In these methods, a second-degree approximation is used to find the minimum function f(x). Taylor series of function f(x) is as follows:
where delta f is a gradient of the function and H its hessian.
Taylor series of gradient looks like this:
We want to find the minimum, which is to solve the equation:
From here:
Now, Hessian should be appointed. Each method belonging to this family has a different way of appointing it. I think we can end this part. I don’t want to bore you with these formulas, but I wanted you to have a brief idea of what this is about.
The difference between BFGS and L-BFGS
As I mentioned earlier, the L-BFGS algorithm works well with large datasets because it needs less memory than the standard BFGS. Both algorithms use the Hessian inverse matrix estimation to control variable space searching. While BFGS stores a dense n x n approximation to the inverse Hessian, L-BFGS stores only a few vectors that represent the approximation implicitly. This is the difference that saves memory.
Dataset
I used the skin segmentation dataset from the UCI Machine Learning Repository for the experiment. The analyzed data set has 3 features and 2 classes. The classes determine if a sample is considered to be a skin or not. The dataset has 245057 instances, so the L-BFGS algorithm will work perfectly here.
Implementation
After creating a console application project and downloading ML.NET from NuGet Packages, you can proceed to implementation and model creation. In the beginning, you should create classes corresponding to the attributes of our dataset. Created classes are shown in the listing:
Then you can go on to load the dataset and divide it into a training and a testing set. I propose to adopt the popular division, i.e. 70% is a training set and 30% is a testing set.
var dataPath = "../../skin-segmentation.csv";var ml = new MLContext();var DataView = ml.Data.LoadFromTextFile<Features>(dataPath, hasHeader: true, separatorChar: ',');var partitions = ml.Data.TrainTestSplit(DataView, testFraction: 0.3);
Now you need to adapt the model structure to the standards proposed by the ML.NET library. This means that the property specifying the class must be called Label. The remaining attributes must be condensed under the name Features.
var pipeline = ml.Transforms.Conversion.MapValueToKey(inputColumnName: "Class", outputColumnName:"Label").Append(ml.Transforms.Concatenate("Features", "V1","V2","V3")).AppendCacheCheckpoint(ml);
Now is the time for creating a training pipeline. Here you choose a classifier in the form of L-BFGS, to which you specify in the parameters the column names of the label and features. You indicate the property that means the predicted label too.
var trainingPipeline = pipeline.Append(ml.MulticlassClassification.Trainers.LbfgsMaximumEntropy("Label","Features")).Append(ml.Transforms.Conversion.MapKeyToValue("PredictedLabel"));
After completing the previous steps, you can now go to training and testing the model:
var trainedModel = trainingPipeline.Fit(partitions.TrainSet);var testMetrics = ml.MulticlassClassification.Evaluate(trainedModel.Transform(partitions.TestSet));
Results and summary
The accuracy of the L-BFGS algorithm was 91,8%. The result seems very good, but of course, deeper analysis and the use of other metrics are needed to confirm its value.In this article, I introduced you to the L-BFGS algorithm and showed you how to use it using ML.NET. Its use in ML.NET is limited to a few lines of code, but I think it’s worth having a view and knowledge of what this algorithm is about.