Project: Apache Spark (App Definition and Development / Streaming & Messaging)
File: mllib-isotonic-regression.md

layout: global title: Isotonic regression - RDD-based API displayTitle: Regression - RDD-based API license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. belongs to the family of regression algorithms. Formally isotonic regression is a problem where given a finite set of real numbers `$Y = {y1, y2, ..., y_n}$` representing observed responses and `$X = {x1, x2, ..., x_n}$` the unknown response values to be fitted finding a function that minimizes `\begin{equation} f(x) = \sum{i=1}^n wi (yi - xi)^2 \end{equation}` with respect to complete order subject to `$x1\le x2\le ...\le xn$` where `$wi$` are positive weights. The resulting function is called isotonic regression and it is unique. It can be viewed as least squares problem under order restriction. Essentially isotonic regression is a best fitting the original data points. `spark.mllib` supports a which uses an approach to . The training input is an RDD of tuples of three double values that represent label, feature and weight in this order. In case there are multiple tuples with the same feature then these tuples are aggregated into a single tuple as follows: Aggregated label is the weighted average of all labels. Aggregated feature is the unique feature value. Aggregated weight is the sum of all

Additionally, IsotonicRegression algorithm has one optional parameter called $isotonic$ defaulting to true. This argument specifies if the isotonic regression is isotonic (monotonically increasing) or antitonic (monotonically decreasing). Training returns an IsotonicRegressionModel that can be used to predict labels for both known and unknown features. The result of isotonic regression is treated as piecewise linear function. The rules for prediction therefore are: If the prediction input exactly matches a training feature then associated prediction is returned. If the prediction input is lower or higher than all training features then prediction with lowest or highest feature is returned respectively. If the prediction input falls between two training features then prediction is treated as piecewise linear function and interpolated value is calculated from the predictions of the two closest features. <div class="codetabs"> <div data-lang="python" markdown="1"> Data are read from a file where each line has a format label,feature i.e. 4710.28,500.00. The data are split to training and testing set. Model is created using the training set and a mean squared error is calculated from the predicted labels and real labels in the test set. Refer to the and for more details on the API. {% includeexample python/mllib/isotonicregression_example.py %} </div> <div data-lang="scala" markdown="1"> Data are read from a file where each line has a format label,feature i.e. 4710.28,500.00. The data are split to training and testing set. Model is created using the training set and a mean squared error is calculated from the predicted labels and real labels in the test set. Refer to the and for details on the API. {% include_example scala/org/apache/spark/examples/mllib/IsotonicRegressionExample.scala %} </div> <div data-lang="java" markdown="1"> Data are read from a file where each line has a format label,feature i.e. 4710.28,500.00. The data are split to training and testing set. Model is created using the training set and a mean squared error is calculated from the predicted labels and real labels in the test set. Refer to the and for details on the API. {% include_example java/org/apache/spark/examples/mllib/JavaIsotonicRegressionExample.java %} </div> </div>