Apache Ignite ML: origins and development
By: Alexey Zinoviev
Tags: Apache Ignite, Machine Learning

Motivation (to do yet another framework)

Why do people create frameworks? What drives them? Ambition, thirst for knowledge, questions from users, challenges of fate?

First motivation: Honestly, I think that, initially, the idea was to create a simple framework at the top of Apache Ignite with models like Linear Regression and Decision Trees to compete with other distributed systems, like Redis or Apache Spark or Oracle, which have their data-science tools. Purely for show.

Second motivation: As I remember, some Ignite users asked about making fast predictions in memory by using pre-trained models. But all of the models were trained in Python, and, as a result, serialized Python models could be easily used for prediction on Ignite caches.

Third motivation: Users much consider the effect that scalability can have on performance. Just like Apache Spark and Apache Mahout, Apache Ignite can be scaled and trained on data partitions.

With Apache Ignite ML, 1000 nodes with 10000 data partitions are required to train a Decision Tree or NaiveBayes classifiers.

Fourth motivation: There are not many useful ML libraries in the JVM world. Unlike with scikit-learn in Python, there is no community-driven gold standard. However, the creation of ML libraries (whether distributed or non distributed) brings new possibilities. Because the Java, Scala, and Kotlin languages use a standard JVM bytecode, libraries that are written in any of the languages can be easily called and reused for data processing.

As a result, we could reduce network traffic, CPU operations, and volumes of allocated memory if all these actions: data source reading, preprocessing, training, model deployment, and prediction could be made on one platform. If data-source reading, preprocessing, training, model deployment, and prediction can be made on one platform, then network traffic, CPU operations, and size of allocated memory can be reduced.

Fifth motivation: If you are using a distributed system, probably your data is too large to fit on one server or, in the case of Apache Ignite, within the memory of one server. If you have a lot of data, training on the data might require a good deal of time. If you lose a node, you might lose your calculations and have to restart the training. Apache Ignite Machine Learning is tolerant of node failures. Therefore, if a node fails during the learning process, recovery procedures are transparent to you, your training processes aren’t interrupted, and you obtain results relatively quickly, almost the same as if all nodes were working as they should. For detailed information, see Partition Based Dataset [2].

Sixth motivation. Last but not least: Be honest, all main Big Data tools are written in JVM languages like Java or Scala (not to mention Kafka, Spark, Hadoop, Flink, Ignite, and Cassandra), and all of the languages are developed under the Apache community rules. But, the data-analysis pipeline has a missing link. It lacks the ML and data-science tools that integrate well with other JVM Big Data tools. This deficiency caused the Flink and Spark communities to implement its ML frameworks and played a significant role in creating the Apache Ignite ML framework.

What was the motivation for me, personally, to participate in the Ignite ML project during the past three years and continue my job?

The main reason was to create competition for the Apache Spark ML framework, which offers a lot of advantages and is widely known but provides limited support for model ensembles, integration with other ML frameworks, and online-learning [3]. After a few fruitless attempts to change something within the Apache Spark, I joined the Apache Ignite community, which began the development of the Ignite ML framework by trying to utilize the basics of Apache Mahout at the top of Apache Ignite.