Project: talenddataquality
Issue TDQ-12300: The new tNaiveBayesModel with native spark mllib
The current tNaiveBayesModel component is based in our own API ([see code|https://github.com/Talend/tdq-studio-ee/tree/master/main/plugins/org.talend.datascience.mllib/src/main/scala/org/apache/spark/mllib/classification/talend]) with Spark1.3, but we've met some compatibility issues(see TDQ-12249) with the different spark versions. It should be more easy to maintenance if we use the native spark mllib API for [NaiveBayes|https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala]. We want to keep the old implementation if run this component in spark 1.3 but to use the native spark mllib api if run this component in spark 1.4+. So we may need to add a combo list in UI for choosing the spark versions: 1.3 or 1.4+ and include different javajets according to the spark version. See the example of the component [tPredict|https://github.com/Talend/tdq-studio-ee/tree/master/main/plugins/org.talend.designer.components.tdqsparkprovider/components/tPredict] which also has a combo list for choosing the different model types to include different javajets. For coding the javajet with Spark 1.4, please see the example of component: [tKmeansModel|https://github.com/Talend/tdq-studio-ee/tree/master/main/plugins/org.talend.designer.components.tdqsparkprovider/components/tKMeansModel], and read the [Spark Documentation|http://spark.apache.org/docs/latest/mllib-naive-bayes.html] for help. P.S. tNaiveBayesModel should has two models inside of this components: 1. the pipelineModel for preprocess, 2. the model of Naive Bayes (like [tKmeansModel|https://github.com/Talend/tdq-studio-ee/blob/master/main/plugins/org.talend.designer.components.tdqsparkprovider/components/tKMeansModel/tKMeansModel_sparkconfig.javajet], see L62 and L135 in config.javajet)
Story points: 5